[jira] [Updated] (KYLIN-2662) NegativeArraySizeException in "Extract Fact Table Distinct Columns"
[
https://issues.apache.org/jira/browse/KYLIN-2662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI updated KYLIN-2662:
Fix Version/s: (was: v2.4.0)
> NegativeArraySizeException in "Extract Fact Table Distinct Columns"
> ---
>
> Key: KYLIN-2662
> URL: https://issues.apache.org/jira/browse/KYLIN-2662
> Project: Kylin
> Issue Type: Bug
>Reporter: liyang
>Assignee: liyang
>Priority: Major
> Labels: scope
>
> The full exception. The root cause suspect to be a cell goes over 32 KB. Need
> verification and fix.
> {code}
> java.lang.NegativeArraySizeException
> at
> org.apache.kylin.dict.TrieDictionary.getValueBytesFromIdWithoutCache(SourceFile:239)
> at
> org.apache.kylin.dict.TrieDictionaryForestBuilder.addTree(SourceFile:134)
> at
> org.apache.kylin.dict.TrieDictionaryForestBuilder.build(SourceFile:110)
> at
> org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder.build(SourceFile:218)
> at
> org.apache.kylin.engine.mr.steps.FactDistinctColumnsReducer.doCleanup(SourceFile:231)
> at
> org.apache.kylin.engine.mr.KylinReducer.cleanup(SourceFile:71)
> at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:179)
> at
> org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627)
> at
> org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
> at
> org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (KYLIN-3250) Upgrade jetty version to 9.2.24.v20180105
[ https://issues.apache.org/jira/browse/KYLIN-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI resolved KYLIN-3250. - Resolution: Fixed > Upgrade jetty version to 9.2.24.v20180105 > - > > Key: KYLIN-3250 > URL: https://issues.apache.org/jira/browse/KYLIN-3250 > Project: Kylin > Issue Type: Improvement > Components: Tools, Build and Test >Reporter: Ted Yu >Assignee: Chao Long >Priority: Minor > Fix For: v2.4.0, v2.3.2 > > > Currently Kylin depends on 9.2.20.v20161216 which is quite old. > Memory leak was fixed in Jetty 9.4 and later backported however the version > of Jetty 9.2.20 came before this fix. See > [https://github.com/eclipse/jetty.project/issues/1804] and > [https://github.com/apache/hbase/blob/master/pom.xml#L1416.] > This issue is to upgrade to > [9.3.22.v20171030|https://mvnrepository.com/artifact/org.eclipse.jetty/jetty-server/9.3.22.v20171030] > which is the latest version of 9.3 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-2484) Spark engine to support source from Kafka
[ https://issues.apache.org/jira/browse/KYLIN-2484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI resolved KYLIN-2484. - Resolution: Fixed > Spark engine to support source from Kafka > - > > Key: KYLIN-2484 > URL: https://issues.apache.org/jira/browse/KYLIN-2484 > Project: Kylin > Issue Type: Improvement > Components: Spark Engine >Affects Versions: v2.0.0 >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI >Priority: Major > Fix For: v2.4.0 > > > The current spark engine uses Spark's HiveConext to read data, so it required > that the source is in Hive; this means the Streaming cube couldn't be built > with Spark engine. > So far will add a validation with a meaningful message when user uses this > combination. Later Kylin need provide solution for this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-1768) NDCuboidMapper throws ArrayIndexOutOfBoundsException when dimension is fixed length encoded to more than 256 bytes
[
https://issues.apache.org/jira/browse/KYLIN-1768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI resolved KYLIN-1768.
-
Resolution: Duplicate
> NDCuboidMapper throws ArrayIndexOutOfBoundsException when dimension is fixed
> length encoded to more than 256 bytes
> --
>
> Key: KYLIN-1768
> URL: https://issues.apache.org/jira/browse/KYLIN-1768
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: v1.5.2
>Reporter: Dayue Gao
>Assignee: Shaofeng SHI
>Priority: Major
> Fix For: v2.4.0
>
>
> When user defined a dimension which is fixed length encoded to more than 256
> bytes, "Build N-Dimension Cuboid Data" step failed in map phase. The stack
> trace is shown below:
> {noformat}
> Error: java.lang.ArrayIndexOutOfBoundsException at
> java.lang.System.arraycopy(Native Method)
> at org.apache.kylin.cube.common.RowKeySplitter.split(RowKeySplitter.java:103)
> at
> org.apache.kylin.engine.mr.steps.NDCuboidMapper.map(NDCuboidMapper.java:125)
> at
> org.apache.kylin.engine.mr.steps.NDCuboidMapper.map(NDCuboidMapper.java:49)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
>
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> {noformat}
> The reason is that `RowKeySplitter` is hardcoded to 65 splits and 256 bytes
> per split, and trying to put a larger encoded dimension throws
> ArrayIndexOutOfBoundsException.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (KYLIN-3115) Incompatible RowKeySplitter initialize between build and merge job
[
https://issues.apache.org/jira/browse/KYLIN-3115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI resolved KYLIN-3115.
-
Resolution: Fixed
> Incompatible RowKeySplitter initialize between build and merge job
> --
>
> Key: KYLIN-3115
> URL: https://issues.apache.org/jira/browse/KYLIN-3115
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Reporter: Wang, Gang
>Assignee: Shaofeng SHI
>Priority: Minor
> Fix For: v2.4.0
>
>
> In class NDCuboidBuilder:
> public NDCuboidBuilder(CubeSegment cubeSegment) {
> this.cubeSegment = cubeSegment;
> this.rowKeySplitter = new RowKeySplitter(cubeSegment, 65, 256);
> this.rowKeyEncoderProvider = new RowKeyEncoderProvider(cubeSegment);
> }
> which will create a bytes array with length 256 to fill in rowkey column
> bytes.
> While, in class MergeCuboidMapper it's initialized with length 255.
> rowKeySplitter = new RowKeySplitter(sourceCubeSegment, 65, 255);
> So, if a dimension is encoded in fixed length and the max length is set to
> 256. The cube building job will succeed. While, the merge job will always
> fail. Since in class MergeCuboidMapper method doMap:
> public void doMap(Text key, Text value, Context context) throws
> IOException, InterruptedException {
> long cuboidID = rowKeySplitter.split(key.getBytes());
> Cuboid cuboid = Cuboid.findForMandatory(cubeDesc, cuboidID);
> in method doMap, it will invoke method RowKeySplitter.split(byte[] bytes):
> for (int i = 0; i < cuboid.getColumns().size(); i++) {
> splitOffsets[i] = offset;
> TblColRef col = cuboid.getColumns().get(i);
> int colLength = colIO.getColumnLength(col);
> SplittedBytes split = this.splitBuffers[this.bufferSize++];
> split.length = colLength;
> System.arraycopy(bytes, offset, split.value, 0, colLength);
> offset += colLength;
> }
> Method System.arraycopy will result in IndexOutOfBoundsException exception,
> if a column value length is 256 in bytes and is being copied to a bytes array
> with length 255.
> The incompatibility is also occurred in class
> FilterRecommendCuboidDataMapper, initialize RowkeySplitter as:
> rowKeySplitter = new RowKeySplitter(originalSegment, 65, 255);
> I think the better way is to always set the max split length as 256.
> And actually dimension encoded in fix length 256 is pretty common in our
> production. Since in Hive, type varchar(256) is pretty common, users do have
> not much Kylin knowledge will prefer to chose fix length encoding on such
> dimensions, and set max length as 256.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3115) Incompatible RowKeySplitter initialize between build and merge job
[
https://issues.apache.org/jira/browse/KYLIN-3115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507695#comment-16507695
]
ASF subversion and git services commented on KYLIN-3115:
Commit f6b1dfb5ef3239ea252b1498bf4c51235361bbcd in kylin's branch
refs/heads/master from shaofengshi
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=f6b1dfb ]
KYLIN-3115 Incompatible RowKeySplitter initialize between build and merge job
> Incompatible RowKeySplitter initialize between build and merge job
> --
>
> Key: KYLIN-3115
> URL: https://issues.apache.org/jira/browse/KYLIN-3115
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Reporter: Wang, Gang
>Assignee: Shaofeng SHI
>Priority: Minor
> Fix For: v2.4.0
>
>
> In class NDCuboidBuilder:
> public NDCuboidBuilder(CubeSegment cubeSegment) {
> this.cubeSegment = cubeSegment;
> this.rowKeySplitter = new RowKeySplitter(cubeSegment, 65, 256);
> this.rowKeyEncoderProvider = new RowKeyEncoderProvider(cubeSegment);
> }
> which will create a bytes array with length 256 to fill in rowkey column
> bytes.
> While, in class MergeCuboidMapper it's initialized with length 255.
> rowKeySplitter = new RowKeySplitter(sourceCubeSegment, 65, 255);
> So, if a dimension is encoded in fixed length and the max length is set to
> 256. The cube building job will succeed. While, the merge job will always
> fail. Since in class MergeCuboidMapper method doMap:
> public void doMap(Text key, Text value, Context context) throws
> IOException, InterruptedException {
> long cuboidID = rowKeySplitter.split(key.getBytes());
> Cuboid cuboid = Cuboid.findForMandatory(cubeDesc, cuboidID);
> in method doMap, it will invoke method RowKeySplitter.split(byte[] bytes):
> for (int i = 0; i < cuboid.getColumns().size(); i++) {
> splitOffsets[i] = offset;
> TblColRef col = cuboid.getColumns().get(i);
> int colLength = colIO.getColumnLength(col);
> SplittedBytes split = this.splitBuffers[this.bufferSize++];
> split.length = colLength;
> System.arraycopy(bytes, offset, split.value, 0, colLength);
> offset += colLength;
> }
> Method System.arraycopy will result in IndexOutOfBoundsException exception,
> if a column value length is 256 in bytes and is being copied to a bytes array
> with length 255.
> The incompatibility is also occurred in class
> FilterRecommendCuboidDataMapper, initialize RowkeySplitter as:
> rowKeySplitter = new RowKeySplitter(originalSegment, 65, 255);
> I think the better way is to always set the max split length as 256.
> And actually dimension encoded in fix length 256 is pretty common in our
> production. Since in Hive, type varchar(256) is pretty common, users do have
> not much Kylin knowledge will prefer to chose fix length encoding on such
> dimensions, and set max length as 256.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3388) Data may become not correct if mappers fail during the redistribute step, "distribute by rand()"
[ https://issues.apache.org/jira/browse/KYLIN-3388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-3388: Summary: Data may become not correct if mappers fail during the redistribute step, "distribute by rand()" (was: Data may become not correct if mappers fail during the cube building step, "distribute by rand()") > Data may become not correct if mappers fail during the redistribute step, > "distribute by rand()" > > > Key: KYLIN-3388 > URL: https://issues.apache.org/jira/browse/KYLIN-3388 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Reporter: Zhong Yanghong >Priority: Critical > Fix For: v2.4.0 > > Attachments: Hive Issue - distribute by rand().png > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3388) Data may become not correct if mappers fail during the cube building step, "distribute by rand()"
[ https://issues.apache.org/jira/browse/KYLIN-3388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507237#comment-16507237 ] Shaofeng SHI edited comment on KYLIN-3388 at 6/11/18 1:06 AM: -- We need to disable redistribute by rand by default. If there is a UHC can be used to distribute, that be fine. [~yaho] Have you started on this? was (Author: shaofengshi): We need to disable redistribute by default. > Data may become not correct if mappers fail during the cube building step, > "distribute by rand()" > - > > Key: KYLIN-3388 > URL: https://issues.apache.org/jira/browse/KYLIN-3388 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Reporter: Zhong Yanghong >Priority: Critical > Fix For: v2.4.0 > > Attachments: Hive Issue - distribute by rand().png > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3137) Spark cubing without hive-site.xml
[
https://issues.apache.org/jira/browse/KYLIN-3137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507545#comment-16507545
]
ASF subversion and git services commented on KYLIN-3137:
Commit 995a7198e6d78f0b21bac108a368306ca8cc48fa in kylin's branch
refs/heads/KYLIN-3115 from shaofengshi
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=995a719 ]
KYLIN-3137 allowing config Spark storage level
> Spark cubing without hive-site.xml
> --
>
> Key: KYLIN-3137
> URL: https://issues.apache.org/jira/browse/KYLIN-3137
> Project: Kylin
> Issue Type: Improvement
> Components: Job Engine, Others, Spark Engine
>Affects Versions: v2.2.0
>Reporter: Ruslan Dautkhanov
>Assignee: Shaofeng SHI
>Priority: Major
> Labels: cdh, cloudera, configuration, hive
> Fix For: v2.4.0
>
>
> Getting following exception while trying to build a cube
> {noformat}
> java.lang.RuntimeException: Cannot find hive-site.xml in
> kylin_hadoop_conf_dir: /etc/hadoop/conf. In order to enable spark cubing, you
> must set kylin.env.hadoop-conf-dir to a dir which contains at least
> core-site.xml, hdfs-site.xml, hive-site.xml, mapred-site.xml, yarn-site.xml
> at
> org.apache.kylin.engine.spark.SparkExecutable.doWork(SparkExecutable.java:117)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:64)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:144)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> I am using Kylin binaries for CDH downloaded from kylin.apache.org.
> Yes, indeed hive-site.xml is not in /etc/hadoop/conf in Cloudera's
> distribution for Hadoop.
> hive-site.xml is in /etc/hive/conf, not in /etc/hadoop/conf
> The other four files:
> core-site.xml, hdfs-site.xml, mapred-site.xml, yarn-site.xml
> can be found in /etc/hadoop/conf but, again, not hive-site.xml which is in
> /etc/hive/conf .
> Would be great to have this adjusted for CDH.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3093) Upgrade curator to 2.12
[ https://issues.apache.org/jira/browse/KYLIN-3093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507548#comment-16507548 ] ASF subversion and git services commented on KYLIN-3093: Commit 38b47c7fce7159c62a59865db0edaa2554238d44 in kylin's branch refs/heads/KYLIN-3115 from shaofengshi [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=38b47c7 ] KYLIN-3093 Upgrade curator to 2.12 > Upgrade curator to 2.12 > --- > > Key: KYLIN-3093 > URL: https://issues.apache.org/jira/browse/KYLIN-3093 > Project: Kylin > Issue Type: Improvement > Components: Tools, Build and Test >Reporter: Ted Yu >Assignee: Shaofeng SHI >Priority: Major > Fix For: v2.4.0, v2.3.2 > > > curator-2.10.0 has several bug fixes over current version (2.7.1), updating > would help improve stability. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3398) Inaccurate arithmetic operation in LookupTableToHFileJob#calculateShardNum
[
https://issues.apache.org/jira/browse/KYLIN-3398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507549#comment-16507549
]
ASF subversion and git services commented on KYLIN-3398:
Commit ff122ebdde3a2486bede103c589e0aa4149e1658 in kylin's branch
refs/heads/KYLIN-3115 from [~nichunen]
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=ff122eb ]
KYLIN-3398 Fix inaccurate arithmetic operations in
LookupTableToHFileJob#calculateShardNum
> Inaccurate arithmetic operation in LookupTableToHFileJob#calculateShardNum
> --
>
> Key: KYLIN-3398
> URL: https://issues.apache.org/jira/browse/KYLIN-3398
> Project: Kylin
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: jiatao.tao
>Priority: Major
>
> There're two issues with the following code:
> {code}
> private int calculateShardNum(KylinConfig kylinConfig, long dataSize) {
> long shardSize = kylinConfig.getExtTableSnapshotShardingMB() * 1024 *
> 1024;
> return dataSize < shardSize ? 1 : (int) (Math.ceil(dataSize /
> shardSize));
> {code}
> getExtTableSnapshotShardingMB returns an int. The multiplication is done
> using 32-bit arithmetic, and then used in a context that expects an
> expression of type "long".
> Second, Math.ceil expects a double. The integer division would lose some
> precision.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3401) The current using zip compress tool has an arbitrary file write vulnerability
[ https://issues.apache.org/jira/browse/KYLIN-3401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507546#comment-16507546 ] ASF subversion and git services commented on KYLIN-3401: Commit 372d1bb08a11cea918ea10ec9a7de79a050267e0 in kylin's branch refs/heads/KYLIN-3115 from [~Wayne0101] [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=372d1bb ] KYLIN-3401 change zip compress tool from java.util.zip to Apache commons-compress library > The current using zip compress tool has an arbitrary file write vulnerability > - > > Key: KYLIN-3401 > URL: https://issues.apache.org/jira/browse/KYLIN-3401 > Project: Kylin > Issue Type: Bug >Reporter: Chao Long >Assignee: Chao Long >Priority: Major > Fix For: v2.4.0, v2.3.2 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3094) Upgrade zookeeper to 3.4.12
[ https://issues.apache.org/jira/browse/KYLIN-3094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507547#comment-16507547 ] ASF subversion and git services commented on KYLIN-3094: Commit 93247fa7f23ac427e8b6a581b0f902366bb9804f in kylin's branch refs/heads/KYLIN-3115 from [~nichunen] [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=93247fa ] KYLIN-3094 Upgrade zookeeper version from 3.4.8 to 3.4.12 > Upgrade zookeeper to 3.4.12 > --- > > Key: KYLIN-3094 > URL: https://issues.apache.org/jira/browse/KYLIN-3094 > Project: Kylin > Issue Type: Improvement >Reporter: Ted Yu >Assignee: nichunen >Priority: Minor > Fix For: v2.4.0, v2.3.2 > > > Current zookeeper release is 3.4.12 > We should upgrade dependency from 3.4.8 to 3.4.12 where there is important > security fix. > One such critical fix is ZOOKEEPER-2146, which can be explored maliciously -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3093) Upgrade curator to 2.12
[ https://issues.apache.org/jira/browse/KYLIN-3093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507550#comment-16507550 ] ASF subversion and git services commented on KYLIN-3093: Commit de17b6ea0f53a4a4819ada47b81589b405e9e9e5 in kylin's branch refs/heads/KYLIN-3115 from shaofengshi [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=de17b6e ] KYLIN-3093 add curator-client > Upgrade curator to 2.12 > --- > > Key: KYLIN-3093 > URL: https://issues.apache.org/jira/browse/KYLIN-3093 > Project: Kylin > Issue Type: Improvement > Components: Tools, Build and Test >Reporter: Ted Yu >Assignee: Shaofeng SHI >Priority: Major > Fix For: v2.4.0, v2.3.2 > > > curator-2.10.0 has several bug fixes over current version (2.7.1), updating > would help improve stability. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3115) Incompatible RowKeySplitter initialize between build and merge job
[
https://issues.apache.org/jira/browse/KYLIN-3115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507551#comment-16507551
]
ASF subversion and git services commented on KYLIN-3115:
Commit 4df26568ecc4f255da3cd9cb503ab5f479538030 in kylin's branch
refs/heads/KYLIN-3115 from shaofengshi
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=4df2656 ]
KYLIN-3115 Incompatible RowKeySplitter initialize between build and merge job
> Incompatible RowKeySplitter initialize between build and merge job
> --
>
> Key: KYLIN-3115
> URL: https://issues.apache.org/jira/browse/KYLIN-3115
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Reporter: Wang, Gang
>Assignee: Shaofeng SHI
>Priority: Minor
> Fix For: v2.4.0
>
>
> In class NDCuboidBuilder:
> public NDCuboidBuilder(CubeSegment cubeSegment) {
> this.cubeSegment = cubeSegment;
> this.rowKeySplitter = new RowKeySplitter(cubeSegment, 65, 256);
> this.rowKeyEncoderProvider = new RowKeyEncoderProvider(cubeSegment);
> }
> which will create a bytes array with length 256 to fill in rowkey column
> bytes.
> While, in class MergeCuboidMapper it's initialized with length 255.
> rowKeySplitter = new RowKeySplitter(sourceCubeSegment, 65, 255);
> So, if a dimension is encoded in fixed length and the max length is set to
> 256. The cube building job will succeed. While, the merge job will always
> fail. Since in class MergeCuboidMapper method doMap:
> public void doMap(Text key, Text value, Context context) throws
> IOException, InterruptedException {
> long cuboidID = rowKeySplitter.split(key.getBytes());
> Cuboid cuboid = Cuboid.findForMandatory(cubeDesc, cuboidID);
> in method doMap, it will invoke method RowKeySplitter.split(byte[] bytes):
> for (int i = 0; i < cuboid.getColumns().size(); i++) {
> splitOffsets[i] = offset;
> TblColRef col = cuboid.getColumns().get(i);
> int colLength = colIO.getColumnLength(col);
> SplittedBytes split = this.splitBuffers[this.bufferSize++];
> split.length = colLength;
> System.arraycopy(bytes, offset, split.value, 0, colLength);
> offset += colLength;
> }
> Method System.arraycopy will result in IndexOutOfBoundsException exception,
> if a column value length is 256 in bytes and is being copied to a bytes array
> with length 255.
> The incompatibility is also occurred in class
> FilterRecommendCuboidDataMapper, initialize RowkeySplitter as:
> rowKeySplitter = new RowKeySplitter(originalSegment, 65, 255);
> I think the better way is to always set the max split length as 256.
> And actually dimension encoded in fix length 256 is pretty common in our
> production. Since in Hive, type varchar(256) is pretty common, users do have
> not much Kylin knowledge will prefer to chose fix length encoding on such
> dimensions, and set max length as 256.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3378) Support Kafka table join with Hive tables
[ https://issues.apache.org/jira/browse/KYLIN-3378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507544#comment-16507544 ] ASF subversion and git services commented on KYLIN-3378: Commit f85d561293cd2cfd36a1674978c719c86cbbee9b in kylin's branch refs/heads/KYLIN-3115 from shaofengshi [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=f85d561 ] KYLIN-3378 Support Kafka table join with Hive tables > Support Kafka table join with Hive tables > - > > Key: KYLIN-3378 > URL: https://issues.apache.org/jira/browse/KYLIN-3378 > Project: Kylin > Issue Type: Improvement > Components: Streaming >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI >Priority: Major > Fix For: v2.4.0 > > > At this moment, if the data source is Kafka, only 1 table allowed in the data > model. In some cases, joining the kafka stream with hive lookup tables is > expected -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3282) hbase timeout cause the endless status.
[
https://issues.apache.org/jira/browse/KYLIN-3282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507411#comment-16507411
]
Xingxing Di edited comment on KYLIN-3282 at 6/10/18 3:10 PM:
-
Is anyone working on this? we had same issue on kylin2.2.0, the same error
occurred while after "Build Base Cuboid" step succeed ， the error cause the job
disappeared in " Monitor view" , and the segment can't refresh for "Segments
overlaps".
Now this cube just stucked in this error, i don't see a regular way to resolve
this situation. In this case I can delete the segment and rebuild , but i think
this error may occur in any hbase persistance cases, so there should be a
regular way to deal with hbase persist timeout situation.
was (Author: dixingx...@yeah.net):
Is anyone working on this? we had same issue on kylin2.2.0, the same error
occurred while after "Build Base Cuboid" step succeed， this error cause the job
disappeared in " Monitor view" , and the segment can't refresh for "Segments
overlaps".
Now this cube just stucked in the error, i don't see a regular way to resolve
this situation.
> hbase timeout cause the endless status.
> ---
>
> Key: KYLIN-3282
> URL: https://issues.apache.org/jira/browse/KYLIN-3282
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: v2.3.0
>Reporter: readme_kylin
>Priority: Major
>
> ri Mar 09 12:52:07 GMT+08:00 2018,
> RpcRetryingCaller\{globalStartTime=1520571112216, pause=100, retries=1},
> java.io.IOException: Call to QZ140/10.0.0.140:16020 failed on local
> exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=8030361,
> waitTime=15002, operationTimeout=15000 expired.
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:157)
> at org.apache.hadoop.hbase.client.HTable.checkAndPut(HTable.java:1233)
> at
> org.apache.kylin.storage.hbase.HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:311)
> at
> org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:305)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:291)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:271)
> at
> org.apache.kylin.job.dao.ExecutableDao.writeJobOutputResource(ExecutableDao.java:88)
> at
> org.apache.kylin.job.dao.ExecutableDao.updateJobOutput(ExecutableDao.java:216)
> at
> org.apache.kylin.job.execution.ExecutableManager.addJobInfo(ExecutableManager.java:480)
> at
> org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:161)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:162)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:67)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:162)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:300)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>
>
> 2018-03-09 12:52:10,191 ERROR [Scheduler 9772827 Job
> 499477a7-4c1a-4c5a-8d4a-0b3218a58dca-516] execution.AbstractExecutable:100 :
> 1th retries for onExecuteFinished fails due to {}
> java.lang.IllegalStateException: Overwriting conflict
> /execute_output/499477a7-4c1a-4c5a-8d4a-0b3218a58dca-13, expect old TS
> 1520571099067, but it is 1520571112216
> at
> org.apache.kylin.storage.hbase.HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:316)
> at
> org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:305)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:291)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:271)
> at
> org.apache.kylin.job.dao.ExecutableDao.writeJobOutputResource(ExecutableDao.java:88)
> at
> org.apache.kylin.job.dao.ExecutableDao.updateJobOutput(ExecutableDao.java:216)
> at
> org.apache.kylin.job.execution.ExecutableManager.addJobInfo(ExecutableManager.java:480)
> at
> org.apache.kylin.job.execution.ExecutableManager.addJobInfo(ExecutableManager.java:490)
> at
> org.apache.kylin.job.execution.AbstractExecutable.addExtraInfo(AbstractExecutable.java:403)
> at
> org.apache.kylin.job.execution.AbstractExecutable.setEndTime(AbstractExecutable.java:415)
> at
> org.apache.kylin.job.execution.AbstractExecutable.onExecuteFinished(AbstractExecutable.java:121)
> at
>
[jira] [Commented] (KYLIN-3282) hbase timeout cause the endless status.
[
https://issues.apache.org/jira/browse/KYLIN-3282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507411#comment-16507411
]
Xingxing Di commented on KYLIN-3282:
Is anyone working on this? we had same issue on kylin2.2.0, the same error
occurred while after "Build Base Cuboid" step succeed， this error cause the job
disappeared in " Monitor view" , and the segment can't refresh for "Segments
overlaps".
Now this cube just stucked in the error, i don't see a regular way to resolve
this situation.
> hbase timeout cause the endless status.
> ---
>
> Key: KYLIN-3282
> URL: https://issues.apache.org/jira/browse/KYLIN-3282
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: v2.3.0
>Reporter: readme_kylin
>Priority: Major
>
> ri Mar 09 12:52:07 GMT+08:00 2018,
> RpcRetryingCaller\{globalStartTime=1520571112216, pause=100, retries=1},
> java.io.IOException: Call to QZ140/10.0.0.140:16020 failed on local
> exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=8030361,
> waitTime=15002, operationTimeout=15000 expired.
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:157)
> at org.apache.hadoop.hbase.client.HTable.checkAndPut(HTable.java:1233)
> at
> org.apache.kylin.storage.hbase.HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:311)
> at
> org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:305)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:291)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:271)
> at
> org.apache.kylin.job.dao.ExecutableDao.writeJobOutputResource(ExecutableDao.java:88)
> at
> org.apache.kylin.job.dao.ExecutableDao.updateJobOutput(ExecutableDao.java:216)
> at
> org.apache.kylin.job.execution.ExecutableManager.addJobInfo(ExecutableManager.java:480)
> at
> org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:161)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:162)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:67)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:162)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:300)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>
>
> 2018-03-09 12:52:10,191 ERROR [Scheduler 9772827 Job
> 499477a7-4c1a-4c5a-8d4a-0b3218a58dca-516] execution.AbstractExecutable:100 :
> 1th retries for onExecuteFinished fails due to {}
> java.lang.IllegalStateException: Overwriting conflict
> /execute_output/499477a7-4c1a-4c5a-8d4a-0b3218a58dca-13, expect old TS
> 1520571099067, but it is 1520571112216
> at
> org.apache.kylin.storage.hbase.HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:316)
> at
> org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:305)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:291)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:271)
> at
> org.apache.kylin.job.dao.ExecutableDao.writeJobOutputResource(ExecutableDao.java:88)
> at
> org.apache.kylin.job.dao.ExecutableDao.updateJobOutput(ExecutableDao.java:216)
> at
> org.apache.kylin.job.execution.ExecutableManager.addJobInfo(ExecutableManager.java:480)
> at
> org.apache.kylin.job.execution.ExecutableManager.addJobInfo(ExecutableManager.java:490)
> at
> org.apache.kylin.job.execution.AbstractExecutable.addExtraInfo(AbstractExecutable.java:403)
> at
> org.apache.kylin.job.execution.AbstractExecutable.setEndTime(AbstractExecutable.java:415)
> at
> org.apache.kylin.job.execution.AbstractExecutable.onExecuteFinished(AbstractExecutable.java:121)
> at
> org.apache.kylin.job.execution.AbstractExecutable.onExecuteFinishedWithRetry(AbstractExecutable.java:98)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:175)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:67)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:162)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:300)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
>
[jira] [Commented] (KYLIN-3093) Upgrade curator to 2.12
[ https://issues.apache.org/jira/browse/KYLIN-3093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507398#comment-16507398 ] ASF subversion and git services commented on KYLIN-3093: Commit de17b6ea0f53a4a4819ada47b81589b405e9e9e5 in kylin's branch refs/heads/master from shaofengshi [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=de17b6e ] KYLIN-3093 add curator-client > Upgrade curator to 2.12 > --- > > Key: KYLIN-3093 > URL: https://issues.apache.org/jira/browse/KYLIN-3093 > Project: Kylin > Issue Type: Improvement > Components: Tools, Build and Test >Reporter: Ted Yu >Assignee: Shaofeng SHI >Priority: Major > Fix For: v2.4.0, v2.3.2 > > > curator-2.10.0 has several bug fixes over current version (2.7.1), updating > would help improve stability. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3399) Leaked lookup table in DictionaryGeneratorCLI#processSegment
[
https://issues.apache.org/jira/browse/KYLIN-3399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507381#comment-16507381
]
Ted Yu commented on KYLIN-3399:
---
Looking at getLookupTable :
{code}
public ILookupTable getLookupTable(CubeSegment cubeSegment, JoinDesc join) {
{code}
There is no declared exception thrown from this method.
It is not clear what type of exception is expected and whether the caller is
able to handle that.
> Leaked lookup table in DictionaryGeneratorCLI#processSegment
>
>
> Key: KYLIN-3399
> URL: https://issues.apache.org/jira/browse/KYLIN-3399
> Project: Kylin
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: jiatao.tao
>Priority: Major
>
> {code}
> for (TableRef lookup : toCheckLookup) {
> logger.info("Checking snapshot of " + lookup);
> JoinDesc join =
> cubeSeg.getModel().getJoinsTree().getJoinByPKSide(lookup);
> cubeMgr.getLookupTable(cubeSeg, join);
> {code}
> The lookup table from the last line doesn't seem to be used.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3398) Inaccurate arithmetic operation in LookupTableToHFileJob#calculateShardNum
[
https://issues.apache.org/jira/browse/KYLIN-3398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507288#comment-16507288
]
ASF GitHub Bot commented on KYLIN-3398:
---
shaofengshi closed pull request #152: KYLIN-3398 Fix inaccurate arithmetic
operations
URL: https://github.com/apache/kylin/pull/152
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/lookup/LookupTableToHFileJob.java
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/lookup/LookupTableToHFileJob.java
index 39ebe996a6..054e1466b6 100644
---
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/lookup/LookupTableToHFileJob.java
+++
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/lookup/LookupTableToHFileJob.java
@@ -233,8 +233,8 @@ private void
removeSnapshotIfExist(ExtTableSnapshotInfoManager extSnapshotInfoMa
}
private int calculateShardNum(KylinConfig kylinConfig, long dataSize) {
-long shardSize = kylinConfig.getExtTableSnapshotShardingMB() * 1024 *
1024;
-return dataSize < shardSize ? 1 : (int) (Math.ceil(dataSize /
shardSize));
+long shardSize = (long) (kylinConfig.getExtTableSnapshotShardingMB())
* 1024 * 1024;
+return dataSize < shardSize ? 1 : (int) (Math.ceil((double) dataSize /
shardSize));
}
private byte[][] getSplitsByShardNum(int shardNum) {
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Inaccurate arithmetic operation in LookupTableToHFileJob#calculateShardNum
> --
>
> Key: KYLIN-3398
> URL: https://issues.apache.org/jira/browse/KYLIN-3398
> Project: Kylin
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: jiatao.tao
>Priority: Major
>
> There're two issues with the following code:
> {code}
> private int calculateShardNum(KylinConfig kylinConfig, long dataSize) {
> long shardSize = kylinConfig.getExtTableSnapshotShardingMB() * 1024 *
> 1024;
> return dataSize < shardSize ? 1 : (int) (Math.ceil(dataSize /
> shardSize));
> {code}
> getExtTableSnapshotShardingMB returns an int. The multiplication is done
> using 32-bit arithmetic, and then used in a context that expects an
> expression of type "long".
> Second, Math.ceil expects a double. The integer division would lose some
> precision.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3398) Inaccurate arithmetic operation in LookupTableToHFileJob#calculateShardNum
[
https://issues.apache.org/jira/browse/KYLIN-3398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507289#comment-16507289
]
ASF subversion and git services commented on KYLIN-3398:
Commit ff122ebdde3a2486bede103c589e0aa4149e1658 in kylin's branch
refs/heads/master from [~nichunen]
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=ff122eb ]
KYLIN-3398 Fix inaccurate arithmetic operations in
LookupTableToHFileJob#calculateShardNum
> Inaccurate arithmetic operation in LookupTableToHFileJob#calculateShardNum
> --
>
> Key: KYLIN-3398
> URL: https://issues.apache.org/jira/browse/KYLIN-3398
> Project: Kylin
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: jiatao.tao
>Priority: Major
>
> There're two issues with the following code:
> {code}
> private int calculateShardNum(KylinConfig kylinConfig, long dataSize) {
> long shardSize = kylinConfig.getExtTableSnapshotShardingMB() * 1024 *
> 1024;
> return dataSize < shardSize ? 1 : (int) (Math.ceil(dataSize /
> shardSize));
> {code}
> getExtTableSnapshotShardingMB returns an int. The multiplication is done
> using 32-bit arithmetic, and then used in a context that expects an
> expression of type "long".
> Second, Math.ceil expects a double. The integer division would lose some
> precision.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3093) Upgrade curator to 2.12
[ https://issues.apache.org/jira/browse/KYLIN-3093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507287#comment-16507287 ] ASF subversion and git services commented on KYLIN-3093: Commit 2e212ae81116ab6d0f99e64cf3849cd8af76b713 in kylin's branch refs/heads/2.3.x from shaofengshi [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=2e212ae ] KYLIN-3093 Upgrade curator to 2.12 > Upgrade curator to 2.12 > --- > > Key: KYLIN-3093 > URL: https://issues.apache.org/jira/browse/KYLIN-3093 > Project: Kylin > Issue Type: Improvement > Components: Tools, Build and Test >Reporter: Ted Yu >Assignee: Shaofeng SHI >Priority: Major > Fix For: v2.4.0, v2.3.2 > > > curator-2.10.0 has several bug fixes over current version (2.7.1), updating > would help improve stability. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3093) Upgrade curator to 2.12
[ https://issues.apache.org/jira/browse/KYLIN-3093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI resolved KYLIN-3093. - Resolution: Fixed Updated the version. While as the scope is "provided", if the hadoop doesn't upgrade, Kylin still uses the old version. > Upgrade curator to 2.12 > --- > > Key: KYLIN-3093 > URL: https://issues.apache.org/jira/browse/KYLIN-3093 > Project: Kylin > Issue Type: Improvement > Components: Tools, Build and Test >Reporter: Ted Yu >Assignee: Shaofeng SHI >Priority: Major > Fix For: v2.4.0, v2.3.2 > > > curator-2.10.0 has several bug fixes over current version (2.7.1), updating > would help improve stability. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KYLIN-3093) Upgrade curator to 2.12
[ https://issues.apache.org/jira/browse/KYLIN-3093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI reassigned KYLIN-3093: --- Assignee: Shaofeng SHI (was: jiatao.tao) > Upgrade curator to 2.12 > --- > > Key: KYLIN-3093 > URL: https://issues.apache.org/jira/browse/KYLIN-3093 > Project: Kylin > Issue Type: Improvement > Components: Tools, Build and Test >Reporter: Ted Yu >Assignee: Shaofeng SHI >Priority: Major > Fix For: v2.4.0, v2.3.2 > > > curator-2.10.0 has several bug fixes over current version (2.7.1), updating > would help improve stability. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3093) Upgrade curator to 2.12
[ https://issues.apache.org/jira/browse/KYLIN-3093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-3093: Fix Version/s: v2.3.2 v2.4.0 > Upgrade curator to 2.12 > --- > > Key: KYLIN-3093 > URL: https://issues.apache.org/jira/browse/KYLIN-3093 > Project: Kylin > Issue Type: Improvement > Components: Tools, Build and Test >Reporter: Ted Yu >Assignee: Shaofeng SHI >Priority: Major > Fix For: v2.4.0, v2.3.2 > > > curator-2.10.0 has several bug fixes over current version (2.7.1), updating > would help improve stability. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3093) Upgrade curator to 2.12
[ https://issues.apache.org/jira/browse/KYLIN-3093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507285#comment-16507285 ] ASF subversion and git services commented on KYLIN-3093: Commit 38b47c7fce7159c62a59865db0edaa2554238d44 in kylin's branch refs/heads/master from shaofengshi [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=38b47c7 ] KYLIN-3093 Upgrade curator to 2.12 > Upgrade curator to 2.12 > --- > > Key: KYLIN-3093 > URL: https://issues.apache.org/jira/browse/KYLIN-3093 > Project: Kylin > Issue Type: Improvement > Components: Tools, Build and Test >Reporter: Ted Yu >Assignee: Shaofeng SHI >Priority: Major > Fix For: v2.4.0, v2.3.2 > > > curator-2.10.0 has several bug fixes over current version (2.7.1), updating > would help improve stability. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3398) Inaccurate arithmetic operation in LookupTableToHFileJob#calculateShardNum
[
https://issues.apache.org/jira/browse/KYLIN-3398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507279#comment-16507279
]
ASF GitHub Bot commented on KYLIN-3398:
---
coveralls commented on issue #152: KYLIN-3398 Fix inaccurate arithmetic
operations
URL: https://github.com/apache/kylin/pull/152#issuecomment-396025219
## Pull Request Test Coverage Report for [Build
3215](https://coveralls.io/builds/17411321)
* **0** of **2** **(0.0%)** changed or added relevant lines in **1** file
are covered.
* **1** unchanged line in **1** file lost coverage.
* Overall coverage remained the same at **24.172%**
---
| Changes Missing Coverage | Covered Lines | Changed/Added Lines | % |
| :-|--||---: |
|
[storage-hbase/src/main/java/org/apache/kylin/storage/hbase/lookup/LookupTableToHFileJob.java](https://coveralls.io/builds/17411321/source?filename=storage-hbase%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fstorage%2Fhbase%2Flookup%2FLookupTableToHFileJob.java#L236)
| 0 | 2 | 0.0%
| Files with Coverage Reduction | New Missed Lines | % |
| :-|--|--: |
|
[core-cube/src/main/java/org/apache/kylin/cube/inmemcubing/MemDiskStore.java](https://coveralls.io/builds/17411321/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2Finmemcubing%2FMemDiskStore.java#L553)
| 1 | 78.12% |
| Totals | [](https://coveralls.io/builds/17411321)
|
| :-- | --: |
| Change from base [Build 3213](https://coveralls.io/builds/17411310): |
0.0% |
| Covered Lines: | 14795 |
| Relevant Lines: | 61207 |
---
# - [Coveralls](https://coveralls.io)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Inaccurate arithmetic operation in LookupTableToHFileJob#calculateShardNum
> --
>
> Key: KYLIN-3398
> URL: https://issues.apache.org/jira/browse/KYLIN-3398
> Project: Kylin
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: jiatao.tao
>Priority: Major
>
> There're two issues with the following code:
> {code}
> private int calculateShardNum(KylinConfig kylinConfig, long dataSize) {
> long shardSize = kylinConfig.getExtTableSnapshotShardingMB() * 1024 *
> 1024;
> return dataSize < shardSize ? 1 : (int) (Math.ceil(dataSize /
> shardSize));
> {code}
> getExtTableSnapshotShardingMB returns an int. The multiplication is done
> using 32-bit arithmetic, and then used in a context that expects an
> expression of type "long".
> Second, Math.ceil expects a double. The integer division would lose some
> precision.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3398) Inaccurate arithmetic operation in LookupTableToHFileJob#calculateShardNum
[
https://issues.apache.org/jira/browse/KYLIN-3398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507278#comment-16507278
]
ASF GitHub Bot commented on KYLIN-3398:
---
codecov-io commented on issue #152: KYLIN-3398 Fix inaccurate arithmetic
operations
URL: https://github.com/apache/kylin/pull/152#issuecomment-396025190
# [Codecov](https://codecov.io/gh/apache/kylin/pull/152?src=pr=h1) Report
> Merging [#152](https://codecov.io/gh/apache/kylin/pull/152?src=pr=desc)
into
[master](https://codecov.io/gh/apache/kylin/commit/93247fa7f23ac427e8b6a581b0f902366bb9804f?src=pr=desc)
will **increase** coverage by `<.01%`.
> The diff coverage is `0%`.
[](https://codecov.io/gh/apache/kylin/pull/152?src=pr=tree)
```diff
@@ Coverage Diff @@
## master #152 +/- ##
+ Coverage 22.14% 22.14% +<.01%
- Complexity 4019 4020 +1
Files 1012 1012
Lines 6120761207
Branches 8768 8768
+ Hits 1355213553 +1
Misses4641446414
+ Partials 1241 1240 -1
```
| [Impacted
Files](https://codecov.io/gh/apache/kylin/pull/152?src=pr=tree) | Coverage Δ
| Complexity Δ | |
|---|---|---|---|
|
[...in/storage/hbase/lookup/LookupTableToHFileJob.java](https://codecov.io/gh/apache/kylin/pull/152/diff?src=pr=tree#diff-c3RvcmFnZS1oYmFzZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUva3lsaW4vc3RvcmFnZS9oYmFzZS9sb29rdXAvTG9va3VwVGFibGVUb0hGaWxlSm9iLmphdmE=)
| `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: |
|
[...rg/apache/kylin/cube/inmemcubing/MemDiskStore.java](https://codecov.io/gh/apache/kylin/pull/152/diff?src=pr=tree#diff-Y29yZS1jdWJlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9jdWJlL2lubWVtY3ViaW5nL01lbURpc2tTdG9yZS5qYXZh)
| `69.6% <0%> (-0.61%)` | `7% <0%> (ø)` | |
|
[...a/org/apache/kylin/dict/Number2BytesConverter.java](https://codecov.io/gh/apache/kylin/pull/152/diff?src=pr=tree#diff-Y29yZS1kaWN0aW9uYXJ5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9kaWN0L051bWJlcjJCeXRlc0NvbnZlcnRlci5qYXZh)
| `82.53% <0%> (+0.79%)` | `18% <0%> (+1%)` | :arrow_up: |
|
[...g/apache/kylin/source/datagen/ColumnGenerator.java](https://codecov.io/gh/apache/kylin/pull/152/diff?src=pr=tree#diff-Y29yZS1tZXRhZGF0YS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUva3lsaW4vc291cmNlL2RhdGFnZW4vQ29sdW1uR2VuZXJhdG9yLmphdmE=)
| `72.29% <0%> (+1.35%)` | `8% <0%> (ø)` | :arrow_down: |
--
[Continue to review full report at
Codecov](https://codecov.io/gh/apache/kylin/pull/152?src=pr=continue).
> **Legend** - [Click here to learn
more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute (impact)`, `ø = not affected`, `? = missing data`
> Powered by
[Codecov](https://codecov.io/gh/apache/kylin/pull/152?src=pr=footer). Last
update
[93247fa...d50873b](https://codecov.io/gh/apache/kylin/pull/152?src=pr=lastupdated).
Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Inaccurate arithmetic operation in LookupTableToHFileJob#calculateShardNum
> --
>
> Key: KYLIN-3398
> URL: https://issues.apache.org/jira/browse/KYLIN-3398
> Project: Kylin
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: jiatao.tao
>Priority: Major
>
> There're two issues with the following code:
> {code}
> private int calculateShardNum(KylinConfig kylinConfig, long dataSize) {
> long shardSize = kylinConfig.getExtTableSnapshotShardingMB() * 1024 *
> 1024;
> return dataSize < shardSize ? 1 : (int) (Math.ceil(dataSize /
> shardSize));
> {code}
> getExtTableSnapshotShardingMB returns an int. The multiplication is done
> using 32-bit arithmetic, and then used in a context that expects an
> expression of type "long".
> Second, Math.ceil expects a double. The integer division would lose some
> precision.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3363) Wrong partition condition appended in JDBC Source
[ https://issues.apache.org/jira/browse/KYLIN-3363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-3363: Fix Version/s: v2.3.2 > Wrong partition condition appended in JDBC Source > - > > Key: KYLIN-3363 > URL: https://issues.apache.org/jira/browse/KYLIN-3363 > Project: Kylin > Issue Type: Bug > Components: RDBMS Source >Affects Versions: v2.3.1 >Reporter: Dong Li >Assignee: Dong Li >Priority: Minor > Fix For: v2.4.0, v2.3.2 > > > create a cube, and select the partition column from lookup table, then select > shard by column from fact table. > then an unexecutable sql will be created to build the flatten table with > sqoop: > select ... from [FACT] where [LOOKUP].[PARTITION_COL] > ... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3345) Use Apache Parent POM 19
[ https://issues.apache.org/jira/browse/KYLIN-3345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-3345: Fix Version/s: v2.3.2 > Use Apache Parent POM 19 > > > Key: KYLIN-3345 > URL: https://issues.apache.org/jira/browse/KYLIN-3345 > Project: Kylin > Issue Type: Improvement > Components: Tools, Build and Test >Reporter: Ted Yu >Assignee: Dong Li >Priority: Major > Fix For: v2.4.0, v2.3.2 > > > Kylin is still using Apache Parent POM 16. Apache Parent POM 19 is out. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3328) Upgrade the metadata of sample cube to latest
[ https://issues.apache.org/jira/browse/KYLIN-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen resolved KYLIN-3328. - Resolution: Fixed > Upgrade the metadata of sample cube to latest > - > > Key: KYLIN-3328 > URL: https://issues.apache.org/jira/browse/KYLIN-3328 > Project: Kylin > Issue Type: Improvement > Components: Metadata >Affects Versions: v2.3.1 >Reporter: nichunen >Assignee: nichunen >Priority: Major > Fix For: v2.4.0 > > > The metadata created by running smple.sh is out of date, updating it will > avoid potential bugs -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3352) Segment pruning bug, e.g. date_col > "max_date+1"
[
https://issues.apache.org/jira/browse/KYLIN-3352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI resolved KYLIN-3352.
-
Resolution: Fixed
> Segment pruning bug, e.g. date_col > "max_date+1"
> -
>
> Key: KYLIN-3352
> URL: https://issues.apache.org/jira/browse/KYLIN-3352
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v2.1.0, v2.2.0, v2.3.0
>Reporter: liyang
>Assignee: liyang
>Priority: Major
> Fix For: v2.4.0, v2.3.2
>
>
> Currently {{date_col > "max_date+1"}} is rounded down to {{date_col >
> "max_date"}} during encoding and further evaluated as {{date_col >=
> "max_date"}} during segment pruning. This causes a segment can be pruned is
> not pruned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3348) "missing LastBuildJobID" error when building new cube segment
[
https://issues.apache.org/jira/browse/KYLIN-3348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI updated KYLIN-3348:
Fix Version/s: v2.3.2
> "missing LastBuildJobID" error when building new cube segment
> -
>
> Key: KYLIN-3348
> URL: https://issues.apache.org/jira/browse/KYLIN-3348
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v2.3.0
>Reporter: liyang
>Assignee: liyang
>Priority: Major
> Fix For: v2.4.0, v2.3.2
>
>
> An unstable exception. Likely to happen when there are multiple concurrent
> builds.
> {{2018-04-18 20:11:16,856 ERROR [pool-33-thread-11]
> threadpool.DefaultScheduler : ExecuteException
> job:cc08da19-f53e-4344-a6c5-05e764834924}}
> {{ org.apache.kylin.job.exception.ExecuteException:
> org.apache.kylin.job.exception.ExecuteException:
> java.lang.IllegalStateException: For cube CUBE[name=cube2], segment
> cube2[2018041423000_2018041423001] missing LastBuildJobID}}
> \{{ at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:140)}}
> \{{ at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:307)}}
> \{{ at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)}}
> \{{ at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)}}
> \{{ at java.lang.Thread.run(Thread.java:748)}}
> {{ Caused by: org.apache.kylin.job.exception.ExecuteException:
> java.lang.IllegalStateException: For cube CUBE[name=cube2], segment
> cube2[2018041423000_2018041423001] missing LastBuildJobID}}
> \{{ at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:140)}}
> \{{ at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:67)}}
> \{{ at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:129)}}
> \{{ ... 4 more}}
> {{ Caused by: java.lang.IllegalStateException: For cube CUBE[name=cube2],
> segment cube2[2018041423000_2018041423001] missing LastBuildJobID}}
> \{{ at
> org.apache.kylin.cube.CubeManager$SegmentAssist.promoteNewlyBuiltSegments(CubeManager.java:810)}}
> \{{ at
> org.apache.kylin.cube.CubeManager.promoteNewlyBuiltSegments(CubeManager.java:535)}}
> \{{ at
> org.apache.kylin.engine.mr.steps.UpdateCubeInfoAfterBuildStep.doWork(UpdateCubeInfoAfterBuildStep.java:78)}}
> \{{ at
> io.kyligence.kap.engine.mr.steps.KapUpdateCubeInfoAfterBuildStep.doWork(SourceFile:47)}}
> \{{ at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:129)}}
> \{{ ... 6 more}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3352) Segment pruning bug, e.g. date_col > "max_date+1"
[
https://issues.apache.org/jira/browse/KYLIN-3352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI updated KYLIN-3352:
Fix Version/s: v2.3.2
> Segment pruning bug, e.g. date_col > "max_date+1"
> -
>
> Key: KYLIN-3352
> URL: https://issues.apache.org/jira/browse/KYLIN-3352
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v2.1.0, v2.2.0, v2.3.0
>Reporter: liyang
>Assignee: liyang
>Priority: Major
> Fix For: v2.4.0, v2.3.2
>
>
> Currently {{date_col > "max_date+1"}} is rounded down to {{date_col >
> "max_date"}} during encoding and further evaluated as {{date_col >=
> "max_date"}} during segment pruning. This causes a segment can be pruned is
> not pruned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (KYLIN-3331) Kylin start script hangs during retrieving hive dependencys
[ https://issues.apache.org/jira/browse/KYLIN-3331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen resolved KYLIN-3331. - Resolution: Fixed > Kylin start script hangs during retrieving hive dependencys > --- > > Key: KYLIN-3331 > URL: https://issues.apache.org/jira/browse/KYLIN-3331 > Project: Kylin > Issue Type: Improvement >Reporter: nichunen >Assignee: nichunen >Priority: Minor > Fix For: v2.4.0 > > > This happens if hive client mode is set to be "cli", hive command may hang if > the cluster is in unhealthy status(for instance, zk is stop). The script > should check this and kill the process if it's timeout. > Fail fast and tell the user is the right way. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3372) Upgrade jackson-databind version due to security concerns
[ https://issues.apache.org/jira/browse/KYLIN-3372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-3372: Fix Version/s: v2.3.2 > Upgrade jackson-databind version due to security concerns > - > > Key: KYLIN-3372 > URL: https://issues.apache.org/jira/browse/KYLIN-3372 > Project: Kylin > Issue Type: Task >Reporter: Ted Yu >Assignee: Chao Long >Priority: Major > Fix For: v2.4.0, v2.3.2 > > > * https://nvd.nist.gov/vuln/detail/CVE-2018-5968 > * https://nvd.nist.gov/vuln/detail/CVE-2018-7489 > * https://nvd.nist.gov/vuln/detail/CVE-2017-7525 > * https://nvd.nist.gov/vuln/detail/CVE-2017-17485 > * https://nvd.nist.gov/vuln/detail/CVE-2017-15095 > We should either remove the dependency or upgrade to version 2.8.11.1 or the > latest, if possible. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3398) Inaccurate arithmetic operation in LookupTableToHFileJob#calculateShardNum
[
https://issues.apache.org/jira/browse/KYLIN-3398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507271#comment-16507271
]
ASF GitHub Bot commented on KYLIN-3398:
---
nichunen opened a new pull request #152: KYLIN-3398 Fix inaccurate arithmetic
operations
URL: https://github.com/apache/kylin/pull/152
In LookupTableToHFileJob#calculateShardNum
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Inaccurate arithmetic operation in LookupTableToHFileJob#calculateShardNum
> --
>
> Key: KYLIN-3398
> URL: https://issues.apache.org/jira/browse/KYLIN-3398
> Project: Kylin
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: jiatao.tao
>Priority: Major
>
> There're two issues with the following code:
> {code}
> private int calculateShardNum(KylinConfig kylinConfig, long dataSize) {
> long shardSize = kylinConfig.getExtTableSnapshotShardingMB() * 1024 *
> 1024;
> return dataSize < shardSize ? 1 : (int) (Math.ceil(dataSize /
> shardSize));
> {code}
> getExtTableSnapshotShardingMB returns an int. The multiplication is done
> using 32-bit arithmetic, and then used in a context that expects an
> expression of type "long".
> Second, Math.ceil expects a double. The integer division would lose some
> precision.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3336) java.lang.NoSuchMethodException: org.apache.kylin.tool.HBaseUsageExtractor.execute([Ljava.lang.String;)
[ https://issues.apache.org/jira/browse/KYLIN-3336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-3336: Affects Version/s: (was: v2.3.2) Fix Version/s: v2.3.2 > java.lang.NoSuchMethodException: > org.apache.kylin.tool.HBaseUsageExtractor.execute([Ljava.lang.String;) > --- > > Key: KYLIN-3336 > URL: https://issues.apache.org/jira/browse/KYLIN-3336 > Project: Kylin > Issue Type: Bug > Components: Tools, Build and Test >Affects Versions: v2.3.0 >Reporter: Peng Xing >Assignee: Peng Xing >Priority: Minor > Fix For: v2.4.0, v2.3.2 > > > When using diag.sh, got following exception: > 2018-04-04 16:14:18,222 ERROR [pool-7-thread-3] tool.DiagnosisInfoCLI:171 : > Error in export HBase usage. > java.lang.NoSuchMethodException: > org.apache.kylin.tool.HBaseUsageExtractor.execute([Ljava.lang.String;) > at java.lang.Class.getDeclaredMethod(Class.java:2017) > at > org.apache.kylin.tool.DiagnosisInfoCLI$3.run(DiagnosisInfoCLI.java:168) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622) > at java.lang.Thread.run(Thread.java:745) > Because getDeclaredMethod(String name, Class... parameterTypes) can not > get the inherited methods -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3336) java.lang.NoSuchMethodException: org.apache.kylin.tool.HBaseUsageExtractor.execute([Ljava.lang.String;)
[ https://issues.apache.org/jira/browse/KYLIN-3336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-3336: Affects Version/s: v2.3.2 > java.lang.NoSuchMethodException: > org.apache.kylin.tool.HBaseUsageExtractor.execute([Ljava.lang.String;) > --- > > Key: KYLIN-3336 > URL: https://issues.apache.org/jira/browse/KYLIN-3336 > Project: Kylin > Issue Type: Bug > Components: Tools, Build and Test >Affects Versions: v2.3.0 >Reporter: Peng Xing >Assignee: Peng Xing >Priority: Minor > Fix For: v2.4.0, v2.3.2 > > > When using diag.sh, got following exception: > 2018-04-04 16:14:18,222 ERROR [pool-7-thread-3] tool.DiagnosisInfoCLI:171 : > Error in export HBase usage. > java.lang.NoSuchMethodException: > org.apache.kylin.tool.HBaseUsageExtractor.execute([Ljava.lang.String;) > at java.lang.Class.getDeclaredMethod(Class.java:2017) > at > org.apache.kylin.tool.DiagnosisInfoCLI$3.run(DiagnosisInfoCLI.java:168) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622) > at java.lang.Thread.run(Thread.java:745) > Because getDeclaredMethod(String name, Class... parameterTypes) can not > get the inherited methods -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3401) The current using zip compress tool has an arbitrary file write vulnerability
[ https://issues.apache.org/jira/browse/KYLIN-3401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI resolved KYLIN-3401. - Resolution: Fixed Fix Version/s: v2.3.2 v2.4.0 > The current using zip compress tool has an arbitrary file write vulnerability > - > > Key: KYLIN-3401 > URL: https://issues.apache.org/jira/browse/KYLIN-3401 > Project: Kylin > Issue Type: Bug >Reporter: Chao Long >Assignee: Chao Long >Priority: Major > Fix For: v2.4.0, v2.3.2 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3094) Upgrade zookeeper to 3.4.12
[ https://issues.apache.org/jira/browse/KYLIN-3094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507270#comment-16507270 ] ASF subversion and git services commented on KYLIN-3094: Commit 1e29f8aaa316a8d6000e73530b2310d73f9093b9 in kylin's branch refs/heads/2.3.x from [~nichunen] [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=1e29f8a ] KYLIN-3094 Upgrade zookeeper version from 3.4.8 to 3.4.12 > Upgrade zookeeper to 3.4.12 > --- > > Key: KYLIN-3094 > URL: https://issues.apache.org/jira/browse/KYLIN-3094 > Project: Kylin > Issue Type: Improvement >Reporter: Ted Yu >Assignee: nichunen >Priority: Minor > Fix For: v2.4.0, v2.3.2 > > > Current zookeeper release is 3.4.12 > We should upgrade dependency from 3.4.8 to 3.4.12 where there is important > security fix. > One such critical fix is ZOOKEEPER-2146, which can be explored maliciously -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3094) Upgrade zookeeper to 3.4.12
[ https://issues.apache.org/jira/browse/KYLIN-3094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI resolved KYLIN-3094. - Resolution: Fixed Fix Version/s: v2.3.2 > Upgrade zookeeper to 3.4.12 > --- > > Key: KYLIN-3094 > URL: https://issues.apache.org/jira/browse/KYLIN-3094 > Project: Kylin > Issue Type: Improvement >Reporter: Ted Yu >Assignee: nichunen >Priority: Minor > Fix For: v2.3.2, v2.4.0 > > > Current zookeeper release is 3.4.12 > We should upgrade dependency from 3.4.8 to 3.4.12 where there is important > security fix. > One such critical fix is ZOOKEEPER-2146, which can be explored maliciously -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3094) Upgrade zookeeper to 3.4.12
[ https://issues.apache.org/jira/browse/KYLIN-3094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-3094: Fix Version/s: v2.4.0 > Upgrade zookeeper to 3.4.12 > --- > > Key: KYLIN-3094 > URL: https://issues.apache.org/jira/browse/KYLIN-3094 > Project: Kylin > Issue Type: Improvement >Reporter: Ted Yu >Assignee: nichunen >Priority: Minor > Fix For: v2.4.0 > > > Current zookeeper release is 3.4.12 > We should upgrade dependency from 3.4.8 to 3.4.12 where there is important > security fix. > One such critical fix is ZOOKEEPER-2146, which can be explored maliciously -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3094) Upgrade zookeeper to 3.4.12
[ https://issues.apache.org/jira/browse/KYLIN-3094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507268#comment-16507268 ] ASF subversion and git services commented on KYLIN-3094: Commit 93247fa7f23ac427e8b6a581b0f902366bb9804f in kylin's branch refs/heads/master from [~nichunen] [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=93247fa ] KYLIN-3094 Upgrade zookeeper version from 3.4.8 to 3.4.12 > Upgrade zookeeper to 3.4.12 > --- > > Key: KYLIN-3094 > URL: https://issues.apache.org/jira/browse/KYLIN-3094 > Project: Kylin > Issue Type: Improvement >Reporter: Ted Yu >Assignee: nichunen >Priority: Minor > > Current zookeeper release is 3.4.12 > We should upgrade dependency from 3.4.8 to 3.4.12 where there is important > security fix. > One such critical fix is ZOOKEEPER-2146, which can be explored maliciously -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3094) Upgrade zookeeper to 3.4.12
[ https://issues.apache.org/jira/browse/KYLIN-3094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507267#comment-16507267 ] ASF GitHub Bot commented on KYLIN-3094: --- shaofengshi closed pull request #151: KYLIN-3094 Upgrade zookeeper version from 3.4.8 to 3.4.12 URL: https://github.com/apache/kylin/pull/151 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/pom.xml b/pom.xml index e0bd0e0023..009a4ae187 100644 --- a/pom.xml +++ b/pom.xml @@ -69,7 +69,7 @@ 1.10.0 -3.4.8 +3.4.12 2.7.1 3.0.1 14.0 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Upgrade zookeeper to 3.4.12 > --- > > Key: KYLIN-3094 > URL: https://issues.apache.org/jira/browse/KYLIN-3094 > Project: Kylin > Issue Type: Improvement >Reporter: Ted Yu >Assignee: nichunen >Priority: Minor > > Current zookeeper release is 3.4.12 > We should upgrade dependency from 3.4.8 to 3.4.12 where there is important > security fix. > One such critical fix is ZOOKEEPER-2146, which can be explored maliciously -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3094) Upgrade zookeeper to 3.4.12
[ https://issues.apache.org/jira/browse/KYLIN-3094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507263#comment-16507263 ] ASF GitHub Bot commented on KYLIN-3094: --- coveralls commented on issue #151: KYLIN-3094 Upgrade zookeeper version from 3.4.8 to 3.4.12 URL: https://github.com/apache/kylin/pull/151#issuecomment-396024008 ## Pull Request Test Coverage Report for [Build 3212](https://coveralls.io/builds/17411252) * **0** of **0** changed or added relevant lines in **0** files are covered. * No unchanged relevant lines lost coverage. * Overall coverage remained the same at **24.174%** --- | Totals | [](https://coveralls.io/builds/17411252) | | :-- | --: | | Change from base [Build 3210](https://coveralls.io/builds/17410687): | 0.0% | | Covered Lines: | 14796 | | Relevant Lines: | 61207 | --- # - [Coveralls](https://coveralls.io) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Upgrade zookeeper to 3.4.12 > --- > > Key: KYLIN-3094 > URL: https://issues.apache.org/jira/browse/KYLIN-3094 > Project: Kylin > Issue Type: Improvement >Reporter: Ted Yu >Assignee: nichunen >Priority: Minor > > Current zookeeper release is 3.4.12 > We should upgrade dependency from 3.4.8 to 3.4.12 where there is important > security fix. > One such critical fix is ZOOKEEPER-2146, which can be explored maliciously -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3094) Upgrade zookeeper to 3.4.12
[ https://issues.apache.org/jira/browse/KYLIN-3094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507262#comment-16507262 ] ASF GitHub Bot commented on KYLIN-3094: --- codecov-io commented on issue #151: KYLIN-3094 Upgrade zookeeper version from 3.4.8 to 3.4.12 URL: https://github.com/apache/kylin/pull/151#issuecomment-396023882 # [Codecov](https://codecov.io/gh/apache/kylin/pull/151?src=pr=h1) Report > Merging [#151](https://codecov.io/gh/apache/kylin/pull/151?src=pr=desc) into [master](https://codecov.io/gh/apache/kylin/commit/9fa82a4e9fddbfa24bdf73b49d081f53ca8191bd?src=pr=desc) will **increase** coverage by `<.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/kylin/pull/151?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #151 +/- ## + Coverage 22.14% 22.14% +<.01% Complexity 4020 4020 Files 1012 1012 Lines 6120761207 Branches 8768 8768 + Hits 1355413555 +1 Misses4641346413 + Partials 1240 1239 -1 ``` | [Impacted Files](https://codecov.io/gh/apache/kylin/pull/151?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...lin/dict/lookup/cache/RocksDBLookupTableCache.java](https://codecov.io/gh/apache/kylin/pull/151/diff?src=pr=tree#diff-Y29yZS1kaWN0aW9uYXJ5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9kaWN0L2xvb2t1cC9jYWNoZS9Sb2Nrc0RCTG9va3VwVGFibGVDYWNoZS5qYXZh) | `76.68% <0%> (+0.51%)` | `27% <0%> (ø)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/kylin/pull/151?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/kylin/pull/151?src=pr=footer). Last update [9fa82a4...6868483](https://codecov.io/gh/apache/kylin/pull/151?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Upgrade zookeeper to 3.4.12 > --- > > Key: KYLIN-3094 > URL: https://issues.apache.org/jira/browse/KYLIN-3094 > Project: Kylin > Issue Type: Improvement >Reporter: Ted Yu >Assignee: nichunen >Priority: Minor > > Current zookeeper release is 3.4.12 > We should upgrade dependency from 3.4.8 to 3.4.12 where there is important > security fix. > One such critical fix is ZOOKEEPER-2146, which can be explored maliciously -- This message was sent by Atlassian JIRA (v7.6.3#76005)
