[jira] [Commented] (KYLIN-2897) Improve the query execution for a set of duplicate queries in a short period
[ https://issues.apache.org/jira/browse/KYLIN-2897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676237#comment-16676237 ] ASF subversion and git services commented on KYLIN-2897: Commit a0e703986a5d965b3db8ee5be16dc3a26738c9eb in kylin's branch refs/heads/master from Zhong [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=a0e7039 ] KYLIN-2897 minor fix > Improve the query execution for a set of duplicate queries in a short period > > > Key: KYLIN-2897 > URL: https://issues.apache.org/jira/browse/KYLIN-2897 > Project: Kylin > Issue Type: Sub-task > Components: Query Engine >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > Fix For: v2.6.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3668) Performing a cube build error on the Chinese hive table field
wxs created KYLIN-3668: -- Summary: Performing a cube build error on the Chinese hive table field Key: KYLIN-3668 URL: https://issues.apache.org/jira/browse/KYLIN-3668 Project: Kylin Issue Type: Bug Components: Tools, Build and Test Affects Versions: all Environment: KYLIN Version2.3 hbase1.2.6 hive1.2.2 Reporter: wxs Fix For: all The Chinese name can be created in hive, and the field name can be named Chinese characters. Easy to understand and remember. The Chinese table of DataSource load hive in KYLIN is displayed normally. But when you do cube build, the first step: create intermediate flat hive table error. KYLIN advantage can be connected to BI tools, field Chinese, for the presentation layer is the highlights and concerns. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-3543) Unclosed Job instance in CreateHTableJob#exportHBaseConfiguration
[
https://issues.apache.org/jira/browse/KYLIN-3543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI closed KYLIN-3543.
---
“Resolved in release 2.5.1 (2018-11-06)"
> Unclosed Job instance in CreateHTableJob#exportHBaseConfiguration
> -
>
> Key: KYLIN-3543
> URL: https://issues.apache.org/jira/browse/KYLIN-3543
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v2.5.0
>Reporter: Ted Yu
>Assignee: Yichen Zhou
>Priority: Minor
> Fix For: v2.5.1
>
>
> {code}
> out = fs.create(new Path(hbaseConfPath));
> job.getConfiguration().writeXml(out);
> {code}
> The job instance should be closed upon return from the method.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (KYLIN-3611) Upgrade Tomcat to 7.0.91, 8.5.34 or later
[ https://issues.apache.org/jira/browse/KYLIN-3611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-3611. --- “Resolved in release 2.5.1 (2018-11-06)" > Upgrade Tomcat to 7.0.91, 8.5.34 or later > - > > Key: KYLIN-3611 > URL: https://issues.apache.org/jira/browse/KYLIN-3611 > Project: Kylin > Issue Type: Improvement >Reporter: Shaofeng SHI >Assignee: zhoujie >Priority: Major > Fix For: v2.6.0, v2.5.1 > > > h2. [SECURITY] CVE-2018-11784 Apache Tomcat - Open Redirect > > > > CVE-2018-11784 Apache Tomcat - Open Redirect > Severity: Moderate > Vendor: The Apache Software Foundation > Versions Affected: > Apache Tomcat 9.0.0.M1 to 9.0.11 > Apache Tomcat 8.5.0 to 8.5.33 > Apache Tomcat 7.0.23 to 7.0.90 > The unsupported 8.0.x release line has not been analysed but is likely > to be affected. > Description: > When the default servlet returned a redirect to a directory (e.g. > redirecting to '/foo/' when the user requested '/foo') a specially > crafted URL could be used to cause the redirect to be generated to any > URI of the attackers choice. > Mitigation: > Users of the affected versions should apply one of the following > mitigations: > - Upgrade to Apache Tomcat 9.0.12 or later. > - Upgrade to Apache Tomcat 8.5.34 or later. > - Upgrade to Apache Tomcat 7.0.91 or later. > - Use mapperDirectoryRedirectEnabled="true" and > mapperContextRootRedirectEnabled="true" on the Context to ensure that > redirects are issued by the Mapper rather than the default Servlet. > See the Context configuration documentation for further important > details. > Credit: > This vulnerability was found by Sergey Bobrov and reported responsibly > to the Apache Tomcat Security Team. > History: > 2018-10-03 Original advisory > References: > [1] [http://tomcat.apache.org/security-9.html] > [2] [http://tomcat.apache.org/security-8.html] > [3] [http://tomcat.apache.org/security-7.html] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-3531) Login failed with case-insensitive username
[
https://issues.apache.org/jira/browse/KYLIN-3531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI closed KYLIN-3531.
---
“Resolved in release 2.5.1 (2018-11-06)"
> Login failed with case-insensitive username
> ---
>
> Key: KYLIN-3531
> URL: https://issues.apache.org/jira/browse/KYLIN-3531
> Project: Kylin
> Issue Type: Bug
> Components: Metadata
>Affects Versions: v2.4.0
>Reporter: Yichen Zhou
>Assignee: Yichen Zhou
>Priority: Major
> Fix For: v2.5.1
>
>
> Username is case-insensitive in login authentication but case-sensitive in
> hbase metadata.
> {quote}org.apache.kylin.common.persistence.WriteConflictException:
> Overwriting conflict /user/Admin, expect old TS 1536031295060, but it is 0
>
> org.apache.kylin.storage.hbase.HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:325)
>
> org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:323)
>
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:308)
>
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:287)
>
> org.apache.kylin.metadata.cachesync.CachedCrudAssist.save(CachedCrudAssist.java:192)
>
> org.apache.kylin.rest.security.KylinUserManager.update(KylinUserManager.java:122)
>
> org.apache.kylin.rest.service.KylinUserService.updateUser(KylinUserService.java:85)
>
> org.apache.kylin.rest.security.KylinAuthenticationProvider.authenticate(KylinAuthenticationProvider.java:117)
>
> org.springframework.security.authentication.ProviderManager.authenticate(ProviderManager.java:174)
>
> org.springframework.security.authentication.ProviderManager.authenticate(ProviderManager.java:199)
>
> org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:180)
>
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (KYLIN-3634) When filter column has null value may cause incorrect query result
[ https://issues.apache.org/jira/browse/KYLIN-3634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-3634. --- “Resolved in release 2.5.1 (2018-11-06)" > When filter column has null value may cause incorrect query result > -- > > Key: KYLIN-3634 > URL: https://issues.apache.org/jira/browse/KYLIN-3634 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: v2.0.0 >Reporter: WangBo >Assignee: WangBo >Priority: Major > Fix For: v2.4.2, v2.5.1 > > Attachments: > 0001-KYLIN-3634-when-filter-column-has-null-value-may-cau.patch, > image-2018-10-27-14-11-57-955.png > > > h1. Question > when a column has null value，and using it as a filter column when querying, > and the filter value is not exist in the table,this may cause incorrect result > h1. An Example > h2. Table A > the table A has three rows,city column of one row has null value > > ||day||...||city||price|| > |20180101| |null|10| > |20180101| |beijing|20| > |20180101| |shanghai|10| > h2. Query SQL > select day,sum(price) from a where city <> 'abc' group by day > h2. Correct Result > exclude the row contains null city value > ||day||col|| > |20180101|30| > h2. InCorrect Result > resullt 0 rows > this happens in our production environment,the kylin version is 2.0.0 > h1. Analysis process > 1,city column dosen't have a value,so the CompareTupleFilter will turn into > ConstantTupleFilter(see GTUtil.java) > 2,if dimensions in the sql dosen't match all the columns using in group > by,the bytesComparator used in hbase aggregation map will only compare the > columns using in group by > 3,when GTAggregateScanner constructs key of aggBufMap,the key may contains > null value,because the comparator of aggBufMap only compares group by > columns,so the tuple share same group by columns may also share the same keys > which contains null value;This may cause kylin server receives tuples > contains null value; > 4,when the code which dynamically generated by calcilte deals tuples using > filter,it first judges whether the column is null.Because filter column in > the tuple contains null value,so it always return false, no tuples will > return. > h1. Solution > when the filter column value is a invalid means not in the table,turn the > CompareTupleFiter into IS_NOT_NULL filter,instead of ConstantTupleFilter.TURE > > Now I have test the feature in our production environment ; > test in “mvn test” had passed，but not test in sandbox > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-3645) Kylin does not clean table metadata when drop project(Kafka Source)
[ https://issues.apache.org/jira/browse/KYLIN-3645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-3645. --- “Resolved in release 2.5.1 (2018-11-06)" > Kylin does not clean table metadata when drop project(Kafka Source) > --- > > Key: KYLIN-3645 > URL: https://issues.apache.org/jira/browse/KYLIN-3645 > Project: Kylin > Issue Type: Bug > Components: Metadata, Streaming >Affects Versions: v2.4.1 > Environment: jdk1.8 > MacOS >Reporter: rongchuan.jin >Assignee: rongchuan.jin >Priority: Minor > Fix For: v2.4.2, v2.5.1 > > > When I use Kafka streaming,there is a bug. My steps: > 1.Create a project named p1 > 2.Import a kafka table named DEFAULT.T1 > 3.drop project p1 > 4.create project p2 > 5.Import a kafka table named DEFAULT.T1 > It comes out an error: > The Streaming Config DEFAULT.T1 already exists. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-3649) segment region count and size are not correct when using mysql as Kylin metadata storage
[
https://issues.apache.org/jira/browse/KYLIN-3649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI closed KYLIN-3649.
---
“Resolved in release 2.5.1 (2018-11-06)"
> segment region count and size are not correct when using mysql as Kylin
> metadata storage
> -
>
> Key: KYLIN-3649
> URL: https://issues.apache.org/jira/browse/KYLIN-3649
> Project: Kylin
> Issue Type: Bug
> Components: Metadata
>Affects Versions: v2.5.0
>Reporter: Lingang Deng
>Assignee: Lingang Deng
>Priority: Major
> Fix For: v2.5.1
>
> Attachments: 0001-KYLIN-3649.patch
>
>
> As titles, segment region count and size are not correct.
> {code:java}
> if ("hbase".equals(getConfig().getMetadataUrl().getScheme())) {
> try {
> logger.debug("Loading HTable info " + cubeName + ", " + tableName);
> // use reflection to isolate NoClassDef errors when HBase is not
> available
> hr = (HBaseResponse)
> Class.forName("org.apache.kylin.rest.service.HBaseInfoUtil")//
> .getMethod("getHBaseInfo", new Class[] { String.class,
> KylinConfig.class })//
> .invoke(null, tableName, this.getConfig());
> } catch (Throwable e) {
> throw new IOException(e);
> }
> }
> {code}
> Judgement is not valid when using mysql as Kylin metadata storage.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (KYLIN-3439) configuration "kylin.web.timezone" is Inconsistent between kylin-defaults.properties and KylinConfigBase.java
[ https://issues.apache.org/jira/browse/KYLIN-3439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-3439. --- “Resolved in release 2.5.1 (2018-11-06)" > configuration "kylin.web.timezone" is Inconsistent between > kylin-defaults.properties and KylinConfigBase.java > -- > > Key: KYLIN-3439 > URL: https://issues.apache.org/jira/browse/KYLIN-3439 > Project: Kylin > Issue Type: Bug >Reporter: Jiatao Tao >Assignee: Jiatao Tao >Priority: Minor > Fix For: v2.6.0, v2.5.1 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-3599) Bulk Add Measures
[ https://issues.apache.org/jira/browse/KYLIN-3599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-3599. --- “Resolved in release 2.5.1 (2018-11-06)" > Bulk Add Measures > - > > Key: KYLIN-3599 > URL: https://issues.apache.org/jira/browse/KYLIN-3599 > Project: Kylin > Issue Type: Improvement > Components: Web , Website >Reporter: Pan, Julian >Assignee: Pan, Julian >Priority: Major > Fix For: v2.5.1 > > Attachments: bluk add button.png, select all or clean.png > > > If user want to add hundreds measure, user need add them one by one in > currently website. > Add a bulk add button to resolve this issues in measure step. > P.S just include common measure (sum, max, min, raw, percentile), due to > count_distinct, topN and extended_column need more setting. Image attached > and the default name generate automatically by pattern [measure_columnName]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-3635) Percentile calculation on Spark engine is wrong
[ https://issues.apache.org/jira/browse/KYLIN-3635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-3635. --- “Resolved in release 2.5.1 (2018-11-06)" > Percentile calculation on Spark engine is wrong > --- > > Key: KYLIN-3635 > URL: https://issues.apache.org/jira/browse/KYLIN-3635 > Project: Kylin > Issue Type: Bug > Components: Spark Engine >Affects Versions: v2.3.0, v2.3.1, v2.4.0, v2.3.2, v2.4.1, v2.5.0 >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI >Priority: Major > Fix For: v2.4.2, v2.5.1 > > > As titled; Received reporting that percentile result is wrong when using > Spark engine. Checked the code and found the object was reused and not reset, > the problem won't happen on normal MR as it would be serialized before next > call, but would be a problem on Spark, as the object was cached in-memory and > be serialized later. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-3546) Kylin doesn't persist FK to intermediate table if FK is not a dimension
[ https://issues.apache.org/jira/browse/KYLIN-3546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-3546. --- “Resolved in release 2.5.1 (2018-11-06)" > Kylin doesn't persist FK to intermediate table if FK is not a dimension > --- > > Key: KYLIN-3546 > URL: https://issues.apache.org/jira/browse/KYLIN-3546 > Project: Kylin > Issue Type: Bug > Components: Streaming >Affects Versions: v2.4.0 >Reporter: wlxie >Assignee: XiaoXiang Yu >Priority: Major > Fix For: v2.4.2, v2.5.1 > > Attachments: base_2018_09_18_09_23_53.zip, kylin_kafka.txt > > > 各位老师好， > 在使用kafka的streaming > table作为事实表创建cube时候，如果维度和指标都不选事实表中的关联键的话，在跑cube作业的时候，创建保存kafka数据的临时表会没有这个关联键，导致后面创建关联表的临时表会报错，具体报错信息请查看附件，谢谢。 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-3527) Hybrid couldn't save when there is only 1 cube
[ https://issues.apache.org/jira/browse/KYLIN-3527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-3527. --- “Resolved in release 2.5.1 (2018-11-06)" > Hybrid couldn't save when there is only 1 cube > -- > > Key: KYLIN-3527 > URL: https://issues.apache.org/jira/browse/KYLIN-3527 > Project: Kylin > Issue Type: Bug > Components: Web >Affects Versions: v2.5.0 >Reporter: Shaofeng SHI >Assignee: Roger >Priority: Minor > Fix For: v2.5.1 > > Attachments: hybrid-not-saved.png, move-in-out-icons.png > > > I created a hybrid with two cubes; Later I want to remove one cube from it, > but I found when there is only 1 cube, the "Submit" button was disabled. > > The hybrid should allows 0, 1 or more cubes. > > Besides, I found the move in and out icons were not properly displayed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-3644) NumberFormatExcetion on null values when building cube with Spark
[
https://issues.apache.org/jira/browse/KYLIN-3644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI closed KYLIN-3644.
---
“Resolved in release 2.5.1 (2018-11-06)"
> NumberFormatExcetion on null values when building cube with Spark
> -
>
> Key: KYLIN-3644
> URL: https://issues.apache.org/jira/browse/KYLIN-3644
> Project: Kylin
> Issue Type: Bug
> Components: Spark Engine
>Affects Versions: v2.5.0
>Reporter: Hubert STEFANI
>Assignee: Chao Long
>Priority: Major
> Fix For: v2.5.1
>
> Attachments: 00_zeppelin_notebook.jpg, 01_overview_table.jpg,
> 02_dimension_cube.jpg, 03_measure_cube.jpg, sortieData.csv
>
>
> We encounter an error any time we try to build a cube with the following
> steps :
> * upload a csv on AWS S3 with following characteristics : the column on
> which the measure will be defined has some null values (Cf. attachment)
> * create a hive table with spark
> * create a model on top of this table,
> * create a cube with a SUM measure
> * chose Spark as Engine
> * Launch build
> Result : The build process fails at '{color:#4383b4}#7 Step Name:
> {color}Build Cube with Spark' with the following error :
>
> ""
> 18/10/23 09:25:39 INFO scheduler.DAGScheduler: Job 0 failed:
> saveAsNewAPIHadoopDataset at SparkCubingByLayer.java:253, took 7,277136 s
> Exception in thread "main" java.lang.RuntimeException: error execute
> org.apache.kylin.engine.spark.SparkCubingByLayer. Root cause: Job aborted due
> to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure:
> Lost task 0.3 in stage 0.0 (TID 4,
> ip-172-31-35-113.eu-west-1.compute.internal, executor 4):
> java.lang.NumberFormatException: For input string: "\N"
> at
> sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2043)
> at sun.misc.FloatingDecimal.parseDouble(FloatingDecimal.java:110)
> at java.lang.Double.parseDouble(Double.java:538)
> at
> org.apache.kylin.measure.basic.DoubleIngester.valueOf(DoubleIngester.java:38)
> at
> org.apache.kylin.measure.basic.DoubleIngester.valueOf(DoubleIngester.java:28)
> at
> org.apache.kylin.engine.mr.common.BaseCuboidBuilder.buildValueOf(BaseCuboidBuilder.java:162)
> at
> org.apache.kylin.engine.mr.common.BaseCuboidBuilder.buildValueObjects(BaseCuboidBuilder.java:127)
> at
> org.apache.kylin.engine.spark.SparkCubingByLayer$EncodeBaseCuboid.call(SparkCubingByLayer.java:297)
> at
> org.apache.kylin.engine.spark.SparkCubingByLayer$EncodeBaseCuboid.call(SparkCubingByLayer.java:257)
> at
> org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1043)
> at
> org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1043)
> "
> Note 1: the build process is OK when run with Map/Reduce Engine.
> Note 2: the error doesn't seem to be related to AWS environment.
>
> Sample of csv :
> ID;CATEGORIE;TEL;MONTANT;MAGASIN;MATRICULE;VILLE;
> 970;161;6-98-6-6-42;838.47034;Magasin_19;Client_Matricule_28;MARSEILLE;
> 971;89;62-15-2-64-86;;;Client_Matricule_1;LYON;
> 972;87;17-64-97-74-42;;;Client_Matricule_105;ORBEC;
> 973;174;79-33-90-0-55;;Magasin_7;Client_Matricule_55;AJACCIO;
> 974;172;89-95-71-6-49;141.64174;Magasin_9;Client_Matricule_105;BASTIA;
> 975;83;7-27-95-28-7;897.28204;;Client_Matricule_199;AJACCIO;
> 976;170;67-72-18-29-34;164.07967;Magasin_3;Client_Matricule_137;LILLE;
> 977;130;14-69-4-23-27;1928.9557;Magasin_1;Client_Matricule_17;NOMNOM;
> 978;43;55-91-84-98-49;891.2691;Magasin_0;Client_Matricule_22;NOMNOM;
> 979;117;98-96-0-54-39;1636.3994;Magasin_9;Client_Matricule_142;MARSEILLE;
> 980;163;37-55-76-53-38;;;Client_Matricule_64;NEWYORK;
> 981;106;32-40-6-46-15;;Magasin_2;Client_Matricule_158;NOMNOM;
> 982;56;95-60-83-89-90;;;Client_Matricule_102;NOMNOM;
> 983;168;21-56-62-0-58;;;Client_Matricule_160;NOMNOM;
> 984;154;92-67-37-94-60;;;Client_Matricule_137;PARIS;
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (KYLIN-3633) Dead lock may happen in building global dictionary
[ https://issues.apache.org/jira/browse/KYLIN-3633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-3633. --- “Resolved in release 2.5.1 (2018-11-06)" > Dead lock may happen in building global dictionary > -- > > Key: KYLIN-3633 > URL: https://issues.apache.org/jira/browse/KYLIN-3633 > Project: Kylin > Issue Type: Bug > Components: Measure - Count Distinct >Affects Versions: v2.4.1 >Reporter: nichunen >Assignee: nichunen >Priority: Major > Fix For: v2.5.1 > > > During the building of global dict, if an error happens, the builder's lock > maybe un-released. This will block all other building jobs with global dict. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-3604) Can't build cube with spark in HBase standalone mode
[
https://issues.apache.org/jira/browse/KYLIN-3604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI closed KYLIN-3604.
---
“Resolved in release 2.5.1 (2018-11-06)"
> Can't build cube with spark in HBase standalone mode
>
>
> Key: KYLIN-3604
> URL: https://issues.apache.org/jira/browse/KYLIN-3604
> Project: Kylin
> Issue Type: Bug
> Components: Storage - HBase
>Affects Versions: v2.5.0
>Reporter: Colin Ma
>Assignee: Colin Ma
>Priority: Blocker
> Fix For: v2.5.1
>
> Attachments: KYLIN-3604.001.patch
>
>
> With Hbase standalone mode, Cube can't be built on step 8 Convert Cuboid Data
> to HFile, the following is the related exception:
> 18/09/29 11:13:21 INFO steps.SparkCubeHFile: Input path:
> {color:#d04437}hdfs://nameservice1/kylin/kylin_metadata/kylin-b65c0e62-69e9-bb11-9d7d-e6e5abc7ef8e/test_spark_cube/cuboid/{color}
> 18/09/29 11:13:21 INFO steps.SparkCubeHFile: Output path:
> {color:#14892c}hdfs://nameservice3/kylin/kylin_metadata/kylin-b65c0e62-69e9-bb11-9d7d-e6e5abc7ef8e/test_spark_cube/hfile{color}
> 18/09/29 11:13:21 INFO steps.SparkCubeHFile: Loading HBase configuration
> {color:#d04437}from:hdfs://nameservice1/kylin/kylin_metadata/kylin-b65c0e62-69e9-bb11-9d7d-e6e5abc7ef8e/hbase-conf.xml{color}
> 18/09/29 11:13:21 ERROR yarn.ApplicationMaster: User class threw exception:
> java.lang.RuntimeException: error execute
> org.apache.kylin.storage.hbase.steps.SparkCubeHFile. Root cause: Wrong FS:
> hdfs://nameservice1/kylin/kylin_metadata/kylin-b65c0e62-69e9-bb11-9d7d-e6e5abc7ef8e/hbase-conf.xml,
> expected: hdfs://nameservice3 java.lang.RuntimeException: error execute
> org.apache.kylin.storage.hbase.steps.SparkCubeHFile. Root cause: Wrong FS:
> hdfs://nameservice1/kylin/kylin_metadata/kylin-b65c0e62-69e9-bb11-9d7d-e6e5abc7ef8e/hbase-conf.xml,
> expected: hdfs://nameservice3 at
> org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:42)
> at org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:44) at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606) at
> org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:637)
> Caused by: java.lang.IllegalArgumentException: Wrong FS:
> hdfs://nameservice1/kylin/kylin_metadata/kylin-b65c0e62-69e9-bb11-9d7d-e6e5abc7ef8e/hbase-conf.xml,
> expected: hdfs://nameservice3 at
> org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:645) at
> org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:193)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:105)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:302)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:298)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:298)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:766) at
> org.apache.kylin.storage.hbase.steps.SparkCubeHFile.execute(SparkCubeHFile.java:183)
> at
> org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37)
> ... 6 more 18/09/29 11:13:21 INFO yarn.ApplicationMaster: Final app status:
> FAILED, exitCode: 15, (reason: User class threw exception:
> java.lang.RuntimeException: error execute
> org.apache.kylin.storage.hbase.steps.SparkCubeHFile. Root cause: Wrong FS:
> hdfs://nameservice1/kylin/kylin_metadata/kylin-b65c0e62-69e9-bb11-9d7d-e6e5abc7ef8e/hbase-conf.xml,
> expected: hdfs://nameservice3) 18/09/29 11:13:21 INFO spark.SparkContext:
> Invoking stop() from shutdown hook 18/09/29 11:13:21 INFO
> server.ServerConnector: Stopped Spark@1785d078
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (KYLIN-3565) User login error message is inaccurate
[ https://issues.apache.org/jira/browse/KYLIN-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-3565. --- “Resolved in release 2.5.1 (2018-11-06)" > User login error message is inaccurate > -- > > Key: KYLIN-3565 > URL: https://issues.apache.org/jira/browse/KYLIN-3565 > Project: Kylin > Issue Type: Bug > Components: REST Service, Web >Affects Versions: v2.4.1 > Environment: CentOS 6.7, HBase 1.2.0+cdh5.14.2+456 >Reporter: Marc Wu >Assignee: Roger >Priority: Minor > Labels: usability > Fix For: v2.5.1 > > Attachments: 2018-09-17_16-56-12.png, 2018-09-17_16-56-38.png, > 2018-09-17_16-59-31.png, image-2018-09-17-20-22-20-294.png, > image-2018-09-17-20-25-54-896.png > > > Hi Kylin team, > > We found one issue when log in Kylin. The error message will misunderstood > user. > > I couldn't log in Kylin even I input correct username & password (enable > LDAP). > So I checked the log on server, it showed me that there exist HBase > connection issues. > Root cause is that the HBase server which Kylin used as metadata server is > down, but the login message showed me that I should check my username or > password. It's really confusing. > Then I read some source codes about login module, and found out Kylin shares > the same error message for different cases. > > We suggested two options: > # redirect to global error page when HBase connection fail after login, > shows error message(e.g. System Error, please contact system administrator). > # enhance the error code for login logic, make the error message more > specific. > > There are login error message and the log. And I log in successfully after > recover HBase servers. > !image-2018-09-17-20-25-54-896.png!!image-2018-09-17-20-22-20-294.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-2200) CompileException on UNION ALL query when result only contains one column
[
https://issues.apache.org/jira/browse/KYLIN-2200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI closed KYLIN-2200.
---
“Resolved in release 2.5.1 (2018-11-06)"
> CompileException on UNION ALL query when result only contains one column
>
>
> Key: KYLIN-2200
> URL: https://issues.apache.org/jira/browse/KYLIN-2200
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: v1.5.4.1
>Reporter: Dayue Gao
>Assignee: Dayue Gao
>Priority: Major
> Fix For: v2.5.1
>
> Attachments: KYLIN-2200.patch
>
>
> {code:sql}
> select count(*) from kylin_sales
> union all
> select count(*) from kylin_sales
> {code}
> got following exception
> {noformat}
> Caused by: org.codehaus.commons.compiler.CompileException: Line 82, Column
> 32: Cannot determine simple type name "Record11_1"
> at
> org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:10092)
> at
> org.codehaus.janino.UnitCompiler.getReferenceType(UnitCompiler.java:5375)
> at
> org.codehaus.janino.UnitCompiler.getReferenceType(UnitCompiler.java:5184)
> at org.codehaus.janino.UnitCompiler.getType2(UnitCompiler.java:5165)
> at
> org.codehaus.janino.UnitCompiler.access$12600(UnitCompiler.java:183)
> at
> org.codehaus.janino.UnitCompiler$16.visitReferenceType(UnitCompiler.java:5096)
> at org.codehaus.janino.Java$ReferenceType.accept(Java.java:2880)
> at org.codehaus.janino.UnitCompiler.getType(UnitCompiler.java:5136)
> at org.codehaus.janino.UnitCompiler.getType2(UnitCompiler.java:5598)
> at
> org.codehaus.janino.UnitCompiler.access$13300(UnitCompiler.java:183)
> at
> org.codehaus.janino.UnitCompiler$16.visitCast(UnitCompiler.java:5104)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (KYLIN-3595) Beeline does not retrieve CLASSPATH and hangs
[ https://issues.apache.org/jira/browse/KYLIN-3595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-3595. --- “Resolved in release 2.5.1 (2018-11-06)" > Beeline does not retrieve CLASSPATH and hangs > - > > Key: KYLIN-3595 > URL: https://issues.apache.org/jira/browse/KYLIN-3595 > Project: Kylin > Issue Type: Bug > Components: Environment >Affects Versions: v2.5.0 >Reporter: Richard Jarvis >Assignee: Richard Jarvis >Priority: Minor > Fix For: v2.5.1 > > > There is no semi-colon at the end of the set command provided to the beeline > script in find-hive-dependency.sh. This causes the script to hang. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-3526) No check for hybrid name with special character
[ https://issues.apache.org/jira/browse/KYLIN-3526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-3526. --- “Resolved in release 2.5.1 (2018-11-06)" > No check for hybrid name with special character > --- > > Key: KYLIN-3526 > URL: https://issues.apache.org/jira/browse/KYLIN-3526 > Project: Kylin > Issue Type: Improvement > Components: Web >Reporter: Na Zhai >Assignee: Roger >Priority: Minor > Fix For: v2.5.1 > > Attachments: 名称没做限制.png > > > When create hybrid，there is no check for hybrid name with special character. > Such as !@&. > Hybrid can be created successfully with name like hy! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-3651) JDBCResourceStore doesn't list all resources
[
https://issues.apache.org/jira/browse/KYLIN-3651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI closed KYLIN-3651.
---
“Resolved in release 2.5.1 (2018-11-06)"
> JDBCResourceStore doesn't list all resources
>
>
> Key: KYLIN-3651
> URL: https://issues.apache.org/jira/browse/KYLIN-3651
> Project: Kylin
> Issue Type: Bug
> Components: Metadata
>Affects Versions: v2.5.0
>Reporter: Shaofeng SHI
>Assignee: Shaofeng SHI
>Priority: Major
> Fix For: v2.5.1
>
>
> The JDBC resource store persists metadata into two tables:
> # kylin_default_instance
> # kylin_default_instance_log
>
> The metadata entry like "/execute_output" will be saved to the _log table;
> But the "list" operation on "/" only lists the subfolders in the first table,
> no in the second table. This will cause partial metadata copy when use
> #ResourceTool.copy.
>
> {code:java}
> ./bin/metastore.sh list /
> [/UUID, /acl, /cube, /cube_desc, /cube_statistics, /dict, /execute, /kafka,
> /model_desc, /project, /streaming, /table, /table_acl, /table_snapshot, /user]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (KYLIN-3556) Interned string should not be used as lock object
[
https://issues.apache.org/jira/browse/KYLIN-3556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI closed KYLIN-3556.
---
“Resolved in release 2.5.1 (2018-11-06)"
> Interned string should not be used as lock object
> -
>
> Key: KYLIN-3556
> URL: https://issues.apache.org/jira/browse/KYLIN-3556
> Project: Kylin
> Issue Type: Bug
> Components: Metadata
>Affects Versions: v2.5.0
>Reporter: Ted Yu
>Assignee: Kaige Liu
>Priority: Minor
> Fix For: v2.5.1
>
>
> In JDBCResourceDAO :
> {code}
> public void execute(Connection connection) throws SQLException {
> synchronized (resPath.intern()) {
> {code}
> Locking on an interned string can cause unexpected locking collisions with
> other part of code.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (KYLIN-3613) Kylin with Standalone HBase Cluster (enabled kerberos) could not find the main cluster namespace at "Create HTable" step
[ https://issues.apache.org/jira/browse/KYLIN-3613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-3613. --- “Resolved in release 2.5.1 (2018-11-06)" > Kylin with Standalone HBase Cluster (enabled kerberos) could not find the > main cluster namespace at "Create HTable" step > - > > Key: KYLIN-3613 > URL: https://issues.apache.org/jira/browse/KYLIN-3613 > Project: Kylin > Issue Type: Bug > Components: Environment >Affects Versions: v2.4.0, v2.4.1, v2.5.0 >Reporter: powerinf >Assignee: WangSheng >Priority: Major > Fix For: v2.4.2, v2.5.1 > > Attachments: KYLIN-3613.patch, Kylin_standalone_hbase.log > > > I deployed two hadoop cluster(also enabled kerberos ,with cross-realm trust) > the main cluster and hbase cluster,Kylin Server can access both clusters > using hdfs shell with fully qualifiered path ,can submit MR job to main > cluster, and can use hive shell to access data warehouse > on Kylin Server, the configurations of hadoop and hive points to main > cluster,and can access hbase cluster using hbase shell. > when I build the cube, at "Create HTable" step, it reported the error > "java.net.UnknownHostException: ctyunbigdata Set hbase.table.sanity.checks to > false > at conf or table descriptor if you want to bypass sanity checks",but after I > restart Kylin serer , resume it can run normally,Why? > more detail message on Kylin_standalone_hbase.log -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-3567) Change scala dependency to provided
[ https://issues.apache.org/jira/browse/KYLIN-3567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-3567. --- “Resolved in release 2.5.1 (2018-11-06)" > Change scala dependency to provided > --- > > Key: KYLIN-3567 > URL: https://issues.apache.org/jira/browse/KYLIN-3567 > Project: Kylin > Issue Type: Bug >Reporter: Chao Long >Assignee: Chao Long >Priority: Minor > Fix For: v2.5.1 > > > The scope of scala dependency in "engine-spark" module should be "provided". > > org.scala-lang > scala-library > > > org.scala-lang > scala-compiler > > > org.scala-lang > scala-reflect > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-3632) Add configuration that can switch on/off preparedStatement cache in Kylin server
[ https://issues.apache.org/jira/browse/KYLIN-3632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-3632. --- “Resolved in release 2.5.1 (2018-11-06)" > Add configuration that can switch on/off preparedStatement cache in Kylin > server > > > Key: KYLIN-3632 > URL: https://issues.apache.org/jira/browse/KYLIN-3632 > Project: Kylin > Issue Type: Improvement > Components: Query Engine >Reporter: Ma Gang >Assignee: Ma Gang >Priority: Minor > Fix For: v2.5.1 > > > From Kylin 2.5 we introduce preparedStatement cache feature, it can be turn > on/off for each request by adding a new field "enableStatementCache" in the > query request, by default it is on. We need to add a switch in server level, > the preparedStatement cache can take effective only when the switch is on, by > default, it will be set to true by default. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-3646) Add instruction about rowkey sequence in "Advanced setting" page
[ https://issues.apache.org/jira/browse/KYLIN-3646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-3646. --- “Resolved in release 2.5.1 (2018-11-06)" > Add instruction about rowkey sequence in "Advanced setting" page > > > Key: KYLIN-3646 > URL: https://issues.apache.org/jira/browse/KYLIN-3646 > Project: Kylin > Issue Type: Improvement > Components: Web >Reporter: Shaofeng SHI >Assignee: Roger >Priority: Major > Fix For: v2.4.2, v2.5.1 > > > The position of dimension in rowkey is important for performance (both build > and query); But this message is not clear for a normal user. We'd better add > instruction on the web GUI, so that user knows to adjust it by some rule. > > In the rowkey part, adding: > "Important: Dimension's positioin on HBase rowkey is critical for > performance. You can drag and drop to adjust the sequence. In short, put > filtering dimension before non-filtering dimension, and put high cardinality > dimension before low cardinality dimension." -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-3647) Fix inconsistent states of job and its sub-task
[ https://issues.apache.org/jira/browse/KYLIN-3647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-3647. --- “Resolved in release 2.5.1 (2018-11-06)" > Fix inconsistent states of job and its sub-task > --- > > Key: KYLIN-3647 > URL: https://issues.apache.org/jira/browse/KYLIN-3647 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.5.0 >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Fix For: v2.4.2, v2.5.1 > > Attachments: Jietu20181024-180244-HD.mov > > > Sometimes a job failed with one sub-task running. But resume that job in > ([http://localhost:7070/kylin/jobs]) wouldn't work. > It will fail again and again. Only solution is restart kylin process. > > The problem is show in attached file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-3520) Deal with NULL values of measures for inmem cubing
[ https://issues.apache.org/jira/browse/KYLIN-3520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-3520. --- “Resolved in release 2.5.1 (2018-11-06)" > Deal with NULL values of measures for inmem cubing > -- > > Key: KYLIN-3520 > URL: https://issues.apache.org/jira/browse/KYLIN-3520 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > Fix For: v2.5.1 > > Attachments: APACHE-KYLIN-3520.patch > > > Previously NULL values will be dealt for dimensions during both layered > cubing and inmem cubing. However, NULL values of measures only are dealt for > layered cubing. NULL values of measures for inmem cubing also should be dealt > with. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-3629) NullPointException throws when use preparedStatement cache in some case
[
https://issues.apache.org/jira/browse/KYLIN-3629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI closed KYLIN-3629.
---
“Resolved in release 2.5.1 (2018-11-06)"
> NullPointException throws when use preparedStatement cache in some case
> ---
>
> Key: KYLIN-3629
> URL: https://issues.apache.org/jira/browse/KYLIN-3629
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Reporter: Ma Gang
>Assignee: Ma Gang
>Priority: Major
> Fix For: v2.5.1
>
> Attachments: KYLIN-3629.patch
>
>
> NullPointException throws when use preparedStatement cache in some case that
> there is no realization exists in the OLAPContext, for example the following
> sql request will cause issue:
> {color:#205081}{"sql":"select t1.lstg_site_id,t1.seller_id, count(1) from
> (select lstg_site_id,seller_id,count(1) from TEST_KYLIN_FACT where
> lstg_site_id=? and seller_id=? group by lstg_site_id,seller_id) t1 left join
> (select lstg_site_id,seller_id,count(1) from TEST_KYLIN_FACT where
> lstg_site_id=? and seller_id=? group by lstg_site_id,seller_id) t2 on
> t1.lstg_site_id = t2.lstg_site_id group by
> t1.lstg_site_id,t1.seller_id","offset":0,"limit":5,"params":[\{"className":"java.lang.Integer","value":"100"},\{"className":"java.lang.Integer","value":"1731"},\{"className":"java.lang.Integer","value":"100"},\{"className":"java.lang.Integer","value":"1732"}],"acceptPartial":true,"project":"default","backdoorToggles":
> {}}{color}
> the exception stacktrace is like:
> {color:#205081}org.apache.kylin.rest.exception.InternalErrorException:
> Unknown error.\n\tat
> org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:424)\n\tat
>
> org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:351)\n\tat
>
> org.apache.kylin.rest.controller.QueryController.query(QueryController.java:86)\n\tat
> sun.reflect.GeneratedMethodAccessor155.invoke(Unknown Source)\n\tat
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat
> java.lang.reflect.Method.invoke(Method.java:498)\n\tat
> org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)\n\tat
>
> org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)\n\tat
>
> org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)\n\tat
>
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)\n\tat
>
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)\n\tat
>
> org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)\n\tat
>
> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)\n\tat
>
> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)\n\tat
>
> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)\n\tat
>
> org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)\n\tat
> javax.servlet.http.HttpServlet.service(HttpServlet.java:661)\n\tat
> org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)\n\tat
> javax.servlet.http.HttpServlet.service(HttpServlet.java:742)\n\tat
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)\n\tat
>
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)\n\tat
> org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)\n\tat
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)\n\tat
>
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)\n\tat
>
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)\n\tat
>
> org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)\n\tat
>
> org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)\n\tat
>
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)\n\tat
>
> org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)\n\tat
>
>
[jira] [Closed] (KYLIN-3528) No details page for hybrid
[ https://issues.apache.org/jira/browse/KYLIN-3528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-3528. --- “Resolved in release 2.5.1 (2018-11-06)" > No details page for hybrid > -- > > Key: KYLIN-3528 > URL: https://issues.apache.org/jira/browse/KYLIN-3528 > Project: Kylin > Issue Type: Improvement > Components: Web >Reporter: Na Zhai >Assignee: Roger >Priority: Minor > Fix For: v2.5.1 > > > When I click hybrid name, there is no detail page for that hybrid. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-3582) String comparison should not use == in RecordEvent
[
https://issues.apache.org/jira/browse/KYLIN-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI closed KYLIN-3582.
---
“Resolved in release 2.5.1 (2018-11-06)"
> String comparison should not use == in RecordEvent
> --
>
> Key: KYLIN-3582
> URL: https://issues.apache.org/jira/browse/KYLIN-3582
> Project: Kylin
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: XiaoXiang Yu
>Priority: Minor
> Fix For: v2.4.2, v2.5.1
>
>
> Here is related code:
> {code}
> for (RecordReserveKeyEnum reserveKey :
> RecordReserveKeyEnum.values()) {
> if (reserveKey.reserveKey == key) {
> {code}
> equals() should be used to compare Strings.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (KYLIN-3562) TS conflict when kylin update metadata in HBase
[
https://issues.apache.org/jira/browse/KYLIN-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI closed KYLIN-3562.
---
“Resolved in release 2.5.1 (2018-11-06)"
> TS conflict when kylin update metadata in HBase
> ---
>
> Key: KYLIN-3562
> URL: https://issues.apache.org/jira/browse/KYLIN-3562
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v2.4.0
>Reporter: Lingang Deng
>Assignee: Jiatao Tao
>Priority: Major
> Fix For: v2.4.2, v2.5.1
>
> Attachments: image-2018-09-17-16-40-56-212.png,
> image-2018-09-25-15-03-51-009.png, image-2018-09-25-16-43-50-277.png
>
>
> Error log was as follows,
> {code:java}
> org.apache.kylin.common.persistence.WriteConflictException: Overwriting
> conflict /user/admin, expect old TS 1536928877043, but it is 1536928907207
> at
> org.apache.kylin.storage.hbase.HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:325)
> at
> org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:318)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:303)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:282)
> at
> org.apache.kylin.metadata.cachesync.CachedCrudAssist.save(CachedCrudAssist.java:192){code}
>
> what disturbs me the most was that the error was happened several hours,
> then my all build job and query job failed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3667) ArrayIndexOutOfBoundsException in NDCuboidBuilder
[ https://issues.apache.org/jira/browse/KYLIN-3667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hubert STEFANI updated KYLIN-3667: -- Description: The former errors https://issues.apache.org/jira/browse/KYLIN-3115 and https://issues.apache.org/jira/browse/KYLIN-1768 still remain in step SparkCubingByLayer. we encounter the following error : java.lang.ArrayIndexOutOfBoundsException at java.lang.System.arraycopy(Native Method) at org.apache.kylin.engine.mr.common.NDCuboidBuilder.buildKeyInternal(NDCuboidBuilder.java:106) at org.apache.kylin.engine.mr.common.NDCuboidBuilder.buildKey2(NDCuboidBuilder.java:87) at org.apache.kylin.engine.spark.SparkCubingByLayer$CuboidFlatMap.call(SparkCubingByLayer.java:425) at org.apache.kylin.engine.spark.SparkCubingByLayer$CuboidFlatMap.call(SparkCubingByLayer.java:370) at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$3$1.apply(JavaRDDLike.scala:143) at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$3$1.apply(JavaRDDLike.scala:143) at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434) at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440) at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:191) at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:63) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53) at org.apache.spark.scheduler.Task.run(Task.scala:99) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) Do we have to (painfully) change dimensions size or should it be fixed through a patch ? was: The former errors https://issues.apache.org/jira/browse/KYLIN-3115 and https://issues.apache.org/jira/browse/KYLIN-1768 still remains in step SparkCubingByLayer. we encounter the following error : java.lang.ArrayIndexOutOfBoundsException at java.lang.System.arraycopy(Native Method) at org.apache.kylin.engine.mr.common.NDCuboidBuilder.buildKeyInternal(NDCuboidBuilder.java:106) at org.apache.kylin.engine.mr.common.NDCuboidBuilder.buildKey2(NDCuboidBuilder.java:87) at org.apache.kylin.engine.spark.SparkCubingByLayer$CuboidFlatMap.call(SparkCubingByLayer.java:425) at org.apache.kylin.engine.spark.SparkCubingByLayer$CuboidFlatMap.call(SparkCubingByLayer.java:370) at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$3$1.apply(JavaRDDLike.scala:143) at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$3$1.apply(JavaRDDLike.scala:143) at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434) at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440) at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:191) at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:63) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53) at org.apache.spark.scheduler.Task.run(Task.scala:99) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) Do we have to (painfully) change dimensions size or should it be fixed through a patch ? > ArrayIndexOutOfBoundsException in NDCuboidBuilder > - > > Key: KYLIN-3667 > URL: https://issues.apache.org/jira/browse/KYLIN-3667 > Project: Kylin > Issue Type: Bug > Components: Spark Engine >Affects Versions: v2.5.0 > Environment: AWS EMR >Reporter: Hubert STEFANI >Priority: Major > > The former errors > https://issues.apache.org/jira/browse/KYLIN-3115 > and > https://issues.apache.org/jira/browse/KYLIN-1768 > still remain in step SparkCubingByLayer. > we encounter the following error : > java.lang.ArrayIndexOutOfBoundsException at java.lang.System.arraycopy(Native > Method) > at > org.apache.kylin.engine.mr.common.NDCuboidBuilder.buildKeyInternal(NDCuboidBuilder.java:106) > at > org.apache.kylin.engine.mr.common.NDCuboidBuilder.buildKey2(NDCuboidBuilder.java:87) > at > org.apache.kylin.engine.spark.SparkCubingByLayer$CuboidFlatMap.call(SparkCubingByLayer.java:425) > at > org.apache.kylin.engine.spark.SparkCubingByLayer$CuboidFlatMap.call(SparkCubingByLayer.java:370) > at > org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$3$1.apply(JavaRDDLike.scala:143) > at >
[jira] [Created] (KYLIN-3667) ArrayIndexOutOfBoundsException in NDCuboidBuilder
Hubert STEFANI created KYLIN-3667: - Summary: ArrayIndexOutOfBoundsException in NDCuboidBuilder Key: KYLIN-3667 URL: https://issues.apache.org/jira/browse/KYLIN-3667 Project: Kylin Issue Type: Bug Components: Spark Engine Affects Versions: v2.5.0 Environment: AWS EMR Reporter: Hubert STEFANI The former errors https://issues.apache.org/jira/browse/KYLIN-3115 and https://issues.apache.org/jira/browse/KYLIN-1768 still remains in step SparkCubingByLayer. we encounter the following error : java.lang.ArrayIndexOutOfBoundsException at java.lang.System.arraycopy(Native Method) at org.apache.kylin.engine.mr.common.NDCuboidBuilder.buildKeyInternal(NDCuboidBuilder.java:106) at org.apache.kylin.engine.mr.common.NDCuboidBuilder.buildKey2(NDCuboidBuilder.java:87) at org.apache.kylin.engine.spark.SparkCubingByLayer$CuboidFlatMap.call(SparkCubingByLayer.java:425) at org.apache.kylin.engine.spark.SparkCubingByLayer$CuboidFlatMap.call(SparkCubingByLayer.java:370) at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$3$1.apply(JavaRDDLike.scala:143) at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$3$1.apply(JavaRDDLike.scala:143) at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434) at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440) at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:191) at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:63) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53) at org.apache.spark.scheduler.Task.run(Task.scala:99) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) Do we have to (painfully) change dimensions size or should it be fixed through a patch ? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3666) MERGE CUBE Step 2 Update Dictionary Data Throws IllegalStateException
[
https://issues.apache.org/jira/browse/KYLIN-3666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16675341#comment-16675341
]
Chao Long commented on KYLIN-3666:
--
I will check it.
> MERGE CUBE Step 2 Update Dictionary Data Throws IllegalStateException
> -
>
> Key: KYLIN-3666
> URL: https://issues.apache.org/jira/browse/KYLIN-3666
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: v2.5.0
> Environment: cdh5.11.0
>Reporter: Xi Chen
>Priority: Major
>
> MERGE CUBE job failed in Step 2, throwing such exception:
>
> {code:java}
> java.lang.IllegalStateException: Metadata uri :
> kylin_metadata_my_service@hdfs,path=viewfs://hadoop-footest/kylin/kylin_metadata_my_service/kylin-9583f16b-93cb-9461-a58c-43c848007972/kylin_sales_cube_clone_spark/metadata
> is not recognized
> at org.apache.kylin.common.KylinConfig.decideUriType(KylinConfig.java:172)
> at
> org.apache.kylin.common.KylinConfig.createInstanceFromUri(KylinConfig.java:182)
> at
> org.apache.kylin.engine.mr.steps.UpdateDictionaryStep.doWork(UpdateDictionaryStep.java:70)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:163)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:69)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:163)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:113)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.IllegalStateException: Metadata uri :
> kylin_metadata_my_service@hdfs,path=viewfs://hadoop-footest/kylin/kylin_metadata_my_service/kylin-9583f16b-93cb-9461-a58c-43c848007972/kylin_sales_cube_clone_spark/metadata
> looks like a file but it's neither a file nor a directory
> at org.apache.kylin.common.KylinConfig.decideUriType(KylinConfig.java:162)
> ... 9 more
> {code}
> The metadataUrl is on hdfs, but decideUriType() method does not recognize it.
> Thus throwing such exception.
> Both MR and Spark engine are affected.
> Would [~Wayne0101] please have a look? I see it's related to KYLIN-3471
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3666) MERGE CUBE Step 2 Update Dictionary Data Throws IllegalStateException
Xi Chen created KYLIN-3666:
--
Summary: MERGE CUBE Step 2 Update Dictionary Data Throws
IllegalStateException
Key: KYLIN-3666
URL: https://issues.apache.org/jira/browse/KYLIN-3666
Project: Kylin
Issue Type: Bug
Components: Job Engine
Affects Versions: v2.5.0
Environment: cdh5.11.0
Reporter: Xi Chen
MERGE CUBE job failed in Step 2, throwing such exception:
{code:java}
java.lang.IllegalStateException: Metadata uri :
kylin_metadata_my_service@hdfs,path=viewfs://hadoop-footest/kylin/kylin_metadata_my_service/kylin-9583f16b-93cb-9461-a58c-43c848007972/kylin_sales_cube_clone_spark/metadata
is not recognized
at org.apache.kylin.common.KylinConfig.decideUriType(KylinConfig.java:172)
at
org.apache.kylin.common.KylinConfig.createInstanceFromUri(KylinConfig.java:182)
at
org.apache.kylin.engine.mr.steps.UpdateDictionaryStep.doWork(UpdateDictionaryStep.java:70)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:163)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:69)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:163)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:113)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: Metadata uri :
kylin_metadata_my_service@hdfs,path=viewfs://hadoop-footest/kylin/kylin_metadata_my_service/kylin-9583f16b-93cb-9461-a58c-43c848007972/kylin_sales_cube_clone_spark/metadata
looks like a file but it's neither a file nor a directory
at org.apache.kylin.common.KylinConfig.decideUriType(KylinConfig.java:162)
... 9 more
{code}
The metadataUrl is on hdfs, but decideUriType() method does not recognize it.
Thus throwing such exception.
Both MR and Spark engine are affected.
Would [~Wayne0101] please have a look? I see it's related to KYLIN-3471
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3656) Improve HLLCounter performance
[ https://issues.apache.org/jira/browse/KYLIN-3656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16674917#comment-16674917 ] Chang chen commented on KYLIN-3656: --- [~yimingliu] you can try the UT for the result. The following is my Test : !image-2018-11-05-18-15-36-463.png! Removing reflection isn't include in this patch. > Improve HLLCounter performance > -- > > Key: KYLIN-3656 > URL: https://issues.apache.org/jira/browse/KYLIN-3656 > Project: Kylin > Issue Type: Improvement >Affects Versions: all >Reporter: Chang chen >Assignee: Chang chen >Priority: Major > Fix For: v2.6.0 > > Attachments: 0001-KYLIN-3656-Improve-HLLCounter-performance.patch, > image-2018-11-05-18-15-36-463.png > > > The current HLLCounter implementation has some room to improve performance, > as we find in our product environment. > # Create HLLCounter from another HLLCounter, we can copy register instead > of merge > # To compute harmonic mean in the HLLCSnapshot, we could > ## using table to cache all 1/2^r without computing on the fly > ## remove floating addition by using integer addition in the bigger loop > ## remove branch, e.g. needn't checking whether registers[i] is zero or not, > although this is minor improvement. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3656) Improve HLLCounter performance
[ https://issues.apache.org/jira/browse/KYLIN-3656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chang chen updated KYLIN-3656: -- Attachment: image-2018-11-05-18-15-36-463.png > Improve HLLCounter performance > -- > > Key: KYLIN-3656 > URL: https://issues.apache.org/jira/browse/KYLIN-3656 > Project: Kylin > Issue Type: Improvement >Affects Versions: all >Reporter: Chang chen >Assignee: Chang chen >Priority: Major > Fix For: v2.6.0 > > Attachments: 0001-KYLIN-3656-Improve-HLLCounter-performance.patch, > image-2018-11-05-18-15-36-463.png > > > The current HLLCounter implementation has some room to improve performance, > as we find in our product environment. > # Create HLLCounter from another HLLCounter, we can copy register instead > of merge > # To compute harmonic mean in the HLLCSnapshot, we could > ## using table to cache all 1/2^r without computing on the fly > ## remove floating addition by using integer addition in the bigger loop > ## remove branch, e.g. needn't checking whether registers[i] is zero or not, > although this is minor improvement. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
