[jira] [Created] (KYLIN-3575) Unclosed Connection in DriverTest

[Ted Yu (JIRA)]

Ted Yu created KYLIN-3575:
-

 Summary: Unclosed Connection in DriverTest
 Key: KYLIN-3575
 URL: https://issues.apache.org/jira/browse/KYLIN-3575
 Project: Kylin
  Issue Type: Bug
Reporter: Ted Yu


In testCalciteProps :
{code}
KylinConnection conn = (KylinConnection) 
driver.connect("jdbc:kylin:test_url/test_db", props);
{code}
The connection should be closed upon return.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KYLIN-3574) Unclosed lookup table in DictionaryGeneratorCLI#processSegment

[Ted Yu (JIRA)]

Ted Yu created KYLIN-3574:
-

 Summary: Unclosed lookup table in 
DictionaryGeneratorCLI#processSegment
 Key: KYLIN-3574
 URL: https://issues.apache.org/jira/browse/KYLIN-3574
 Project: Kylin
  Issue Type: Bug
Reporter: Ted Yu


Here is related code:
{code}
JoinDesc join = 
cubeSeg.getModel().getJoinsTree().getJoinByPKSide(lookup);
cubeMgr.getLookupTable(cubeSeg, join);
{code}
The returned lookup table should be closed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] hit-lacus commented on a change in pull request #241: KYLIN-3546 Add column which belongs to root fact table in join relation but lost

[GitBox]

hit-lacus commented on a change in pull request #241: KYLIN-3546 Add column 
which belongs to root fact table in join relation but lost
URL: https://github.com/apache/kylin/pull/241#discussion_r219443869
 
 

 ##
 File path: 
source-kafka/src/main/java/org/apache/kylin/source/kafka/KafkaInputBase.java
 ##
 @@ -89,7 +95,28 @@ public DataModelDesc getDataModel() {
 
 @Override
 public List getAllColumns() {
-return flatDesc.getFactColumns();
+final Set factTableColumnSet = Sets.newHashSet();
+TableRef rootFactTable = getDataModel().getRootFactTable();
+for (TblColRef col : flatDesc.getAllColumns()) {
+if (col.getTableRef().equals(rootFactTable)) {
+factTableColumnSet.add(col);
+}
+}
+// Add column which belongs to root fact table in join 
relation but lost
+for (JoinTableDesc joinTableDesc : 
getDataModel().getJoinTables()) {
+JoinDesc jd = joinTableDesc.getJoin();
+for (TblColRef colRef : jd.getForeignKeyColumns()) {
+if (colRef.getTableRef().equals(rootFactTable)) {
+factTableColumnSet.add(colRef);
+}
+}
+for (TblColRef colRef : jd.getPrimaryKeyColumns()) {
 
 Review comment:
   OK, I will fix it.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] shaofengshi commented on a change in pull request #241: KYLIN-3546 Add column which belongs to root fact table in join relation but lost

[GitBox]

shaofengshi commented on a change in pull request #241: KYLIN-3546 Add column 
which belongs to root fact table in join relation but lost
URL: https://github.com/apache/kylin/pull/241#discussion_r219434659
 
 

 ##
 File path: 
source-kafka/src/main/java/org/apache/kylin/source/kafka/KafkaInputBase.java
 ##
 @@ -89,7 +95,28 @@ public DataModelDesc getDataModel() {
 
 @Override
 public List getAllColumns() {
-return flatDesc.getFactColumns();
+final Set factTableColumnSet = Sets.newHashSet();
+TableRef rootFactTable = getDataModel().getRootFactTable();
+for (TblColRef col : flatDesc.getAllColumns()) {
+if (col.getTableRef().equals(rootFactTable)) {
+factTableColumnSet.add(col);
+}
+}
+// Add column which belongs to root fact table in join 
relation but lost
+for (JoinTableDesc joinTableDesc : 
getDataModel().getJoinTables()) {
+JoinDesc jd = joinTableDesc.getJoin();
+for (TblColRef colRef : jd.getForeignKeyColumns()) {
+if (colRef.getTableRef().equals(rootFactTable)) {
+factTableColumnSet.add(colRef);
+}
+}
+for (TblColRef colRef : jd.getPrimaryKeyColumns()) {
 
 Review comment:
   PK columns are on lookup tables, they won't be fact table, so this loop has 
no necessary to be here.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] shaofengshi closed pull request #199: KYLIN-3497 Make JDBC Module more testable

[GitBox]

shaofengshi closed pull request #199: KYLIN-3497 Make JDBC Module more testable
URL: https://github.com/apache/kylin/pull/199
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/jdbc/pom.xml b/jdbc/pom.xml
index 661356c412..a04d81b4c5 100644
--- a/jdbc/pom.xml
+++ b/jdbc/pom.xml
@@ -52,6 +52,13 @@
 test
 
 
+
+org.mockito
+mockito-core
+${mockito.version}
+test
+
+
 
 
 
diff --git a/jdbc/src/main/java/org/apache/kylin/jdbc/IRemoteClient.java 
b/jdbc/src/main/java/org/apache/kylin/jdbc/IRemoteClient.java
index dfd8d76719..630eedd1fc 100644
--- a/jdbc/src/main/java/org/apache/kylin/jdbc/IRemoteClient.java
+++ b/jdbc/src/main/java/org/apache/kylin/jdbc/IRemoteClient.java
@@ -23,13 +23,12 @@
 import java.util.List;
 import java.util.Map;
 
-import org.apache.calcite.avatica.AvaticaParameter;
 import org.apache.calcite.avatica.ColumnMetaData;
 import org.apache.kylin.jdbc.KylinMeta.KMetaProject;
 
 public interface IRemoteClient extends Closeable {
 
-public static class QueryResult {
+class QueryResult {
 public final List columnMeta;
 public final Iterable iterable;
 
@@ -42,16 +41,16 @@ public QueryResult(List columnMeta, 
Iterable iterable) {
 /**
  * Connect to Kylin restful service. IOException will be thrown if 
authentication failed.
  */
-public void connect() throws IOException;
+void connect() throws IOException;
 
 /**
  * Retrieve meta data of given project.
  */
-public KMetaProject retrieveMetaData(String project) throws IOException;
+KMetaProject retrieveMetaData(String project) throws IOException;
 
 /**
  * Execute query remotely and get back result.
  */
-public QueryResult executeQuery(String sql, List params, 
List paramValues, Map queryToggles) throws IOException;
+QueryResult executeQuery(String sql, List paramValues, Map queryToggles) throws IOException;
 
 }
diff --git a/jdbc/src/main/java/org/apache/kylin/jdbc/JdbcFactory.java 
b/jdbc/src/main/java/org/apache/kylin/jdbc/JdbcFactory.java
new file mode 100644
index 00..f95a30125a
--- /dev/null
+++ b/jdbc/src/main/java/org/apache/kylin/jdbc/JdbcFactory.java
@@ -0,0 +1,24 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kylin.jdbc;
+
+import org.apache.calcite.avatica.AvaticaFactory;
+
+public interface JdbcFactory extends RemoteClientFactory, AvaticaFactory {
+}
diff --git a/jdbc/src/main/java/org/apache/kylin/jdbc/KylinClient.java 
b/jdbc/src/main/java/org/apache/kylin/jdbc/KylinClient.java
index 9f8c7dd24e..b98b2d108c 100644
--- a/jdbc/src/main/java/org/apache/kylin/jdbc/KylinClient.java
+++ b/jdbc/src/main/java/org/apache/kylin/jdbc/KylinClient.java
@@ -34,11 +34,11 @@
 
 import javax.xml.bind.DatatypeConverter;
 
-import org.apache.calcite.avatica.AvaticaParameter;
 import org.apache.calcite.avatica.ColumnMetaData;
 import org.apache.calcite.avatica.ColumnMetaData.Rep;
 import org.apache.calcite.avatica.ColumnMetaData.ScalarType;
 import org.apache.http.HttpResponse;
+import org.apache.http.client.HttpClient;
 import org.apache.http.client.methods.HttpGet;
 import org.apache.http.client.methods.HttpPost;
 import org.apache.http.client.methods.HttpRequestBase;
@@ -64,19 +64,20 @@
 
 import com.fasterxml.jackson.core.type.TypeReference;
 import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.annotations.VisibleForTesting;
 
 public class KylinClient implements IRemoteClient {
 
 private static final Logger logger = 
LoggerFactory.getLogger(KylinClient.class);
 
-private final KylinConnection conn;
+private final KylinConnectionInfo connInfo;
 private final Properties connProps;
-private DefaultHttpClient httpClient;
+private HttpClient httpClient;
 private final ObjectMapper jsonMapper;
 
-public KylinClient(KylinConnection conn) {
-this.conn = conn;
-this.connProps = 

[GitHub] shaofengshi closed pull request #219: KYLIN-3515 Add uuid for materialized table of hive view

[GitBox]

shaofengshi closed pull request #219: KYLIN-3515 Add uuid for materialized 
table of hive view
URL: https://github.com/apache/kylin/pull/219
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/core-cube/src/main/java/org/apache/kylin/cube/CubeManager.java 
b/core-cube/src/main/java/org/apache/kylin/cube/CubeManager.java
index 23dcaf37eb..a774c0d333 100755
--- a/core-cube/src/main/java/org/apache/kylin/cube/CubeManager.java
+++ b/core-cube/src/main/java/org/apache/kylin/cube/CubeManager.java
@@ -1023,8 +1023,8 @@ public DictionaryInfo saveDictionary(CubeSegment cubeSeg, 
TblColRef col, IReadab
 return dictAssist.getDictionary(cubeSeg, col);
 }
 
-public SnapshotTable buildSnapshotTable(CubeSegment cubeSeg, String 
lookupTable) throws IOException {
-return dictAssist.buildSnapshotTable(cubeSeg, lookupTable);
+public SnapshotTable buildSnapshotTable(CubeSegment cubeSeg, String 
lookupTable, String uuid) throws IOException {
+return dictAssist.buildSnapshotTable(cubeSeg, lookupTable, uuid);
 }
 
 private TableMetadataManager getMetadataManager() {
@@ -1098,7 +1098,7 @@ private void saveDictionaryInfo(CubeSegment cubeSeg, 
TblColRef col, DictionaryIn
 return (Dictionary) info.getDictionaryObject();
 }
 
-public SnapshotTable buildSnapshotTable(CubeSegment cubeSeg, String 
lookupTable) throws IOException {
+public SnapshotTable buildSnapshotTable(CubeSegment cubeSeg, String 
lookupTable, String uuid) throws IOException {
 // work on copy instead of cached objects
 CubeInstance cubeCopy = 
cubeSeg.getCubeInstance().latestCopyForWrite(); // get a latest copy
 CubeSegment segCopy = cubeCopy.getSegmentById(cubeSeg.getUuid());
@@ -1107,7 +1107,7 @@ public SnapshotTable buildSnapshotTable(CubeSegment 
cubeSeg, String lookupTable)
 SnapshotManager snapshotMgr = getSnapshotManager();
 
 TableDesc tableDesc = new 
TableDesc(metaMgr.getTableDesc(lookupTable, segCopy.getProject()));
-IReadableTable hiveTable = 
SourceManager.createReadableTable(tableDesc);
+IReadableTable hiveTable = 
SourceManager.createReadableTable(tableDesc, uuid);
 SnapshotTable snapshot = snapshotMgr.buildSnapshot(hiveTable, 
tableDesc, cubeSeg.getConfig());
 
 CubeDesc cubeDesc = cubeSeg.getCubeDesc();
diff --git 
a/core-cube/src/main/java/org/apache/kylin/cube/cli/DictionaryGeneratorCLI.java 
b/core-cube/src/main/java/org/apache/kylin/cube/cli/DictionaryGeneratorCLI.java
index 7fcf3208d1..6de42ac3bd 100644
--- 
a/core-cube/src/main/java/org/apache/kylin/cube/cli/DictionaryGeneratorCLI.java
+++ 
b/core-cube/src/main/java/org/apache/kylin/cube/cli/DictionaryGeneratorCLI.java
@@ -42,26 +42,28 @@
 
 private static final Logger logger = 
LoggerFactory.getLogger(DictionaryGeneratorCLI.class);
 
-public static void processSegment(KylinConfig config, String cubeName, 
String segmentID, DistinctColumnValuesProvider factTableValueProvider, 
DictionaryProvider dictProvider) throws IOException {
+public static void processSegment(KylinConfig config, String cubeName, 
String segmentID, String uuid,
+DistinctColumnValuesProvider factTableValueProvider, 
DictionaryProvider dictProvider) throws IOException {
 CubeInstance cube = CubeManager.getInstance(config).getCube(cubeName);
 CubeSegment segment = cube.getSegmentById(segmentID);
 
-processSegment(config, segment, factTableValueProvider, dictProvider);
+processSegment(config, segment, uuid, factTableValueProvider, 
dictProvider);
 }
 
-private static void processSegment(KylinConfig config, CubeSegment 
cubeSeg, DistinctColumnValuesProvider factTableValueProvider, 
DictionaryProvider dictProvider) throws IOException {
+private static void processSegment(KylinConfig config, CubeSegment 
cubeSeg, String uuid,
+DistinctColumnValuesProvider factTableValueProvider, 
DictionaryProvider dictProvider) throws IOException {
 CubeManager cubeMgr = CubeManager.getInstance(config);
 
 // dictionary
 for (TblColRef col : 
cubeSeg.getCubeDesc().getAllColumnsNeedDictionaryBuilt()) {
 logger.info("Building dictionary for " + col);
 IReadableTable inpTable = 
factTableValueProvider.getDistinctValuesFor(col);
-
+
 Dictionary preBuiltDict = null;
 if (dictProvider != null) {
 preBuiltDict = dictProvider.getDictionary(col);
 }
-
+
 if (preBuiltDict != null) {
 logger.debug("Dict for '" + col.getName() + "' has already 
been built, save it");
 cubeMgr.saveDictionary(cubeSeg, 

Re:Kylin concurrent issue for query engine;

[Ma Gang]

Could you run jstack on the Kylin process and send out result? So that we can 
know what the query threads are blocked on.
I encountered the same issue when doing load test, and found that most of query 
threads are blocked on class loading, and the class is code gen by calcite, so 
we may need calcite to fix it(Just a simple fix to change the generated class 
name). If you use Kylin 2.5 version, and most of your queries have similar 
pattern, you may try to use PreparedStatement, that may greatly increase the 
load performance. Kylin already support to cached preparedStatement parse 
result in Kylin server side since 2.5 
(https://issues.apache.org/jira/browse/KYLIN-3434)

At 2018-09-20 14:30:56, "huaicui" <270922...@qq.com> wrote:
>We use Kylin to design our project; But we have a problem:
>A single query will be responded in 0.07 to 0.2 seconds. Following the
>number of query concurrency, response time will be longer. When the number
>of query concurrency is about 150, this value will be extended to 4 seconds
>or more. Can you help me to fix this issue or give me some suggestion?
>
> 
> 
> 
> 
>
>--
>Sent from: http://apache-kylin.74782.x6.nabble.com/

[GitHub] hit-lacus opened a new pull request #241: KYLIN-3546 Add column which belongs to root fact table in join relation but lost

[GitBox]

hit-lacus opened a new pull request #241: KYLIN-3546 Add column which belongs 
to root fact table in join relation but lost
URL: https://github.com/apache/kylin/pull/241
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] asfgit commented on issue #241: KYLIN-3546 Add column which belongs to root fact table in join relation but lost

[GitBox]

asfgit commented on issue #241: KYLIN-3546 Add column which belongs to root 
fact table in join relation but lost
URL: https://github.com/apache/kylin/pull/241#issuecomment-423427023
 
 
   Can one of the admins verify this patch?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] asfgit commented on issue #241: KYLIN-3546 Add column which belongs to root fact table in join relation but lost

[GitBox]

asfgit commented on issue #241: KYLIN-3546 Add column which belongs to root 
fact table in join relation but lost
URL: https://github.com/apache/kylin/pull/241#issuecomment-423427022
 
 
   Can one of the admins verify this patch?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services