[kylin] branch realtime-streaming updated: KYLIN-3791 Map return by Maps.transformValues is a immutable view
This is an automated email from the ASF dual-hosted git repository.
magang pushed a commit to branch realtime-streaming
in repository https://gitbox.apache.org/repos/asf/kylin.git
The following commit(s) were added to refs/heads/realtime-streaming by this
push:
new 67101ce KYLIN-3791 Map return by Maps.transformValues is a immutable
view
67101ce is described below
commit 67101ce6c3f3ab8dcad855cd98e6c97deda063f0
Author: hit-lacus
AuthorDate: Sat Jan 26 21:47:37 2019 +0800
KYLIN-3791 Map return by Maps.transformValues is a immutable view
---
.../stream/core/storage/StreamingSegmentManager.java | 18 --
1 file changed, 12 insertions(+), 6 deletions(-)
diff --git
a/stream-core/src/main/java/org/apache/kylin/stream/core/storage/StreamingSegmentManager.java
b/stream-core/src/main/java/org/apache/kylin/stream/core/storage/StreamingSegmentManager.java
index 28a294d..38c53fe 100644
---
a/stream-core/src/main/java/org/apache/kylin/stream/core/storage/StreamingSegmentManager.java
+++
b/stream-core/src/main/java/org/apache/kylin/stream/core/storage/StreamingSegmentManager.java
@@ -62,12 +62,18 @@ public class StreamingSegmentManager implements Closeable {
private final String cubeName;
private final CubeInstance cubeInstance;
-//Cube window defines how streaming events are divided and put into
different segments , for example 1 hour per segment(indexer).
-//If the received events' timestamp is completely out of order and belongs
to a very wide range, there will be multiple active segment indexers created
and serve the indexing and querying.
+/**
+ * Cube window defines how streaming events are divided and put into
different segments , for example 1 hour per segment(indexer).
+ * If the received events' timestamp is completely out of order and
belongs to a very wide range,
+ * there will be multiple active segment indexers created and serve the
indexing and querying.
+ * */
private final long cubeWindow;
-//Cube duration defines how long the oldest streaming segment becomes
immutable and does not allow additional modification.
-//Any further long latency events that can't find a corresponding segment
to serve the index, the events will be put to a specific segment for long
latency events only.
+/**
+ * Cube duration defines how long the oldest streaming segment becomes
immutable and does not allow additional modification.
+ * Any further long latency events that can't find a corresponding segment
to serve the index,
+ * the events will be put to a specific segment for long latency events
only.
+ * */
private final long cubeDuration;
private final long maxCubeDuration;
@@ -223,13 +229,13 @@ public class StreamingSegmentManager implements Closeable
{
private void restoreSegmentsFromCP(List segmentFolders, Map checkpointStoreStats,
Map
segmentSourceStartPositions, CubeSegment latestRemoteSegment) {
if (segmentSourceStartPositions != null) {
-this.segmentSourceStartPositions =
Maps.transformValues(segmentSourceStartPositions, new Function() {
+
this.segmentSourceStartPositions.putAll(Maps.transformValues(segmentSourceStartPositions,
new Function() {
@Nullable
@Override
public ISourcePosition apply(@Nullable String input) {
return sourcePositionHandler.parsePosition(input);
}
-});
+}));
}
for (File segmentFolder : segmentFolders) {
try {
[kylin] branch document updated: minor add more detail for data source sdk
This is an automated email from the ASF dual-hosted git repository.
shaofengshi pushed a commit to branch document
in repository https://gitbox.apache.org/repos/asf/kylin.git
The following commit(s) were added to refs/heads/document by this push:
new 8f67b78 minor add more detail for data source sdk
8f67b78 is described below
commit 8f67b78d2f78055d917e4c987ae76e21dee5dee6
Author: edouardzyc
AuthorDate: Mon Jan 28 11:22:19 2019 +0800
minor add more detail for data source sdk
---
website/_dev/datasource_sdk.cn.md | 80 +---
website/_dev/datasource_sdk.md| 85 +++
2 files changed, 152 insertions(+), 13 deletions(-)
diff --git a/website/_dev/datasource_sdk.cn.md
b/website/_dev/datasource_sdk.cn.md
index 0a50652..80d6ded 100644
--- a/website/_dev/datasource_sdk.cn.md
+++ b/website/_dev/datasource_sdk.cn.md
@@ -19,8 +19,78 @@ permalink: /cn/development/datasource_sdk.html
开发者不需要编码, 只需要为新的数据源新建一个配置文件 *{dialect}.xml*,
配置文件结构:
-* 根节点:
-DATASOURCE_DEF NAME="kylin" ID="default", ID的值为方言的名称.
+* 根节点:
+
+```
+
+```
+
+ID的值通常和配置文件的名字相同。
+DIALECT的值的定义主要是为了区分不同数据库对于标识符的引用。
+举个例子 Mysql 使用 ``, Microsoft sql server 使用 [].
+Kylin 里定义的DIALECT 和 Apache Calcite 里定义DIALECT 的对应关系:
+
+
+
+ Kylin 里定义的方言
+ Apache Calcite 里定义的方言
+
+
+ default
+ SqlDialect.CALCITE
+
+
+ calcite
+ SqlDialect.CALCITE
+
+
+ greenplum
+ SqlDialect.DatabaseProduct.POSTGRESQL
+
+
+ postgresql
+ SqlDialect.DatabaseProduct.POSTGRESQL
+
+
+ mysql
+ SqlDialect.DatabaseProduct.MYSQL
+
+
+ sql.keyword-default-uppercase
+ whether default should be transform to uppercase
+
+
+ mssql
+ SqlDialect.DatabaseProduct.MSSQL
+
+
+ oracle
+ SqlDialect.DatabaseProduct.ORACLE
+
+
+ vertica
+ SqlDialect.DatabaseProduct.VERTICA
+
+
+ redshift
+ SqlDialect.DatabaseProduct.REDSHIFT
+
+
+ hive
+ SqlDialect.DatabaseProduct.HIVE
+
+
+ h2
+ SqlDialect.DatabaseProduct.H2
+
+
+ unkown
+ SqlDialect.DUMMY
+
+
+
+
+
* 属性节点:
定义方言的属性。
@@ -141,9 +211,9 @@
kylin.query.pushdown.runner-class-name=org.apache.kylin.query.pushdown.PushdownR
kylin.source.default=16
kylin.source.jdbc.dialect=mysql
kylin.source.jdbc.adaptor=org.apache.kylin.sdk.datasource.adaptor.MysqlAdaptor
-kylin.source.jdbc.user={mysql username}
-kylin.source.jdbc.pass={mysql password}
-kylin.source.jdbc.connection-url=jdbc:mysql://{HOST_URL}:3306/{Database name}
+kylin.source.jdbc.user={mysql 用户名}
+kylin.source.jdbc.pass={mysql 用户密码}
+kylin.source.jdbc.connection-url=jdbc:mysql://{主机url}:3306/{数据库名称}
kylin.source.jdbc.driver=com.mysql.jdbc.Driver
{% endhighlight %}
diff --git a/website/_dev/datasource_sdk.md b/website/_dev/datasource_sdk.md
index 9d8258d..94fa310 100644
--- a/website/_dev/datasource_sdk.md
+++ b/website/_dev/datasource_sdk.md
@@ -5,7 +5,7 @@ categories: development
permalink: /development/datasource_sdk.html
---
- Available since Apache Kylin v2.6.0
+> Available since Apache Kylin v2.6.0
## Data source SDK
@@ -21,16 +21,85 @@ Developers do not need coding, what they should do is just
create a new configur
Structure of the configuration:
-* Root node:
-DATASOURCE_DEF NAME="kylin" ID="default", the value of ID should be
name of dialect.
+* Root node:
+
+```
+
+```
+
+The value of ID is normally the same with configuration file.
+The value of DIALECT is defined mainly for quote string for database
identifier.
+For example Mysql use ``, Microsoft sql server use [].
+Mapping of Kylin DIALECT and Apache Calcite Dialect as belows:
+
+
+
+Dialect in Kylin
+Dialect in Apache Calcite
+
+
+ default
+ SqlDialect.CALCITE
+
+
+ calcite
+ SqlDialect.CALCITE
+
+
+ greenplum
+ SqlDialect.DatabaseProduct.POSTGRESQL
+
+
+ postgresql
+ SqlDialect.DatabaseProduct.POSTGRESQL
+
+
+ mysql
+ SqlDialect.DatabaseProduct.MYSQL
+
+
+ sql.keyword-default-uppercase
+ whether default should be transform to uppercase
+
+
+ mssql
+ SqlDialect.DatabaseProduct.MSSQL
+
+
+ oracle
+ SqlDialect.DatabaseProduct.ORACLE
+
+
+ vertica
+ SqlDialect.DatabaseProduct.VERTICA
+
+
+ redshift
+ SqlDialect.DatabaseProduct.REDSHIFT
+
+
+ hive
+ SqlDialect.DatabaseProduct.HIVE
+
+
+ h2
+ SqlDialect.DatabaseProduct.H2
+
+
+ unkown
+ SqlDialect.DUMMY
+
+
+
+
* Property node:
Define the properties of the dialect.
-属性
-描述
+Property
+Description
sql.default-converted-enabled
@@ -143,9 +212,9 @@
kylin.query.pushdown.runner-class-name=org.apache.kylin.query.pushdown.PushdownR
kylin.source.default=16
kylin.source.jdbc.dialect=mysql
[kylin] branch master updated: KYLIN-2725 fix cube desc signature issue
This is an automated email from the ASF dual-hosted git repository.
nju_yaho pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kylin.git
The following commit(s) were added to refs/heads/master by this push:
new 7437af8 KYLIN-2725 fix cube desc signature issue
7437af8 is described below
commit 7437af838c5fe1fc9eb2055cc5e9f81e5dc4aab5
Author: kyotoYaho
AuthorDate: Mon Jan 28 12:06:09 2019 +0800
KYLIN-2725 fix cube desc signature issue
---
.../java/org/apache/kylin/tool/metrics/systemcube/CubeDescCreator.java | 1 -
1 file changed, 1 deletion(-)
diff --git
a/tool/src/main/java/org/apache/kylin/tool/metrics/systemcube/CubeDescCreator.java
b/tool/src/main/java/org/apache/kylin/tool/metrics/systemcube/CubeDescCreator.java
index 9eb5ab2..f525e44 100644
---
a/tool/src/main/java/org/apache/kylin/tool/metrics/systemcube/CubeDescCreator.java
+++
b/tool/src/main/java/org/apache/kylin/tool/metrics/systemcube/CubeDescCreator.java
@@ -455,7 +455,6 @@ public class CubeDescCreator {
desc.setStorageType(storageType);
desc.setAggregationGroups(Lists.newArrayList(aggGroup));
desc.getOverrideKylinProps().putAll(overrideProperties);
-desc.setSignature(desc.calculateSignature());
desc.updateRandomUuid();
return desc;
}
[kylin] branch 2.5.x updated: KYLIN 3790 return wrong error code when command executor is interrupted
This is an automated email from the ASF dual-hosted git repository.
shaofengshi pushed a commit to branch 2.5.x
in repository https://gitbox.apache.org/repos/asf/kylin.git
The following commit(s) were added to refs/heads/2.5.x by this push:
new 9b1e516 KYLIN 3790 return wrong error code when command executor is
interrupted
9b1e516 is described below
commit 9b1e516bfd66827536690ec6bbd566c308f3fb81
Author: chenzhx
AuthorDate: Fri Jan 25 19:08:24 2019 +0800
KYLIN 3790 return wrong error code when command executor is interrupted
---
.../src/main/java/org/apache/kylin/common/util/CliCommandExecutor.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/core-common/src/main/java/org/apache/kylin/common/util/CliCommandExecutor.java
b/core-common/src/main/java/org/apache/kylin/common/util/CliCommandExecutor.java
index 5fef77b..6b8cf76 100644
---
a/core-common/src/main/java/org/apache/kylin/common/util/CliCommandExecutor.java
+++
b/core-common/src/main/java/org/apache/kylin/common/util/CliCommandExecutor.java
@@ -151,7 +151,7 @@ public class CliCommandExecutor {
} catch (InterruptedException e) {
// do nothing
}
-return Pair.newPair(0, "Killed");
+return Pair.newPair(1, "Killed");
}
try {
[kylin] branch master updated: KYLIN 3790 return wrong error code when command executor is interrupted
This is an automated email from the ASF dual-hosted git repository.
shaofengshi pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kylin.git
The following commit(s) were added to refs/heads/master by this push:
new 2cea735 KYLIN 3790 return wrong error code when command executor is
interrupted
2cea735 is described below
commit 2cea735c48f1f708ebb863d3d7a48f5c9ce6703c
Author: chenzhx
AuthorDate: Fri Jan 25 19:08:24 2019 +0800
KYLIN 3790 return wrong error code when command executor is interrupted
---
.../src/main/java/org/apache/kylin/common/util/CliCommandExecutor.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/core-common/src/main/java/org/apache/kylin/common/util/CliCommandExecutor.java
b/core-common/src/main/java/org/apache/kylin/common/util/CliCommandExecutor.java
index 5fef77b..6b8cf76 100644
---
a/core-common/src/main/java/org/apache/kylin/common/util/CliCommandExecutor.java
+++
b/core-common/src/main/java/org/apache/kylin/common/util/CliCommandExecutor.java
@@ -151,7 +151,7 @@ public class CliCommandExecutor {
} catch (InterruptedException e) {
// do nothing
}
-return Pair.newPair(0, "Killed");
+return Pair.newPair(1, "Killed");
}
try {
[kylin] branch 2.4.x updated: KYLIN-3494 ArrayIndexOutOfBoundsException may occur in streaming cube building
This is an automated email from the ASF dual-hosted git repository.
shaofengshi pushed a commit to branch 2.4.x
in repository https://gitbox.apache.org/repos/asf/kylin.git
The following commit(s) were added to refs/heads/2.4.x by this push:
new af8719c KYLIN-3494 ArrayIndexOutOfBoundsException may occur in
streaming cube building
af8719c is described below
commit af8719c8cf0934a6fe58f4f35e89a62733ac42fb
Author: shaofengshi
AuthorDate: Mon Jan 28 08:04:09 2019 +0800
KYLIN-3494 ArrayIndexOutOfBoundsException may occur in streaming cube
building
---
.../src/main/java/org/apache/kylin/source/kafka/KafkaMRInput.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/source-kafka/src/main/java/org/apache/kylin/source/kafka/KafkaMRInput.java
b/source-kafka/src/main/java/org/apache/kylin/source/kafka/KafkaMRInput.java
index bc2426d..ad37e7f 100644
--- a/source-kafka/src/main/java/org/apache/kylin/source/kafka/KafkaMRInput.java
+++ b/source-kafka/src/main/java/org/apache/kylin/source/kafka/KafkaMRInput.java
@@ -110,7 +110,7 @@ public class KafkaMRInput implements IMRInput {
@Override
public Collection parseMapperInput(Object mapperInput) {
Text text = (Text) mapperInput;
-String[] columns = Bytes.toString(text.getBytes(), 0,
text.getLength()).split(delimiter);
+String[] columns = Bytes.toString(text.getBytes(), 0,
text.getLength()).split(delimiter, -1);
return Collections.singletonList(columns);
}
[kylin] branch 2.6.x updated: KYLIN-3494 ArrayIndexOutOfBoundsException may occur in streaming cube building
This is an automated email from the ASF dual-hosted git repository.
shaofengshi pushed a commit to branch 2.6.x
in repository https://gitbox.apache.org/repos/asf/kylin.git
The following commit(s) were added to refs/heads/2.6.x by this push:
new 990ec5d KYLIN-3494 ArrayIndexOutOfBoundsException may occur in
streaming cube building
990ec5d is described below
commit 990ec5dac4fb175b0c4d2dab609794cd26f4e62a
Author: shaofengshi
AuthorDate: Mon Jan 28 08:04:09 2019 +0800
KYLIN-3494 ArrayIndexOutOfBoundsException may occur in streaming cube
building
---
.../src/main/java/org/apache/kylin/source/kafka/KafkaMRInput.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/source-kafka/src/main/java/org/apache/kylin/source/kafka/KafkaMRInput.java
b/source-kafka/src/main/java/org/apache/kylin/source/kafka/KafkaMRInput.java
index d709572..6f99e29 100644
--- a/source-kafka/src/main/java/org/apache/kylin/source/kafka/KafkaMRInput.java
+++ b/source-kafka/src/main/java/org/apache/kylin/source/kafka/KafkaMRInput.java
@@ -87,7 +87,7 @@ public class KafkaMRInput extends KafkaInputBase implements
IMRInput {
@Override
public Collection parseMapperInput(Object mapperInput) {
Text text = (Text) mapperInput;
-String[] columns = Bytes.toString(text.getBytes(), 0,
text.getLength()).split(delimiter);
+String[] columns = Bytes.toString(text.getBytes(), 0,
text.getLength()).split(delimiter, -1);
return Collections.singletonList(columns);
}
[kylin] branch 2.5.x updated: KYLIN-3494 ArrayIndexOutOfBoundsException may occur in streaming cube building
This is an automated email from the ASF dual-hosted git repository.
shaofengshi pushed a commit to branch 2.5.x
in repository https://gitbox.apache.org/repos/asf/kylin.git
The following commit(s) were added to refs/heads/2.5.x by this push:
new 710d5dc KYLIN-3494 ArrayIndexOutOfBoundsException may occur in
streaming cube building
710d5dc is described below
commit 710d5dc514ef3bcf87fef87101324ba47da20beb
Author: shaofengshi
AuthorDate: Mon Jan 28 08:04:09 2019 +0800
KYLIN-3494 ArrayIndexOutOfBoundsException may occur in streaming cube
building
---
.../src/main/java/org/apache/kylin/source/kafka/KafkaMRInput.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/source-kafka/src/main/java/org/apache/kylin/source/kafka/KafkaMRInput.java
b/source-kafka/src/main/java/org/apache/kylin/source/kafka/KafkaMRInput.java
index 1c94f9c..90f9b70 100644
--- a/source-kafka/src/main/java/org/apache/kylin/source/kafka/KafkaMRInput.java
+++ b/source-kafka/src/main/java/org/apache/kylin/source/kafka/KafkaMRInput.java
@@ -99,7 +99,7 @@ public class KafkaMRInput extends KafkaInputBase implements
IMRInput {
@Override
public Collection parseMapperInput(Object mapperInput) {
Text text = (Text) mapperInput;
-String[] columns = Bytes.toString(text.getBytes(), 0,
text.getLength()).split(delimiter);
+String[] columns = Bytes.toString(text.getBytes(), 0,
text.getLength()).split(delimiter, -1);
return Collections.singletonList(columns);
}
[kylin] branch 2.5.x updated (99587c4 -> a2db3e5)
This is an automated email from the ASF dual-hosted git repository.
shaofengshi pushed a change to branch 2.5.x
in repository https://gitbox.apache.org/repos/asf/kylin.git.
from 99587c4 KYLIN-3570 Automatically build system cube
new 1ebdaa9 KYLIN-3699 Spark cubing failed when build with empty data
new 8489982 KYLIN-3720 add column family check when save/update cube desc
new fb2522e KYLIN-3738 Edit cube measure may make the decimal type change
unexpectly revert KYLIN-2243 8c0c44b887e2caa21b097c2334f8d21c42462e80
new 72a9a9f KYLIN-3729 CLUSTER BY CAST(field AS STRING) will accelerate
base cuboid build with UHC global dict
new b07575a KYLIN-3643 Inner column filter can not pushdown
new c628cb7 KYLIN-3731 Segment pruning ignore complex data type
new 733ef38 KYLIN-3748 No realization found exception thrown when a ready
cube is building
new af6e685 KYLIN-3722 Disable limit push down for multi olapcontext
new 4e36aba KYLIN-3767 Print the malformed JSON data consumed from Kafka
Topic
new 4370f57 KYLIN-3772 CubeMetaIngester works abnormally
new 4456fb1 KYLIN-3770 ZipFileUtils is not compatible CubeMetaIngester
new 2383f6b KYLIN-3703 get negative value when query kylin
new a2db3e5 KYLIN-3494: Build cube with spark reports
ArrayIndexOutOfBoundsException
The 7462 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../org/apache/kylin/common/KylinConfigBase.java | 4 +++
.../org/apache/kylin/common/util/HadoopUtil.java | 24
.../apache/kylin/cube/CubeCapabilityChecker.java | 21 --
.../apache/kylin/cube/common/SegmentPruner.java| 4 +++
.../java/org/apache/kylin/gridtable/GTUtil.java| 4 +++
.../java/org/apache/kylin/job/JoinedFlatTable.java | 2 +-
.../apache/kylin/metadata/datatype/DataType.java | 9 +-
.../apache/kylin/metadata/model/FunctionDesc.java | 2 +-
.../kylin/engine/mr/common/CubeStatsReader.java| 2 +-
.../mr/steps/FactDistinctColumnsReducer.java | 2 +-
.../mr/steps/UpdateCubeInfoAfterBuildStep.java | 3 ++
.../kylin/engine/spark/SparkFactDistinct.java | 2 +-
.../org/apache/kylin/engine/spark/SparkUtil.java | 33 +-
.../org/apache/kylin/query/ITKylinQueryTest.java | 17 ++-
.../query01.sql => sql_limit/query06.sql} | 10 +--
.../apache/kylin/query/relnode/OLAPContext.java| 2 ++
.../apache/kylin/query/relnode/OLAPLimitRel.java | 3 +-
.../org/apache/kylin/query/relnode/OLAPRel.java| 9 --
.../kylin/rest/controller/CubeController.java | 28 ++
.../kylin/source/kafka/TimedJsonStreamParser.java | 2 ++
.../apache/kylin/tool/AbstractInfoExtractor.java | 12
.../org/apache/kylin/tool/CubeMetaExtractor.java | 2 ++
webapp/app/js/controllers/cubeMeasures.js | 1 -
23 files changed, 169 insertions(+), 29 deletions(-)
copy kylin-it/src/test/resources/query/{sql_expression/query01.sql =>
sql_limit/query06.sql} (79%)
