[GitHub] shaofengshi commented on issue #276: KYLIN-3605 upgrade hadoop-common and zookeeper version to fix securit…
shaofengshi commented on issue #276: KYLIN-3605 upgrade hadoop-common and zookeeper version to fix securit… URL: https://github.com/apache/kylin/pull/276#issuecomment-425696064 Merged. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] shaofengshi closed pull request #276: KYLIN-3605 upgrade hadoop-common and zookeeper version to fix securit…
shaofengshi closed pull request #276: KYLIN-3605 upgrade hadoop-common and
zookeeper version to fix securit…
URL: https://github.com/apache/kylin/pull/276
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/LICENSE b/LICENSE
index a237c61e69..26e015cf79 100644
--- a/LICENSE
+++ b/LICENSE
@@ -265,5 +265,8 @@ RocksDB is dual-licensed under both the GPLv2 and Apache
2.0 License.You may
select, at your option, one of the above-listed licenses.
https://github.com/facebook/rocksdb/blob/master/LICENSE.Apache
-
-
+==
+For HadoopMetrics2Reporter:
+==
+HadoopMetrics2Reporter is licensed under Apache 2.0 License.
+https://github.com/joshelser/dropwizard-hadoop-metrics2
diff --git a/core-common/pom.xml b/core-common/pom.xml
index 3b5f7fc16b..594e39b39b 100644
--- a/core-common/pom.xml
+++ b/core-common/pom.xml
@@ -86,11 +86,6 @@
${dropwizard.version}
-
-com.github.joshelser
-
dropwizard-metrics-hadoop-metrics2-reporter
-0.1.2
-
commons-dbcp
commons-dbcp
@@ -101,5 +96,12 @@
mysql-connector-java
provided
+
+
+org.mockito
+mockito-core
+${mockito.version}
+test
+
diff --git
a/core-common/src/main/java/org/apache/kylin/common/metrics/metrics2/HadoopMetrics2Reporter.java
b/core-common/src/main/java/org/apache/kylin/common/metrics/metrics2/HadoopMetrics2Reporter.java
new file mode 100644
index 00..c0e48a6b7e
--- /dev/null
+++
b/core-common/src/main/java/org/apache/kylin/common/metrics/metrics2/HadoopMetrics2Reporter.java
@@ -0,0 +1,495 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kylin.common.metrics.metrics2;
+
+import java.text.SimpleDateFormat;
+import java.util.Collections;
+import java.util.Date;
+import java.util.Iterator;
+import java.util.Locale;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.SortedMap;
+import java.util.TreeMap;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.hadoop.metrics2.MetricsCollector;
+import org.apache.hadoop.metrics2.MetricsInfo;
+import org.apache.hadoop.metrics2.MetricsRecordBuilder;
+import org.apache.hadoop.metrics2.MetricsSource;
+import org.apache.hadoop.metrics2.MetricsSystem;
+import org.apache.hadoop.metrics2.lib.Interns;
+import org.apache.hadoop.metrics2.lib.MetricsRegistry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.codahale.metrics.Counter;
+import com.codahale.metrics.Gauge;
+import com.codahale.metrics.Histogram;
+import com.codahale.metrics.Meter;
+import com.codahale.metrics.MetricFilter;
+import com.codahale.metrics.MetricRegistry;
+import com.codahale.metrics.ScheduledReporter;
+import com.codahale.metrics.Snapshot;
+import com.codahale.metrics.Timer;
+
+/**
+ * Modified from https://github.com/joshelser/dropwizard-hadoop-metrics2,
Copyright by Josh Elser
+ *
+ * A {@link com.codahale.metrics.Reporter} which also acts as a Hadoop Metrics2
+ * {@link MetricsSource}. Configure it like other Reporters.
+ *
+ *
+ * final HadoopMetrics2Reporter metrics2Reporter =
HadoopMetrics2Reporter.forRegistry(metrics)
+ * .build(DefaultMetricsSystem.initialize("Phoenix"), // The
application-level name
+ *"QueryServer", // Component name
+ *"Phoenix Query Server", // Component description
+ *"General"); // Name for each metric record
+ *
+ * metrics2Reporter.start(30, TimeUnit.SECONDS);
+ *
+ */
+public class HadoopMetrics2Reporter extends ScheduledReporter implements
MetricsSource {
+private static final Logger LOG =
LoggerFactory.getLogger(HadoopMetrics2Reporter.class);
+private static final String EMPTY_STRING = "";
+@SuppressWarnings("rawtypes")
+
[GitHub] codecov-io edited a comment on issue #276: KYLIN-3605 upgrade hadoop-common and zookeeper version to fix securit…
codecov-io edited a comment on issue #276: KYLIN-3605 upgrade hadoop-common and zookeeper version to fix securit… URL: https://github.com/apache/kylin/pull/276#issuecomment-425686030 # [Codecov](https://codecov.io/gh/apache/kylin/pull/276?src=pr=h1) Report > Merging [#276](https://codecov.io/gh/apache/kylin/pull/276?src=pr=desc) into [master](https://codecov.io/gh/apache/kylin/commit/8db033097833be5118655a6d88a5b96e862055ae?src=pr=desc) will **increase** coverage by `0.13%`. > The diff coverage is `80.98%`. [](https://codecov.io/gh/apache/kylin/pull/276?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #276 +/- ## + Coverage 21.14% 21.28% +0.13% - Complexity 4405 4435 +30 Files 1086 1087 +1 Lines 6980169976 +175 Branches 1011010126 +16 + Hits 1476014891 +131 - Misses5364153681 +40 - Partials 1400 1404 +4 ``` | [Impacted Files](https://codecov.io/gh/apache/kylin/pull/276?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...ylin/common/metrics/metrics2/Metrics2Reporter.java](https://codecov.io/gh/apache/kylin/pull/276/diff?src=pr=tree#diff-Y29yZS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL2NvbW1vbi9tZXRyaWNzL21ldHJpY3MyL01ldHJpY3MyUmVwb3J0ZXIuamF2YQ==) | `0% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...ommon/metrics/metrics2/HadoopMetrics2Reporter.java](https://codecov.io/gh/apache/kylin/pull/276/diff?src=pr=tree#diff-Y29yZS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL2NvbW1vbi9tZXRyaWNzL21ldHJpY3MyL0hhZG9vcE1ldHJpY3MyUmVwb3J0ZXIuamF2YQ==) | `80.98% <80.98%> (ø)` | `30 <30> (?)` | | | [...lin/dict/lookup/cache/RocksDBLookupTableCache.java](https://codecov.io/gh/apache/kylin/pull/276/diff?src=pr=tree#diff-Y29yZS1kaWN0aW9uYXJ5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9kaWN0L2xvb2t1cC9jYWNoZS9Sb2Nrc0RCTG9va3VwVGFibGVDYWNoZS5qYXZh) | `76.16% <0%> (-0.52%)` | `27% <0%> (ø)` | | | [...ache/kylin/cube/inmemcubing/DoggedCubeBuilder.java](https://codecov.io/gh/apache/kylin/pull/276/diff?src=pr=tree#diff-Y29yZS1jdWJlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9jdWJlL2lubWVtY3ViaW5nL0RvZ2dlZEN1YmVCdWlsZGVyLmphdmE=) | `0% <0%> (ø)` | `0% <0%> (ø)` | :arrow_down: | | [.../routing/rules/RemoveBlackoutRealizationsRule.java](https://codecov.io/gh/apache/kylin/pull/276/diff?src=pr=tree#diff-cXVlcnkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL3F1ZXJ5L3JvdXRpbmcvcnVsZXMvUmVtb3ZlQmxhY2tvdXRSZWFsaXphdGlvbnNSdWxlLmphdmE=) | `0% <0%> (ø)` | `0% <0%> (ø)` | :arrow_down: | | [...pache/kylin/cube/inmemcubing/InMemCubeBuilder.java](https://codecov.io/gh/apache/kylin/pull/276/diff?src=pr=tree#diff-Y29yZS1jdWJlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9jdWJlL2lubWVtY3ViaW5nL0luTWVtQ3ViZUJ1aWxkZXIuamF2YQ==) | `0% <0%> (ø)` | `0% <0%> (ø)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/kylin/pull/276?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/kylin/pull/276?src=pr=footer). Last update [8db0330...8bf7b3d](https://codecov.io/gh/apache/kylin/pull/276?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (KYLIN-3606) Unstable KylinUserManagerTest
Shaofeng SHI created KYLIN-3606:
---
Summary: Unstable KylinUserManagerTest
Key: KYLIN-3606
URL: https://issues.apache.org/jira/browse/KYLIN-3606
Project: Kylin
Issue Type: Bug
Affects Versions: v2.6.0
Reporter: Shaofeng SHI
{code:java}
[[1;34mINFO[m] [1;32mTests run: [0;1;32m3[m, Failures: 0, Errors: 0,
Skipped: 0, Time elapsed: 4.187 s - in
org.apache.kylin.rest.security.[1mQueryWithTableACLTest[m
[[1;34mINFO[m] Running
org.apache.kylin.rest.security.[1mKylinUserManagerTest[m
2018-09-30 04:01:45,114 INFO [main] common.KylinConfig:113 : Destroy
KylinConfig
2018-09-30 04:01:45,160 INFO [main] common.KylinConfig:332 : Loading
kylin-defaults.properties from
/home/travis/build/apache/kylin/core-common/target/classes/kylin-defaults.properties
2018-09-30 04:01:45,169 INFO [main] common.KylinConfig:287 : Use
KYLIN_CONF=../examples/test_metadata/
2018-09-30 04:01:45,170 INFO [main] common.KylinConfig:99 : Initialized a new
KylinConfig from getInstanceFromEnv : 1932094362
2018-09-30 04:01:45,170 INFO [main] common.KylinConfigBase:198 : Kylin Config
was updated with kylin.metadata.url : ../examples/test_metadata/
2018-09-30 04:01:45,170 INFO [main] common.KylinConfigBase:198 : Kylin Config
was updated with kylin.env.hdfs-working-dir : file:///tmp/kylin
2018-09-30 04:01:45,193 INFO [main] util.MultiNodeManagerTestBase:50 : Chosen
port for CacheServiceTest is 45238
2018-09-30 04:01:45,194 INFO [main] common.KylinConfigBase:198 : Kylin Config
was updated with kylin.server.cluster-servers : localhost:45238
2018-09-30 04:01:45,194 INFO [main] common.KylinConfigBase:198 : Kylin Config
was updated with kylin.server.cluster-servers : localhost:45238
2018-09-30 04:01:45,194 INFO [main] common.KylinConfigBase:198 : Kylin Config
was updated with kylin.metadata.url : ../examples/test_metadata
2018-09-30 04:01:45,273 INFO [Thread-2-SendThread(localhost:2181)]
zookeeper.ClientCnxn:1029 : Opening socket connection to server
localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown
error)
2018-09-30 04:01:45,274 INFO [Thread-2-SendThread(localhost:2181)]
zookeeper.ClientCnxn:1166 : Socket error occurred: localhost/127.0.0.1:2181:
Connection refused
2018-09-30 04:01:45,275 INFO [main] util.log:192 : Logging initialized @92503ms
2018-09-30 04:01:46,021 INFO [main] server.Server:346 : jetty-9.3.22.v20171030
2018-09-30 04:01:46,225 INFO [main] handler.ContextHandler:781 : Started
o.e.j.s.ServletContextHandler@545311c{/,null,AVAILABLE}
2018-09-30 04:01:46,254 INFO [main] server.AbstractConnector:278 : Started
ServerConnector@4bdfeead{HTTP/1.1,[http/1.1]}{0.0.0.0:45238}
2018-09-30 04:01:46,254 INFO [main] server.Server:414 : Started @93482ms
2018-09-30 04:01:46,255 INFO [main] security.KylinUserManager:61 :
Initializing KylinUserManager with config ../examples/test_metadata/
2018-09-30 04:01:46,256 INFO [main] persistence.ResourceStore:88 : Using
metadata url ../examples/test_metadata/ for resource store
2018-09-30 04:01:46,256 DEBUG [main] cachesync.CachedCrudAssist:118 : Reloading
ManagedUser from
/home/travis/build/apache/kylin/server/../examples/test_metadata/user
2018-09-30 04:01:46,257 DEBUG [main] cachesync.CachedCrudAssist:127 : Loaded 0
ManagedUser(s) out of 0 resource
2018-09-30 04:01:46,257 INFO [main] common.KylinConfig:432 : Creating new
manager instance of class org.apache.kylin.metadata.cachesync.Broadcaster
2018-09-30 04:01:46,258 DEBUG [main] cachesync.Broadcaster:101 : 1 nodes in the
cluster: [localhost:45238]
2018-09-30 04:01:46,259 INFO [main] security.KylinUserManager:61 :
Initializing KylinUserManager with config ../examples/test_metadata
2018-09-30 04:01:46,259 INFO [main] persistence.ResourceStore:88 : Using
metadata url ../examples/test_metadata for resource store
2018-09-30 04:01:46,259 DEBUG [main] cachesync.CachedCrudAssist:118 : Reloading
ManagedUser from
/home/travis/build/apache/kylin/server/../examples/test_metadata/user
2018-09-30 04:01:46,260 DEBUG [main] cachesync.CachedCrudAssist:127 : Loaded 0
ManagedUser(s) out of 0 resource
2018-09-30 04:01:46,260 INFO [main] common.KylinConfig:432 : Creating new
manager instance of class org.apache.kylin.metadata.cachesync.Broadcaster
2018-09-30 04:01:46,260 DEBUG [main] cachesync.Broadcaster:101 : 1 nodes in the
cluster: [localhost:45238]
2018-09-30 04:01:46,261 DEBUG [main] cachesync.CachedCrudAssist:190 : Saving
ManagedUser at /user/U1
2018-09-30 04:01:46,263 DEBUG [pool-173-thread-1] cachesync.Broadcaster:113 :
Servers in the cluster: [localhost:45238]
2018-09-30 04:01:46,375 INFO [Thread-2-SendThread(localhost:2181)]
zookeeper.ClientCnxn:1029 : Opening socket connection to server
localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown
error)
2018-09-30 04:01:46,376 INFO [Thread-2-SendThread(localhost:2181)]
[GitHub] shaofengshi closed pull request #248: KYLIN-3588, fix potentially duplicate putting in RemoveBlackoutRealiz…
shaofengshi closed pull request #248: KYLIN-3588, fix potentially duplicate
putting in RemoveBlackoutRealiz…
URL: https://github.com/apache/kylin/pull/248
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/query/src/main/java/org/apache/kylin/query/routing/rules/RemoveBlackoutRealizationsRule.java
b/query/src/main/java/org/apache/kylin/query/routing/rules/RemoveBlackoutRealizationsRule.java
index c64f6dae73..278bb46452 100644
---
a/query/src/main/java/org/apache/kylin/query/routing/rules/RemoveBlackoutRealizationsRule.java
+++
b/query/src/main/java/org/apache/kylin/query/routing/rules/RemoveBlackoutRealizationsRule.java
@@ -59,13 +59,16 @@ public static boolean accept(IRealization real) {
private static IRealizationFilter getFilterImpl(KylinConfig conf) {
IRealizationFilter filter = filters.get(conf);
if (filter == null) {
-try {
-Class clz =
ClassUtil.forName(conf.getQueryRealizationFilter(), IRealizationFilter.class);
-filter =
clz.getConstructor(KylinConfig.class).newInstance(conf);
-} catch (Exception e) {
-throw new RuntimeException(e);
+synchronized (RemoveBlackoutRealizationsRule.class) {
+try {
+Class clz =
ClassUtil.forName(conf.getQueryRealizationFilter(),
+IRealizationFilter.class);
+filter =
clz.getConstructor(KylinConfig.class).newInstance(conf);
+} catch (Exception e) {
+throw new RuntimeException(e);
+}
+filters.put(conf, filter);
}
-filters.put(conf, filter);
}
return filter;
}
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] shaofengshi closed pull request #249: KYLIN-3581 Obey the contract of 'x.equals(y) should be consistent with x.compareTo(y)'
shaofengshi closed pull request #249: KYLIN-3581 Obey the contract of
'x.equals(y) should be consistent with x.compareTo(y)'
URL: https://github.com/apache/kylin/pull/249
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/core-cube/src/main/java/org/apache/kylin/cube/inmemcubing/DoggedCubeBuilder.java
b/core-cube/src/main/java/org/apache/kylin/cube/inmemcubing/DoggedCubeBuilder.java
index d761505b2c..06e4a5d582 100644
---
a/core-cube/src/main/java/org/apache/kylin/cube/inmemcubing/DoggedCubeBuilder.java
+++
b/core-cube/src/main/java/org/apache/kylin/cube/inmemcubing/DoggedCubeBuilder.java
@@ -347,6 +347,19 @@ public boolean fetchNext() throws IOException {
}
}
+@Override
+public boolean equals(Object o) {
+if (this == o) return true;
+if (o == null || getClass() != o.getClass()) return false;
+MergeSlot mergeSlot = (MergeSlot) o;
+return compareTo(mergeSlot) == 0;
+}
+
+@Override
+public int hashCode() {
+return Long.hashCode(currentCuboidId);
+}
+
@Override
public int compareTo(MergeSlot o) {
long cuboidComp = this.currentCuboidId - o.currentCuboidId;
diff --git
a/core-cube/src/main/java/org/apache/kylin/cube/inmemcubing/InMemCubeBuilder.java
b/core-cube/src/main/java/org/apache/kylin/cube/inmemcubing/InMemCubeBuilder.java
index db3eb5dd80..ef61ce9a5a 100644
---
a/core-cube/src/main/java/org/apache/kylin/cube/inmemcubing/InMemCubeBuilder.java
+++
b/core-cube/src/main/java/org/apache/kylin/cube/inmemcubing/InMemCubeBuilder.java
@@ -521,5 +521,18 @@ public int compareTo(CuboidTask o) {
long comp = this.childCuboidId - o.childCuboidId;
return comp < 0 ? -1 : (comp > 0 ? 1 : 0);
}
+
+@Override
+public boolean equals(Object o) {
+if (this == o) return true;
+if (o == null || getClass() != o.getClass()) return false;
+CuboidTask that = (CuboidTask) o;
+return compareTo(that) == 0;
+}
+
+@Override
+public int hashCode() {
+return Long.hashCode(childCuboidId);
+}
}
}
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] shaofengshi closed pull request #251: KYLIN-3583, KYLIN-3579
shaofengshi closed pull request #251: KYLIN-3583, KYLIN-3579
URL: https://github.com/apache/kylin/pull/251
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/core-cube/src/main/java/org/apache/kylin/cube/cuboid/algorithm/BPUSCalculator.java
b/core-cube/src/main/java/org/apache/kylin/cube/cuboid/algorithm/BPUSCalculator.java
index ea91c6cb81..6316858d58 100755
---
a/core-cube/src/main/java/org/apache/kylin/cube/cuboid/algorithm/BPUSCalculator.java
+++
b/core-cube/src/main/java/org/apache/kylin/cube/cuboid/algorithm/BPUSCalculator.java
@@ -60,15 +60,15 @@ protected BPUSCalculator(CuboidStats cuboidStats,
ImmutableMap initC
cuboidAggCostMap.put(cuboid, getCuboidCost(cuboid));
}
}
-Set mandatoryCuboidSetWithStats = cuboidAggCostMap.keySet();
+
//Initialize stats for selection cuboids
long baseCuboidCost = getCuboidCost(cuboidStats.getBaseCuboid());
for (Long cuboid : cuboidStats.getAllCuboidsForSelection()) {
long leastCost = baseCuboidCost;
-for (Long cuboidTarget : mandatoryCuboidSetWithStats) {
-if ((cuboid | cuboidTarget) == cuboidTarget) {
-if (leastCost > cuboidAggCostMap.get(cuboidTarget)) {
-leastCost = cuboidAggCostMap.get(cuboidTarget);
+for (Map.Entry cuboidTargetEntry :
cuboidAggCostMap.entrySet()) {
+if ((cuboid | cuboidTargetEntry.getKey()) ==
cuboidTargetEntry.getKey()) {
+if (leastCost > cuboidTargetEntry.getValue()) {
+leastCost = cuboidTargetEntry.getValue();
}
}
}
@@ -106,9 +106,9 @@ protected BPUSCalculator(CuboidStats cuboidStats,
ImmutableMap initC
}
double totalCostSaving = 0;
int benefitCount = 0;
-for (Long cuboid : cuboidAggCostMapCopy.keySet()) {
-if (cuboidAggCostMapCopy.get(cuboid) <
processCuboidAggCostMap.get(cuboid)) {
-totalCostSaving += processCuboidAggCostMap.get(cuboid) -
cuboidAggCostMapCopy.get(cuboid);
+for (Map.Entry entry : cuboidAggCostMapCopy.entrySet()) {
+if (entry.getValue() <
processCuboidAggCostMap.get(entry.getKey())) {
+totalCostSaving += processCuboidAggCostMap.get(entry.getKey())
- entry.getValue();
benefitCount++;
}
}
diff --git
a/core-cube/src/main/java/org/apache/kylin/cube/gridtable/CubeCodeSystem.java
b/core-cube/src/main/java/org/apache/kylin/cube/gridtable/CubeCodeSystem.java
index 357747663d..4c71fea011 100644
---
a/core-cube/src/main/java/org/apache/kylin/cube/gridtable/CubeCodeSystem.java
+++
b/core-cube/src/main/java/org/apache/kylin/cube/gridtable/CubeCodeSystem.java
@@ -168,13 +168,13 @@ public Object decodeColumnValue(int col, ByteBuffer buf) {
// deal with holistic distinct count
if (dependentMetricsMap != null) {
-for (Integer child : dependentMetricsMap.keySet()) {
-if (columns.get(child)) {
-Integer parent = dependentMetricsMap.get(child);
+for (Map.Entry childEntry :
dependentMetricsMap.entrySet()) {
+if (columns.get(childEntry.getKey())) {
+Integer parent = childEntry.getValue();
if (columns.get(parent) == false)
throw new IllegalStateException();
-int childIdx = columns.trueBitIndexOf(child);
+int childIdx = columns.trueBitIndexOf(childEntry.getKey());
int parentIdx = columns.trueBitIndexOf(parent);
result[childIdx].setDependentAggregator(result[parentIdx]);
}
diff --git
a/core-metrics/src/main/java/org/apache/kylin/metrics/lib/impl/RecordEventTimeDetail.java
b/core-metrics/src/main/java/org/apache/kylin/metrics/lib/impl/RecordEventTimeDetail.java
index c611d0f943..827b5a10e1 100644
---
a/core-metrics/src/main/java/org/apache/kylin/metrics/lib/impl/RecordEventTimeDetail.java
+++
b/core-metrics/src/main/java/org/apache/kylin/metrics/lib/impl/RecordEventTimeDetail.java
@@ -71,7 +71,7 @@ public RecordEventTimeDetail(long timeStamp) {
this.second = calendar.get(Calendar.SECOND);
long timeStampForWeekBegin = timeStamp;
-timeStampForWeekBegin -= 360 * 24 *
(calendar.get(Calendar.DAY_OF_WEEK) - 1);
+timeStampForWeekBegin -= 360L * 24 *
(calendar.get(Calendar.DAY_OF_WEEK) - 1);
calendar.setTimeInMillis(timeStampForWeekBegin);
this.week_begin_date = dateFormat.format(calendar.getTime());
}
[GitHub] codecov-io commented on issue #251: KYLIN-3583, KYLIN-3579
codecov-io commented on issue #251: KYLIN-3583, KYLIN-3579 URL: https://github.com/apache/kylin/pull/251#issuecomment-425690030 # [Codecov](https://codecov.io/gh/apache/kylin/pull/251?src=pr=h1) Report > :exclamation: No coverage uploaded for pull request base (`master@f42e937`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit). > The diff coverage is `0%`. [](https://codecov.io/gh/apache/kylin/pull/251?src=pr=tree) ```diff @@Coverage Diff@@ ## master #251 +/- ## = Coverage ? 21.15% Complexity? 4404 = Files ? 1086 Lines ?69801 Branches ?10110 = Hits ?14763 Misses?53638 Partials ? 1400 ``` | [Impacted Files](https://codecov.io/gh/apache/kylin/pull/251?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [.../kylin/metrics/lib/impl/RecordEventTimeDetail.java](https://codecov.io/gh/apache/kylin/pull/251/diff?src=pr=tree#diff-Y29yZS1tZXRyaWNzL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9tZXRyaWNzL2xpYi9pbXBsL1JlY29yZEV2ZW50VGltZURldGFpbC5qYXZh) | `0% <0%> (ø)` | `0 <0> (?)` | | | [...rg/apache/kylin/cube/gridtable/CubeCodeSystem.java](https://codecov.io/gh/apache/kylin/pull/251/diff?src=pr=tree#diff-Y29yZS1jdWJlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9jdWJlL2dyaWR0YWJsZS9DdWJlQ29kZVN5c3RlbS5qYXZh) | `25.37% <0%> (ø)` | `8 <0> (?)` | | | [...he/kylin/cube/cuboid/algorithm/BPUSCalculator.java](https://codecov.io/gh/apache/kylin/pull/251/diff?src=pr=tree#diff-Y29yZS1jdWJlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9jdWJlL2N1Ym9pZC9hbGdvcml0aG0vQlBVU0NhbGN1bGF0b3IuamF2YQ==) | `0% <0%> (ø)` | `0 <0> (?)` | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/kylin/pull/251?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/kylin/pull/251?src=pr=footer). Last update [f42e937...6bdbe40](https://codecov.io/gh/apache/kylin/pull/251?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io commented on issue #276: KYLIN-3605 upgrade hadoop-common and zookeeper version to fix securit…
codecov-io commented on issue #276: KYLIN-3605 upgrade hadoop-common and zookeeper version to fix securit… URL: https://github.com/apache/kylin/pull/276#issuecomment-425686030 # [Codecov](https://codecov.io/gh/apache/kylin/pull/276?src=pr=h1) Report > :exclamation: No coverage uploaded for pull request base (`master@f42e937`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit). > The diff coverage is `0%`. [](https://codecov.io/gh/apache/kylin/pull/276?src=pr=tree) ```diff @@Coverage Diff@@ ## master #276 +/- ## = Coverage ? 21.09% Complexity? 4405 = Files ? 1087 Lines ?69965 Branches ?10120 = Hits ?14761 Misses?53804 Partials ? 1400 ``` | [Impacted Files](https://codecov.io/gh/apache/kylin/pull/276?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...ylin/common/metrics/metrics2/Metrics2Reporter.java](https://codecov.io/gh/apache/kylin/pull/276/diff?src=pr=tree#diff-Y29yZS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL2NvbW1vbi9tZXRyaWNzL21ldHJpY3MyL01ldHJpY3MyUmVwb3J0ZXIuamF2YQ==) | `0% <ø> (ø)` | `0 <0> (?)` | | | [...ommon/metrics/metrics2/HadoopMetrics2Reporter.java](https://codecov.io/gh/apache/kylin/pull/276/diff?src=pr=tree#diff-Y29yZS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL2NvbW1vbi9tZXRyaWNzL21ldHJpY3MyL0hhZG9vcE1ldHJpY3MyUmVwb3J0ZXIuamF2YQ==) | `0% <0%> (ø)` | `0 <0> (?)` | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/kylin/pull/276?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/kylin/pull/276?src=pr=footer). Last update [f42e937...0ae4686](https://codecov.io/gh/apache/kylin/pull/276?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] shaofengshi closed pull request #277: KYLIN-3602 Enable more checkstyle rules
shaofengshi closed pull request #277: KYLIN-3602 Enable more checkstyle rules URL: https://github.com/apache/kylin/pull/277 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/dev-support/checkstyle-suppressions.xml b/dev-support/checkstyle-suppressions.xml index 06de62554f..f110b7c3ef 100644 --- a/dev-support/checkstyle-suppressions.xml +++ b/dev-support/checkstyle-suppressions.xml @@ -25,4 +25,40 @@ + + + + + + + + + + + + + + + + + + + + + + + diff --git a/dev-support/checkstyle.xml b/dev-support/checkstyle.xml index 45325bf950..d8eb73fc89 100644 --- a/dev-support/checkstyle.xml +++ b/dev-support/checkstyle.xml @@ -28,11 +28,31 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io commented on issue #277: KYLIN-3602 Enable more checkstyle rules
codecov-io commented on issue #277: KYLIN-3602 Enable more checkstyle rules URL: https://github.com/apache/kylin/pull/277#issuecomment-425644108 # [Codecov](https://codecov.io/gh/apache/kylin/pull/277?src=pr=h1) Report > Merging [#277](https://codecov.io/gh/apache/kylin/pull/277?src=pr=desc) into [master](https://codecov.io/gh/apache/kylin/commit/bd8961c884cdc8ec5a9903a1d5a317bb626a2dff?src=pr=desc) will **increase** coverage by `0.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/kylin/pull/277?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #277 +/- ## + Coverage 21.14% 21.15% +0.01% - Complexity 4405 4406 +1 Files 1086 1086 Lines 6980269802 Branches 1011010110 + Hits 1475914768 +9 + Misses5364253637 -5 + Partials 1401 1397 -4 ``` | [Impacted Files](https://codecov.io/gh/apache/kylin/pull/277?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...rg/apache/kylin/cube/inmemcubing/MemDiskStore.java](https://codecov.io/gh/apache/kylin/pull/277/diff?src=pr=tree#diff-Y29yZS1jdWJlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9jdWJlL2lubWVtY3ViaW5nL01lbURpc2tTdG9yZS5qYXZh) | `70.21% <0%> (+0.6%)` | `7% <0%> (ø)` | :arrow_down: | | [...g/apache/kylin/source/datagen/ColumnGenerator.java](https://codecov.io/gh/apache/kylin/pull/277/diff?src=pr=tree#diff-Y29yZS1tZXRhZGF0YS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUva3lsaW4vc291cmNlL2RhdGFnZW4vQ29sdW1uR2VuZXJhdG9yLmphdmE=) | `72.29% <0%> (+1.35%)` | `8% <0%> (ø)` | :arrow_down: | | [.../apache/kylin/cube/cuboid/TreeCuboidScheduler.java](https://codecov.io/gh/apache/kylin/pull/277/diff?src=pr=tree#diff-Y29yZS1jdWJlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9jdWJlL2N1Ym9pZC9UcmVlQ3Vib2lkU2NoZWR1bGVyLmphdmE=) | `66.15% <0%> (+2.3%)` | `0% <0%> (ø)` | :arrow_down: | | [...he/kylin/dict/lookup/cache/RocksDBLookupTable.java](https://codecov.io/gh/apache/kylin/pull/277/diff?src=pr=tree#diff-Y29yZS1kaWN0aW9uYXJ5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9kaWN0L2xvb2t1cC9jYWNoZS9Sb2Nrc0RCTG9va3VwVGFibGUuamF2YQ==) | `78.37% <0%> (+5.4%)` | `6% <0%> (+1%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/kylin/pull/277?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/kylin/pull/277?src=pr=footer). Last update [bd8961c...fefc0ec](https://codecov.io/gh/apache/kylin/pull/277?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] coveralls commented on issue #277: KYLIN-3602 Enable more checkstyle rules
coveralls commented on issue #277: KYLIN-3602 Enable more checkstyle rules URL: https://github.com/apache/kylin/pull/277#issuecomment-425644083 ## Pull Request Test Coverage Report for [Build 3719](https://coveralls.io/builds/19261940) * **0** of **0** changed or added relevant lines in **0** files are covered. * No unchanged relevant lines lost coverage. * Overall coverage increased (+**0.007%**) to **23.161%** --- | Totals | [](https://coveralls.io/builds/19261940) | | :-- | --: | | Change from base [Build 3717](https://coveralls.io/builds/19261333): | 0.007% | | Covered Lines: | 16167 | | Relevant Lines: | 69802 | --- # - [Coveralls](https://coveralls.io) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] asfgit commented on issue #277: KYLIN-3602 Enable more checkstyle rules
asfgit commented on issue #277: KYLIN-3602 Enable more checkstyle rules URL: https://github.com/apache/kylin/pull/277#issuecomment-425642269 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ZhouYC627 opened a new pull request #277: KYLIN-3602 Enable more checkstyle rules
ZhouYC627 opened a new pull request #277: KYLIN-3602 Enable more checkstyle rules URL: https://github.com/apache/kylin/pull/277 Enable more checkstyle rules and suppress the checkstyle errors of old files in _dev-support/checkstyle-suppressions.xml_ https://issues.apache.org/jira/browse/KYLIN-3602 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] asfgit commented on issue #277: KYLIN-3602 Enable more checkstyle rules
asfgit commented on issue #277: KYLIN-3602 Enable more checkstyle rules URL: https://github.com/apache/kylin/pull/277#issuecomment-425642270 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] shaofengshi closed pull request #270: KYLIN-2924 enable google error-prone in compile phase
shaofengshi closed pull request #270: KYLIN-2924 enable google error-prone in compile phase URL: https://github.com/apache/kylin/pull/270 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] asfgit commented on issue #276: KYLIN-3605 upgrade hadoop-common and zookeeper version to fix securit…
asfgit commented on issue #276: KYLIN-3605 upgrade hadoop-common and zookeeper version to fix securit… URL: https://github.com/apache/kylin/pull/276#issuecomment-425635060 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (KYLIN-3605) Fix security issues reported by snyk.io
Shaofeng SHI created KYLIN-3605: --- Summary: Fix security issues reported by snyk.io Key: KYLIN-3605 URL: https://issues.apache.org/jira/browse/KYLIN-3605 Project: Kylin Issue Type: Improvement Reporter: Shaofeng SHI [org.apache.commons:commons-compress|https://github.com/apache/commons-compress] defines an API for working with compression and archive formats. Affected versions of this package are vulnerable to Directory Traversal. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] shaofengshi closed pull request #272: KYLIN-3232 Add document for ops tools
shaofengshi closed pull request #272: KYLIN-3232 Add document for ops tools
URL: https://github.com/apache/kylin/pull/272
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/website/_data/docs-cn.yml b/website/_data/docs-cn.yml
index f2b1bb35a6..93f29ecd76 100644
--- a/website/_data/docs-cn.yml
+++ b/website/_data/docs-cn.yml
@@ -53,6 +53,7 @@
- tutorial/squirrel
- tutorial/Qlik
- tutorial/superset
+ - tutorial/tools
- title: 帮助
diff --git a/website/_data/docs.yml b/website/_data/docs.yml
index 9146cfbbf8..ead957ec9e 100644
--- a/website/_data/docs.yml
+++ b/website/_data/docs.yml
@@ -65,6 +65,7 @@
- tutorial/hue
- tutorial/Qlik
- tutorial/superset
+ - tutorial/tools
- title: How To
docs:
diff --git a/website/_docs/tutorial/tools.cn.md
b/website/_docs/tutorial/tools.cn.md
new file mode 100644
index 00..20cb65bfca
--- /dev/null
+++ b/website/_docs/tutorial/tools.cn.md
@@ -0,0 +1,156 @@
+---
+layout: docs-cn
+title: "Kylin 中的工具类"
+categories: tutorial
+permalink: /cn/docs/tutorial/tools.html
+---
+Kylin
有很多好的工具类。这篇文档会介绍以下几个工具类：KylinConfigCLI.java，CubeMetaExtractor.java，CubeMetaIngester.java，CubeMigrationCLI.java
和 CubeMigrationCheckCLI.java。在使用这些工具类前，首先要切换到 KYLIN_HOME 目录下。
+
+## KylinConfigCLI.java
+
+### 作用
+KylinConfigCLI 工具类会将您输入的 Kylin 参数的值输出。
+
+### 如何使用
+类名后只能写一个参数，conf_name 即您想要知道其值的参数名称。
+{% highlight Groff markup %}
+./bin/kylin.sh org.apache.kylin.tool.KylinConfigCLI
+{% endhighlight %}
+例如：
+{% highlight Groff markup %}
+./bin/kylin.sh org.apache.kylin.tool.KylinConfigCLI kylin.server.mode
+{% endhighlight %}
+结果：
+{% highlight Groff markup %}
+all
+{% endhighlight %}
+如果您不知道参数的准确名称，您可以使用以下命令，然后所有以该前缀为前缀的参数的值都会被列出。
+{% highlight Groff markup %}
+./bin/kylin.sh org.apache.kylin.tool.KylinConfigCLI .
+{% endhighlight %}
+例如：
+{% highlight Groff markup %}
+./bin/kylin.sh org.apache.kylin.tool.KylinConfigCLI kylin.job.
+{% endhighlight %}
+结果：
+{% highlight Groff markup %}
+max-concurrent-jobs=10
+retry=3
+sampling-percentage=100
+{% endhighlight %}
+
+## CubeMetaExtractor.java
+
+### 作用
+CubeMetaExtractor.java 用于提取与 Cube 相关的信息以达到调试/分发的目的。
+
+### 如何使用
+类名后至少写两个参数。
+{% highlight Groff markup %}
+./bin/kylin.sh org.apache.kylin.tool.CubeMetaExtractor -
-destDir
+{% endhighlight %}
+例如：
+{% highlight Groff markup %}
+./bin/kylin.sh org.apache.kylin.tool.CubeMetaExtractor -cube querycube
-destDir /root/newconfigdir1
+{% endhighlight %}
+结果：
+命令执行成功后，您想要抽取的 Cube / project / hybrid 将会存在于您指定的 destDir 目录中。
+
+下面会列出所有支持的参数：
+
+| Parameter | Description
|
+| - |
:--
|
+| allProjects | Specify realizations
in all projects to extract |
+| compress| Specify whether to
compress the output with zip. Default true.
|
+| cube| Specify which Cube
to extract
|
+| destDir | (Required) Specify
the dest dir to save the related information
|
+| engineType| Specify the engine
type to overwrite. Default is empty, keep origin.
|
+| hybrid| Specify which hybrid
to extract |
+| includeJobs | Set this to true if
want to extract job info/outputs too. Default false
|
+| includeSegmentDetails | Set this to true if
want to extract segment details too, such as dict, tablesnapshot. Default false
|
+| includeSegments | Set this to true if
want extract the segments info. Default true
|
+| onlyOutput| When include jobs,
only extract output of job. Default true
|
+| packagetype | Specify the package
type
|
+| project | Specify realizations
in which project to extract|
+| storageType | Specify the storage
type to overwrite. Default
[GitHub] asfgit commented on issue #275: Fix pictures' URL invalid
asfgit commented on issue #275: Fix pictures' URL invalid URL: https://github.com/apache/kylin/pull/275#issuecomment-425629403 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] asfgit commented on issue #275: Fix pictures' URL invalid
asfgit commented on issue #275: Fix pictures' URL invalid URL: https://github.com/apache/kylin/pull/275#issuecomment-425629402 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] coveralls commented on issue #268: KYLIN-3594 Obey JDBC standard
coveralls commented on issue #268: KYLIN-3594 Obey JDBC standard URL: https://github.com/apache/kylin/pull/268#issuecomment-425625274 ## Pull Request Test Coverage Report for [Build 3714](https://coveralls.io/builds/19260478) * **4** of **7** **(57.14%)** changed or added relevant lines in **2** files are covered. * **3** unchanged lines in **2** files lost coverage. * Overall coverage decreased (**-0.002%**) to **23.17%** --- | Changes Missing Coverage | Covered Lines | Changed/Added Lines | % | | :-|--||---: | | [query/src/main/java/org/apache/kylin/query/util/QueryUtil.java](https://coveralls.io/builds/19260478/source?filename=query%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fquery%2Futil%2FQueryUtil.java#L103) | 4 | 5 | 80.0% | [server-base/src/main/java/org/apache/kylin/rest/service/QueryService.java](https://coveralls.io/builds/19260478/source?filename=server-base%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Frest%2Fservice%2FQueryService.java#L561) | 0 | 2 | 0.0% | Files with Coverage Reduction | New Missed Lines | % | | :-|--|--: | | [core-cube/src/main/java/org/apache/kylin/cube/inmemcubing/MemDiskStore.java](https://coveralls.io/builds/19260478/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2Finmemcubing%2FMemDiskStore.java#L553) | 1 | 78.12% | | [core-cube/src/main/java/org/apache/kylin/cube/cuboid/TreeCuboidScheduler.java](https://coveralls.io/builds/19260478/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2Fcuboid%2FTreeCuboidScheduler.java#L124) | 2 | 68.46% | | Totals | [](https://coveralls.io/builds/19260478) | | :-- | --: | | Change from base [Build 3711](https://coveralls.io/builds/19259592): | -0.002% | | Covered Lines: | 16161 | | Relevant Lines: | 69750 | --- # - [Coveralls](https://coveralls.io) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io commented on issue #268: KYLIN-3594 Obey JDBC standard
codecov-io commented on issue #268: KYLIN-3594 Obey JDBC standard URL: https://github.com/apache/kylin/pull/268#issuecomment-425625134 # [Codecov](https://codecov.io/gh/apache/kylin/pull/268?src=pr=h1) Report > Merging [#268](https://codecov.io/gh/apache/kylin/pull/268?src=pr=desc) into [master](https://codecov.io/gh/apache/kylin/commit/10587a65fe0552179a5c8a6e1151686ce1c8a135?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `42.85%`. [](https://codecov.io/gh/apache/kylin/pull/268?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #268 +/- ## - Coverage 21.16% 21.15% -0.01% - Complexity 4405 4406 +1 Files 1086 1086 Lines 6974569750 +5 Branches 1008810088 - Hits 1476114758 -3 - Misses5358653591 +5 - Partials 1398 1401 +3 ``` | [Impacted Files](https://codecov.io/gh/apache/kylin/pull/268?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...va/org/apache/kylin/rest/service/QueryService.java](https://codecov.io/gh/apache/kylin/pull/268/diff?src=pr=tree#diff-c2VydmVyLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL3Jlc3Qvc2VydmljZS9RdWVyeVNlcnZpY2UuamF2YQ==) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...in/java/org/apache/kylin/query/util/QueryUtil.java](https://codecov.io/gh/apache/kylin/pull/268/diff?src=pr=tree#diff-cXVlcnkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL3F1ZXJ5L3V0aWwvUXVlcnlVdGlsLmphdmE=) | `61.33% <60%> (-0.64%)` | `19 <1> (+1)` | | | [.../apache/kylin/cube/cuboid/TreeCuboidScheduler.java](https://codecov.io/gh/apache/kylin/pull/268/diff?src=pr=tree#diff-Y29yZS1jdWJlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9jdWJlL2N1Ym9pZC9UcmVlQ3Vib2lkU2NoZWR1bGVyLmphdmE=) | `63.84% <0%> (-2.31%)` | `0% <0%> (ø)` | | | [...rg/apache/kylin/cube/inmemcubing/MemDiskStore.java](https://codecov.io/gh/apache/kylin/pull/268/diff?src=pr=tree#diff-Y29yZS1jdWJlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9jdWJlL2lubWVtY3ViaW5nL01lbURpc2tTdG9yZS5qYXZh) | `69.6% <0%> (-0.61%)` | `7% <0%> (ø)` | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/kylin/pull/268?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/kylin/pull/268?src=pr=footer). Last update [10587a6...1708a23](https://codecov.io/gh/apache/kylin/pull/268?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
Re: [DISCUSS] Columnar storage engine for Apache Kylin
Hi Shaofeng, I'd like to add one more character: cloud-native storage support. Quite a few users are using S3 on AWS, or Azure Data Lake Storage on Azure. If new storage engine could be more cloud friendly, more user could get benefits from it. With Warm regards Billy Liu ShaoFeng Shi 于2018年9月28日周五 下午2:15写道： > > Hi Kylin developers. > > HBase has been Kylin’s storage engine since the first day; Kylin on HBase > has been verified as a success which can support low latency & high > concurrency queries on a very large data scale. Thanks to HBase, most Kylin > users can get on average less than 1-second query response. > > But we also see some limitations when putting Cubes into HBase; I shared > some of them in the HBaseConf Asia 2018[1] this August. The typical > limitations include: > >- Rowkey is the primary index, no secondary index so far; > > Filtering by row key’s prefix and suffix can get very different performance > result. So the user needs to do a good design about the row key; otherwise, > the query would be slow. This is difficult sometimes because the user might > not predict the filtering patterns ahead of cube design. > >- HBase is a key-value instead of a columnar storage > > Kylin combines multiple measures (columns) into fewer column families for > smaller data size (row key size is remarkable). This causes HBase often > needing to read more data than requested. > >- HBase couldn't run on YARN > > This makes the deployment and auto-scaling a little complicated, especially > in the cloud. > > In one word, HBase is complicated to be Kylin’s storage. The maintenance, > debugging is also hard for normal developers. Now we’re planning to seek a > simple, light-weighted, read-only storage engine for Kylin. The new > solution should have the following characteristics: > >- Columnar layout with compression for efficient I/O; >- Index by each column for quick filtering and seeking; >- MapReduce / Spark API for parallel processing; >- HDFS compliant for scalability and availability; >- Mature, stable and extensible; > > With the plugin architecture[2] introduced in Kylin 1.5, adding multiple > storages to Kylin is possible. Some companies like Kyligence Inc and > Meituan.com, have developed their customized storage engine for Kylin in > their product or platform. In their experience, columnar storage is a good > supplement for the HBase engine. Kaisen Kang from Meituan.com has shared > their KOD (Kylin on Druid) solution[3] in this August’s Kylin meetup in > Beijing. > > We plan to do a PoC with Apache Parquet + Apache Spark in the next phase. > Parquet is a standard columnar file format and has been widely supported by > many projects like Hive, Impala, Drill, etc. Parquet is adding the page > level column index to support fine-grained filtering. Apache Spark can > provide the parallel computing over Parquet and can be deployed on > YARN/Mesos and Kubernetes. With this combination, the data persistence and > computation are separated, which makes the scaling in/out much easier than > before. Benefiting from Spark's flexibility, we can not only push down more > computation from Kylin to the Hadoop cluster. Except for Parquet, Apache > ORC is also a candidate. > > Now I raise this discussion to get your ideas about Kylin’s next-generation > storage engine. If you have good ideas or any related data, welcome discuss in > the community. > > Thank you! > > [1] Apache Kylin on HBase > https://www.slideshare.net/ShiShaoFeng1/apache-kylin-on-hbase-extreme-olap-engine-for-big-data > [2] Apache Kylin Plugin Architecture > https://kylin.apache.org/development/plugin_arch.html > [3] 基于Druid的Kylin存储引擎实践 https://blog.bcmeng.com/post/kylin-on-druid.html-- > Best regards, > > Shaofeng Shi 史少锋
[GitHub] coveralls commented on issue #274: KYLIN-3597 Improve code smell
coveralls commented on issue #274: KYLIN-3597 Improve code smell URL: https://github.com/apache/kylin/pull/274#issuecomment-425620750 ## Pull Request Test Coverage Report for [Build 3713](https://coveralls.io/builds/19260169) * **21** of **38** **(55.26%)** changed or added relevant lines in **3** files are covered. * **2** unchanged lines in **2** files lost coverage. * Overall coverage increased (+**0.002%**) to **23.174%** --- | Changes Missing Coverage | Covered Lines | Changed/Added Lines | % | | :-|--||---: | | [core-cube/src/main/java/org/apache/kylin/gridtable/GTUtil.java](https://coveralls.io/builds/19260169/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fgridtable%2FGTUtil.java#L197) | 0 | 2 | 0.0% | [server-base/src/main/java/org/apache/kylin/rest/controller/CubeController.java](https://coveralls.io/builds/19260169/source?filename=server-base%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Frest%2Fcontroller%2FCubeController.java#L370) | 0 | 2 | 0.0% | [core-cube/src/main/java/org/apache/kylin/gridtable/GTFilterScanner.java](https://coveralls.io/builds/19260169/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fgridtable%2FGTFilterScanner.java#L85) | 21 | 34 | 61.76% | Files with Coverage Reduction | New Missed Lines | % | | :-|--|--: | | [server-base/src/main/java/org/apache/kylin/rest/controller/CubeController.java](https://coveralls.io/builds/19260169/source?filename=server-base%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Frest%2Fcontroller%2FCubeController.java#L369) | 1 | 0.0% | | [core-common/src/main/java/org/apache/kylin/common/persistence/IdentifierFileResourceStore.java](https://coveralls.io/builds/19260169/source?filename=core-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcommon%2Fpersistence%2FIdentifierFileResourceStore.java#L40) | 1 | 0.0% | | Totals | [](https://coveralls.io/builds/19260169) | | :-- | --: | | Change from base [Build 3711](https://coveralls.io/builds/19259592): | 0.002% | | Covered Lines: | 16162 | | Relevant Lines: | 69742 | --- # - [Coveralls](https://coveralls.io) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io commented on issue #274: KYLIN-3597 Improve code smell
codecov-io commented on issue #274: KYLIN-3597 Improve code smell URL: https://github.com/apache/kylin/pull/274#issuecomment-425620454 # [Codecov](https://codecov.io/gh/apache/kylin/pull/274?src=pr=h1) Report > Merging [#274](https://codecov.io/gh/apache/kylin/pull/274?src=pr=desc) into [master](https://codecov.io/gh/apache/kylin/commit/10587a65fe0552179a5c8a6e1151686ce1c8a135?src=pr=desc) will **increase** coverage by `<.01%`. > The diff coverage is `42.1%`. [](https://codecov.io/gh/apache/kylin/pull/274?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #274 +/- ## + Coverage 21.16% 21.16% +<.01% Complexity 4405 4405 Files 1086 1086 Lines 6974569742 -3 Branches 1008810087 -1 Hits 1476114761 + Misses5358653582 -4 - Partials 1398 1399 +1 ``` | [Impacted Files](https://codecov.io/gh/apache/kylin/pull/274?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...torage/gtrecord/SortedIteratorMergerWithLimit.java](https://codecov.io/gh/apache/kylin/pull/274/diff?src=pr=tree#diff-Y29yZS1zdG9yYWdlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9zdG9yYWdlL2d0cmVjb3JkL1NvcnRlZEl0ZXJhdG9yTWVyZ2VyV2l0aExpbWl0LmphdmE=) | `78.72% <ø> (ø)` | `2 <0> (ø)` | :arrow_down: | | [...ommon/persistence/IdentifierFileResourceStore.java](https://codecov.io/gh/apache/kylin/pull/274/diff?src=pr=tree#diff-Y29yZS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL2NvbW1vbi9wZXJzaXN0ZW5jZS9JZGVudGlmaWVyRmlsZVJlc291cmNlU3RvcmUuamF2YQ==) | `0% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...c/main/java/org/apache/kylin/gridtable/GTUtil.java](https://codecov.io/gh/apache/kylin/pull/274/diff?src=pr=tree#diff-Y29yZS1jdWJlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9ncmlkdGFibGUvR1RVdGlsLmphdmE=) | `1.37% <0%> (ø)` | `2 <0> (ø)` | :arrow_down: | | [...g/apache/kylin/rest/controller/CubeController.java](https://codecov.io/gh/apache/kylin/pull/274/diff?src=pr=tree#diff-c2VydmVyLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL3Jlc3QvY29udHJvbGxlci9DdWJlQ29udHJvbGxlci5qYXZh) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...va/org/apache/kylin/gridtable/GTFilterScanner.java](https://codecov.io/gh/apache/kylin/pull/274/diff?src=pr=tree#diff-Y29yZS1jdWJlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9ncmlkdGFibGUvR1RGaWx0ZXJTY2FubmVyLmphdmE=) | `37.34% <47.05%> (+0.76%)` | `2 <1> (ø)` | :arrow_down: | | [...lin/dict/lookup/cache/RocksDBLookupTableCache.java](https://codecov.io/gh/apache/kylin/pull/274/diff?src=pr=tree#diff-Y29yZS1kaWN0aW9uYXJ5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9kaWN0L2xvb2t1cC9jYWNoZS9Sb2Nrc0RCTG9va3VwVGFibGVDYWNoZS5qYXZh) | `76.16% <0%> (-0.52%)` | `27% <0%> (ø)` | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/kylin/pull/274?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/kylin/pull/274?src=pr=footer). Last update [10587a6...fbb2f66](https://codecov.io/gh/apache/kylin/pull/274?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] hit-lacus opened a new pull request #274: KYLIN-3597 Improve code smell
hit-lacus opened a new pull request #274: KYLIN-3597 Improve code smell URL: https://github.com/apache/kylin/pull/274 https://issues.apache.org/jira/projects/KYLIN/issues/KYLIN-3597 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] asfgit commented on issue #274: KYLIN-3597 Improve code smell
asfgit commented on issue #274: KYLIN-3597 Improve code smell URL: https://github.com/apache/kylin/pull/274#issuecomment-425619076 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] asfgit commented on issue #274: KYLIN-3597 Improve code smell
asfgit commented on issue #274: KYLIN-3597 Improve code smell URL: https://github.com/apache/kylin/pull/274#issuecomment-425619077 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
