kylin git commit: KYLIN-2367 ammend test case

[liyang]

Repository: kylin
Updated Branches:
  refs/heads/master b0aadb12d -> 07e5e5a75


KYLIN-2367 ammend test case


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/07e5e5a7
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/07e5e5a7
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/07e5e5a7

Branch: refs/heads/master
Commit: 07e5e5a75f4f692a53ac48c64c3cb44b84022c88
Parents: b0aadb1
Author: Li Yang 
Authored: Tue Jan 10 15:45:05 2017 +0800
Committer: Li Yang 
Committed: Tue Jan 10 15:45:05 2017 +0800

--
 .../test_case_data/localmeta/table/DEFAULT.TEST_KYLIN_FACT.json| 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/07e5e5a7/examples/test_case_data/localmeta/table/DEFAULT.TEST_KYLIN_FACT.json
--
diff --git 
a/examples/test_case_data/localmeta/table/DEFAULT.TEST_KYLIN_FACT.json 
b/examples/test_case_data/localmeta/table/DEFAULT.TEST_KYLIN_FACT.json
index 2b33e66..df53de9 100644
--- a/examples/test_case_data/localmeta/table/DEFAULT.TEST_KYLIN_FACT.json
+++ b/examples/test_case_data/localmeta/table/DEFAULT.TEST_KYLIN_FACT.json
@@ -48,7 +48,7 @@
   }, {
 "id" : "10",
 "name" : "ITEM_COUNT",
-"datatype" : "int",
+"datatype" : "bigint",
 "data_gen" : "RAND"
   }, {
 "id" : "11",

kylin git commit: alter query test case to fix CI

[liyang]

Repository: kylin
Updated Branches:
  refs/heads/yang22 2fec0603e -> fb424042e


alter query test case to fix CI


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/fb424042
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/fb424042
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/fb424042

Branch: refs/heads/yang22
Commit: fb424042ef3e29efe26e90874e0ff425148b89b6
Parents: 2fec060
Author: Li Yang 
Authored: Tue Jan 10 15:27:14 2017 +0800
Committer: Li Yang 
Committed: Tue Jan 10 15:27:14 2017 +0800

--
 .../test_case_data/localmeta/table/DEFAULT.TEST_KYLIN_FACT.json| 2 +-
 kylin-it/src/test/resources/query/sql_limit/query01.sql| 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/fb424042/examples/test_case_data/localmeta/table/DEFAULT.TEST_KYLIN_FACT.json
--
diff --git 
a/examples/test_case_data/localmeta/table/DEFAULT.TEST_KYLIN_FACT.json 
b/examples/test_case_data/localmeta/table/DEFAULT.TEST_KYLIN_FACT.json
index fdfb322..53dc96b 100644
--- a/examples/test_case_data/localmeta/table/DEFAULT.TEST_KYLIN_FACT.json
+++ b/examples/test_case_data/localmeta/table/DEFAULT.TEST_KYLIN_FACT.json
@@ -33,7 +33,7 @@
   }, {
 "id" : "8",
 "name" : "ITEM_COUNT",
-"datatype" : "int"
+"datatype" : "bigint"
   }, {
 "id" : "9",
 "name" : "SELLER_ID",

http://git-wip-us.apache.org/repos/asf/kylin/blob/fb424042/kylin-it/src/test/resources/query/sql_limit/query01.sql
--
diff --git a/kylin-it/src/test/resources/query/sql_limit/query01.sql 
b/kylin-it/src/test/resources/query/sql_limit/query01.sql
index fca8175..296863f 100644
--- a/kylin-it/src/test/resources/query/sql_limit/query01.sql
+++ b/kylin-it/src/test/resources/query/sql_limit/query01.sql
@@ -16,6 +16,6 @@
 -- limitations under the License.
 --
 
-select * from test_kylin_fact
+select cal_dt, lstg_format_name, seller_id from test_kylin_fact
   where lstg_format_name='FP-GTC' 
  limit 20

kylin git commit: move protobuf 3.1.0 to test scope

[billyliu]

Repository: kylin
Updated Branches:
  refs/heads/calcite-11 a99f5d331 -> bbb0ed855


move protobuf 3.1.0 to test scope


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/bbb0ed85
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/bbb0ed85
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/bbb0ed85

Branch: refs/heads/calcite-11
Commit: bbb0ed855e2e1b57a1825ff4eb44c1b18c16af90
Parents: a99f5d3
Author: Billy Liu 
Authored: Tue Jan 10 11:46:25 2017 +0800
Committer: Billy Liu 
Committed: Tue Jan 10 11:46:25 2017 +0800

--
 server/pom.xml | 12 ++--
 1 file changed, 6 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/bbb0ed85/server/pom.xml
--
diff --git a/server/pom.xml b/server/pom.xml
index b40883a..4416c2a 100644
--- a/server/pom.xml
+++ b/server/pom.xml
@@ -50,12 +50,6 @@
 
 
 
-com.google.protobuf
-protobuf-java
-${avatica.protobuf.version}
-
-
-
 com.thetransactioncompany
 cors-filter
 
@@ -91,6 +85,12 @@
 
 
 
+com.google.protobuf
+protobuf-java
+${avatica.protobuf.version}
+test
+
+
 org.apache.kylin
 kylin-core-common
 test-jar

kylin git commit: KYLIN-2348 allow multiple models serving one query

[liyang]

Repository: kylin
Updated Branches:
  refs/heads/master 00c8f31e6 -> 3a36c66a1


KYLIN-2348 allow multiple models serving one query


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/3a36c66a
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/3a36c66a
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/3a36c66a

Branch: refs/heads/master
Commit: 3a36c66a1c2bbb372b0a3f3243f181332d7ed882
Parents: 00c8f31
Author: Li Yang 
Authored: Tue Jan 10 11:32:58 2017 +0800
Committer: Li Yang 
Committed: Tue Jan 10 11:44:17 2017 +0800

--
 .../apache/kylin/common/util/Dictionary.java|  2 -
 .../apache/kylin/query/ITKylinQueryTest.java| 16 +--
 .../org/apache/kylin/query/KylinTestBase.java   |  1 +
 .../src/test/resources/query/sql/query101.sql   | 18 +++
 .../src/test/resources/query/sql/query102.sql   | 18 +++
 .../src/test/resources/query/sql/query103.sql   | 18 +++
 .../resources/query/sql_multi_model/query01.sql | 49 
 .../relnode/OLAPToEnumerableConverter.java  | 18 +++
 .../kylin/query/routing/ModelChooser.java   | 43 +
 .../apache/kylin/query/schema/OLAPTable.java|  8 ++--
 10 files changed, 161 insertions(+), 30 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/3a36c66a/core-common/src/main/java/org/apache/kylin/common/util/Dictionary.java
--
diff --git 
a/core-common/src/main/java/org/apache/kylin/common/util/Dictionary.java 
b/core-common/src/main/java/org/apache/kylin/common/util/Dictionary.java
index 03996a7..1e172bc 100644
--- a/core-common/src/main/java/org/apache/kylin/common/util/Dictionary.java
+++ b/core-common/src/main/java/org/apache/kylin/common/util/Dictionary.java
@@ -254,6 +254,4 @@ abstract public class Dictionary implements Serializable 
{
  */
 public abstract void readFields(DataInput in) throws IOException;
 
-
-
 }

http://git-wip-us.apache.org/repos/asf/kylin/blob/3a36c66a/kylin-it/src/test/java/org/apache/kylin/query/ITKylinQueryTest.java
--
diff --git 
a/kylin-it/src/test/java/org/apache/kylin/query/ITKylinQueryTest.java 
b/kylin-it/src/test/java/org/apache/kylin/query/ITKylinQueryTest.java
index 840a03f..aaf0925 100644
--- a/kylin-it/src/test/java/org/apache/kylin/query/ITKylinQueryTest.java
+++ b/kylin-it/src/test/java/org/apache/kylin/query/ITKylinQueryTest.java
@@ -47,7 +47,7 @@ import org.junit.rules.ExpectedException;
 
 import com.google.common.collect.Maps;
 
-@Ignore("KylinQueryTest is contained by ITCombinationTest")
+//@Ignore("KylinQueryTest is contained by ITCombinationTest")
 public class ITKylinQueryTest extends KylinTestBase {
 
 @Rule
@@ -149,7 +149,7 @@ public class ITKylinQueryTest extends KylinTestBase {
 @Test
 public void testSingleRunQuery() throws Exception {
 
-String queryFileName = getQueryFolderPrefix() + 
"src/test/resources/query/temp/sample.sql";
+String queryFileName = getQueryFolderPrefix() + 
"src/test/resources/query/sql_multi_model/query01.sql";
 
 File sqlFile = new File(queryFileName);
 if (sqlFile.exists()) {
@@ -270,7 +270,17 @@ public class ITKylinQueryTest extends KylinTestBase {
 this.batchExecuteQuery(getQueryFolderPrefix() + 
"src/test/resources/query/sql_intersect_count");
 }
 }
-
+
+
+@Test
+public void testMultiModelQuery() throws Exception {
+if ("left".equalsIgnoreCase(joinType)) {
+joinType = "default";
+execAndCompQuery(getQueryFolderPrefix() + 
"src/test/resources/query/sql_multi_model", null, true);
+joinType = "left";
+}
+}
+
 @Test
 public void testDimDistinctCountQuery() throws Exception {
 execAndCompQuery(getQueryFolderPrefix() + 
"src/test/resources/query/sql_distinct_dim", null, true);

http://git-wip-us.apache.org/repos/asf/kylin/blob/3a36c66a/kylin-it/src/test/java/org/apache/kylin/query/KylinTestBase.java
--
diff --git a/kylin-it/src/test/java/org/apache/kylin/query/KylinTestBase.java 
b/kylin-it/src/test/java/org/apache/kylin/query/KylinTestBase.java
index 4f1f591..ae69f09 100644
--- a/kylin-it/src/test/java/org/apache/kylin/query/KylinTestBase.java
+++ b/kylin-it/src/test/java/org/apache/kylin/query/KylinTestBase.java
@@ -335,6 +335,7 @@ public class KylinTestBase {
 printInfo("Query Result from H2 - " + queryName);
 H2Connection h2Conn = new H2Connection(h2Connection, null);
 
h2Conn.getConfig().setProperty(DatabaseConfig.PROPERTY_DATATYPE_FACTORY, new 
TestH2DataTypeFactory());
+  

[1/8] kylin git commit: KYLIN-2371 Allow overwrite default spark conf at cube level

[shaofengshi]

Repository: kylin
Updated Branches:
  refs/heads/master 75625ae51 -> ee74a74e4


KYLIN-2371 Allow overwrite default spark conf at cube level


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/ee74a74e
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/ee74a74e
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/ee74a74e

Branch: refs/heads/master
Commit: ee74a74e4d17a45b3832912381220af39e71f3d5
Parents: 66bca9a
Author: shaofengshi 
Authored: Mon Jan 9 15:23:22 2017 +0800
Committer: shaofengshi 
Committed: Mon Jan 9 16:58:11 2017 +0800

--
 build/conf/kylin-spark-conf.properties  |  5 +--
 build/conf/kylin.properties |  6 ++--
 .../apache/kylin/common/KylinConfigBase.java| 12 ---
 .../kylin/job/constant/ExecutableConstants.java |  1 +
 .../spark/SparkBatchCubingJobBuilder2.java  |  2 +-
 .../kylin/engine/spark/SparkCubingByLayer.java  |  7 ++--
 .../kylin/engine/spark/SparkExecutable.java | 38 +---
 .../test_case_data/sandbox/kylin.properties |  3 +-
 8 files changed, 55 insertions(+), 19 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/ee74a74e/build/conf/kylin-spark-conf.properties
--
diff --git a/build/conf/kylin-spark-conf.properties 
b/build/conf/kylin-spark-conf.properties
index 5e6dafe..143e7e4 100644
--- a/build/conf/kylin-spark-conf.properties
+++ b/build/conf/kylin-spark-conf.properties
@@ -1,5 +1,5 @@
 spark.yarn.submit.file.replication=1
-spark.yarn.executor.memoryOverhead=200
+spark.yarn.executor.memoryOverhead=1024
 spark.yarn.driver.memoryOverhead=384
 spark.master=yarn
 spark.submit.deployMode=cluster
@@ -20,7 +20,8 @@ spark.executor.cores=4
 spark.executor.instances=8
 spark.history.kerberos.keytab=none
 spark.history.kerberos.principal=none
-#spark.yarn.jar=hdfs://namenode:8020/apps/spark/spark-assembly-1.6.3-hadoop2.6.0.jar
 spark.driver.extraJavaOptions=-Dhdp.version=current
 spark.yarn.am.extraJavaOptions=-Dhdp.version=current
 spark.executor.extraJavaOptions=-Dhdp.version=current
+#spark.yarn.jar=hdfs://namenode:8020/kylin/spark/spark-assembly-1.6.3-hadoop2.6.0.jar
+#spark.io.compression.codec=org.apache.spark.io.SnappyCompressionCodec

http://git-wip-us.apache.org/repos/asf/kylin/blob/ee74a74e/build/conf/kylin.properties
--
diff --git a/build/conf/kylin.properties b/build/conf/kylin.properties
index 54430f0..6efa423 100644
--- a/build/conf/kylin.properties
+++ b/build/conf/kylin.properties
@@ -131,16 +131,16 @@ kylin.engine.mr.mapper-input-rows=100
 ### Spark Engine ###
 
 # Hadoop conf folder, will export this as "HADOOP_CONF_DIR" before run 
spark-submit
-kylin.engine.spark.env.hadoop-conf-dir=/etc/hadoop/conf
+#kylin.engine.spark.env.hadoop-conf-dir=/etc/hive/conf
 
 # Spark job submission properties file, default be 
$KYLIN_HOME/conf/kylin-spark-conf.properties
 #kylin.engine.spark.properties-file=
 
 # Estimate the RDD partition numbers
-kylin.engine.spark.rdd-partition-cut-mb=50
+kylin.engine.spark.rdd-partition-cut-mb=10
 
 # Minimal partition numbers of rdd
-kylin.engine.spark.min-partition=10
+kylin.engine.spark.min-partition=1
 
 # Max partition numbers of rdd
 kylin.engine.spark.max-partition=5000

http://git-wip-us.apache.org/repos/asf/kylin/blob/ee74a74e/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
--
diff --git 
a/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java 
b/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
index 7d6ac2b..36ddbf4 100644
--- a/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
+++ b/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
@@ -695,6 +695,10 @@ abstract public class KylinConfigBase implements 
Serializable {
 return getPropertiesByPrefix("kylin.engine.mr.config-override.");
 }
 
+public Map getSparkConfigOverride() {
+return getPropertiesByPrefix("kylin.engine.spark.config-override.");
+}
+
 public double getDefaultHadoopJobReducerInputMB() {
 return 
Double.parseDouble(getOptional("kylin.engine.mr.reduce-input-mb", "500"));
 }
@@ -739,8 +743,8 @@ abstract public class KylinConfigBase implements 
Serializable {
 // ENGINE.SPARK
 // 

 
-public String getSparkHadoopConfDir() {
-return getRequired("kylin.engine.spark.env.hadoop-conf-dir");
+public String getHadoopConfDir() {
+return 

[4/8] kylin git commit: KYLIN-2331 by layer spark cubing

[shaofengshi]

http://git-wip-us.apache.org/repos/asf/kylin/blob/b7d87bb4/core-metadata/src/main/java/org/apache/kylin/measure/MeasureAggregators.java
--
diff --git 
a/core-metadata/src/main/java/org/apache/kylin/measure/MeasureAggregators.java 
b/core-metadata/src/main/java/org/apache/kylin/measure/MeasureAggregators.java
index 8951b69..710f324 100644
--- 
a/core-metadata/src/main/java/org/apache/kylin/measure/MeasureAggregators.java
+++ 
b/core-metadata/src/main/java/org/apache/kylin/measure/MeasureAggregators.java
@@ -84,6 +84,24 @@ public class MeasureAggregators implements Serializable {
 }
 }
 
+public void aggregate(Object[] values1, Object[] values2, Object[] result) 
{
+assert values1.length == values2.length && values2.length == 
descLength && values1.length == result.length;
+
+for (int i = 0; i < descLength; i++) {
+result[i] = aggs[i].aggregate(values1[i], values2[i]);
+}
+
+}
+
+public void aggregate(Object[] values1, Object[] values2, Object[] result, 
boolean[] aggrMask) {
+assert values1.length == values2.length && values2.length == 
descLength && values1.length == result.length && result.length == 
aggrMask.length;
+for (int i = 0; i < descLength; i++) {
+if (aggrMask[i]) {
+result[i] = aggs[i].aggregate(values1[i], values2[i]);
+}
+}
+}
+
 public void collectStates(Object[] states) {
 for (int i = 0; i < descLength; i++) {
 states[i] = aggs[i].getState();

http://git-wip-us.apache.org/repos/asf/kylin/blob/b7d87bb4/core-metadata/src/main/java/org/apache/kylin/measure/MeasureCodec.java
--
diff --git 
a/core-metadata/src/main/java/org/apache/kylin/measure/MeasureCodec.java 
b/core-metadata/src/main/java/org/apache/kylin/measure/MeasureCodec.java
index edaf806..2d73e59 100644
--- a/core-metadata/src/main/java/org/apache/kylin/measure/MeasureCodec.java
+++ b/core-metadata/src/main/java/org/apache/kylin/measure/MeasureCodec.java
@@ -18,19 +18,19 @@
 
 package org.apache.kylin.measure;
 
-import java.nio.ByteBuffer;
-import java.util.Collection;
-
 import org.apache.kylin.metadata.datatype.DataType;
 import org.apache.kylin.metadata.datatype.DataTypeSerializer;
 import org.apache.kylin.metadata.model.MeasureDesc;
 
+import java.nio.ByteBuffer;
+import java.util.Collection;
+
 /**
  * @author yangli9
  * 
  */
 @SuppressWarnings({ "rawtypes" })
-public class MeasureCodec {
+public class MeasureCodec implements java.io.Serializable {
 
 private int nMeasures;
 private DataTypeSerializer[] serializers;

http://git-wip-us.apache.org/repos/asf/kylin/blob/b7d87bb4/core-metadata/src/main/java/org/apache/kylin/measure/MeasureIngester.java
--
diff --git 
a/core-metadata/src/main/java/org/apache/kylin/measure/MeasureIngester.java 
b/core-metadata/src/main/java/org/apache/kylin/measure/MeasureIngester.java
index 0076252..26b7298 100644
--- a/core-metadata/src/main/java/org/apache/kylin/measure/MeasureIngester.java
+++ b/core-metadata/src/main/java/org/apache/kylin/measure/MeasureIngester.java
@@ -18,14 +18,14 @@
 
 package org.apache.kylin.measure;
 
-import java.util.Collection;
-import java.util.Map;
-
 import org.apache.kylin.common.util.Dictionary;
 import org.apache.kylin.metadata.model.MeasureDesc;
 import org.apache.kylin.metadata.model.TblColRef;
 
-abstract public class MeasureIngester {
+import java.util.Collection;
+import java.util.Map;
+
+abstract public class MeasureIngester implements java.io.Serializable {
 
 public static MeasureIngester create(MeasureDesc measure) {
 return measure.getFunction().getMeasureType().newIngester();
@@ -42,6 +42,10 @@ abstract public class MeasureIngester {
 
 abstract public V valueOf(String[] values, MeasureDesc measureDesc, 
Map dictionaryMap);
 
+public void reset() {
+
+}
+
 public V reEncodeDictionary(V value, MeasureDesc measureDesc, 
Map oldDicts, Map 
newDicts) {
 throw new UnsupportedOperationException();
 }

http://git-wip-us.apache.org/repos/asf/kylin/blob/b7d87bb4/core-metadata/src/main/java/org/apache/kylin/measure/MeasureType.java
--
diff --git 
a/core-metadata/src/main/java/org/apache/kylin/measure/MeasureType.java 
b/core-metadata/src/main/java/org/apache/kylin/measure/MeasureType.java
index 89ff382..3338c8c 100644
--- a/core-metadata/src/main/java/org/apache/kylin/measure/MeasureType.java
+++ b/core-metadata/src/main/java/org/apache/kylin/measure/MeasureType.java
@@ -18,11 +18,6 @@
 
 package org.apache.kylin.measure;
 
-import java.util.Collection;
-import java.util.Collections;
-import java.util.List;
-import java.util.Map;

[3/8] kylin git commit: KYLIN-2331 by layer spark cubing

[shaofengshi]

http://git-wip-us.apache.org/repos/asf/kylin/blob/b7d87bb4/engine-mr/src/main/java/org/apache/kylin/engine/mr/common/BaseCuboidBuilder.java
--
diff --git 
a/engine-mr/src/main/java/org/apache/kylin/engine/mr/common/BaseCuboidBuilder.java
 
b/engine-mr/src/main/java/org/apache/kylin/engine/mr/common/BaseCuboidBuilder.java
new file mode 100644
index 000..07b636b
--- /dev/null
+++ 
b/engine-mr/src/main/java/org/apache/kylin/engine/mr/common/BaseCuboidBuilder.java
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+*/
+
+package org.apache.kylin.engine.mr.common;
+
+import com.google.common.collect.Sets;
+import org.apache.kylin.common.KylinConfig;
+import org.apache.kylin.common.util.Dictionary;
+import org.apache.kylin.cube.CubeSegment;
+import org.apache.kylin.cube.cuboid.Cuboid;
+import org.apache.kylin.cube.kv.AbstractRowKeyEncoder;
+import org.apache.kylin.cube.model.CubeDesc;
+import org.apache.kylin.cube.model.CubeJoinedFlatTableEnrich;
+import org.apache.kylin.measure.BufferedMeasureCodec;
+import org.apache.kylin.measure.MeasureIngester;
+import org.apache.kylin.metadata.model.FunctionDesc;
+import org.apache.kylin.metadata.model.MeasureDesc;
+import org.apache.kylin.metadata.model.ParameterDesc;
+import org.apache.kylin.metadata.model.TblColRef;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ */
+public class BaseCuboidBuilder implements java.io.Serializable {
+
+protected static final Logger logger = 
LoggerFactory.getLogger(BaseCuboidBuilder.class);
+public static final String HIVE_NULL = "\\N";
+protected String cubeName;
+protected Cuboid baseCuboid;
+protected CubeDesc cubeDesc;
+protected CubeSegment cubeSegment;
+protected Set nullStrs;
+protected CubeJoinedFlatTableEnrich intermediateTableDesc;
+protected MeasureIngester[] aggrIngesters;
+protected Map dictionaryMap;
+protected AbstractRowKeyEncoder rowKeyEncoder;
+protected BufferedMeasureCodec measureCodec;
+
+protected KylinConfig kylinConfig;
+
+public BaseCuboidBuilder(KylinConfig kylinConfig, CubeDesc cubeDesc, 
CubeSegment cubeSegment, CubeJoinedFlatTableEnrich intermediateTableDesc,
+ AbstractRowKeyEncoder rowKeyEncoder, 
MeasureIngester[] aggrIngesters, Map 
dictionaryMap) {
+this.kylinConfig = kylinConfig;
+this.cubeDesc = cubeDesc;
+this.cubeSegment = cubeSegment;
+this.intermediateTableDesc = intermediateTableDesc;
+this.rowKeyEncoder = rowKeyEncoder;
+this.aggrIngesters = aggrIngesters;
+this.dictionaryMap = dictionaryMap;
+
+init();
+measureCodec = new BufferedMeasureCodec(cubeDesc.getMeasures());
+}
+
+public BaseCuboidBuilder(KylinConfig kylinConfig, CubeDesc cubeDesc, 
CubeSegment cubeSegment, CubeJoinedFlatTableEnrich intermediateTableDesc) {
+this.kylinConfig = kylinConfig;
+this.cubeDesc = cubeDesc;
+this.cubeSegment = cubeSegment;
+this.intermediateTableDesc = intermediateTableDesc;
+
+init();
+rowKeyEncoder = AbstractRowKeyEncoder.createInstance(cubeSegment, 
baseCuboid);
+measureCodec = new BufferedMeasureCodec(cubeDesc.getMeasures());
+aggrIngesters = MeasureIngester.create(cubeDesc.getMeasures());
+dictionaryMap = cubeSegment.buildDictionaryMap();
+
+}
+
+private void init() {
+long baseCuboidId = Cuboid.getBaseCuboidId(cubeDesc);
+baseCuboid = Cuboid.findById(cubeDesc, baseCuboidId);
+initNullBytes();
+}
+
+private void initNullBytes() {
+nullStrs = Sets.newHashSet();
+nullStrs.add(HIVE_NULL);
+String[] nullStrings = cubeDesc.getNullStrings();
+if (nullStrings != null) {
+for (String s : nullStrings) {
+nullStrs.add(s);
+}
+}
+}
+
+protected boolean isNull(String v) {
+return nullStrs.contains(v);
+}
+
+public byte[] buildKey(String[] 

[6/8] kylin git commit: KYLIN-2344 Package spark into Kylin binary package

[shaofengshi]

KYLIN-2344 Package spark into Kylin binary package


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/e864cd3b
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/e864cd3b
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/e864cd3b

Branch: refs/heads/master
Commit: e864cd3b1c06700b1a1054d1f520eaabedc25d82
Parents: b7d87bb
Author: shaofengshi 
Authored: Tue Jan 3 10:28:38 2017 +0800
Committer: shaofengshi 
Committed: Mon Jan 9 16:58:11 2017 +0800

--
 build/bin/check-env.sh  |  6 ++
 build/bin/find-spark-dependency.sh  | 45 +++
 build/bin/kylin.sh  |  6 ++
 build/conf/kylin-spark-conf.properties  |  2 +-
 build/conf/kylin.properties |  3 -
 build/script/compress.sh|  5 +-
 build/script/download-spark.sh  | 52 +
 build/script/functions.sh   | 60 
 build/script/package.sh |  1 +
 .../org/apache/kylin/common/KylinConfig.java| 14 +
 .../apache/kylin/common/KylinConfigBase.java| 27 -
 .../spark/SparkBatchCubingJobBuilder2.java  |  4 +-
 12 files changed, 200 insertions(+), 25 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/e864cd3b/build/bin/check-env.sh
--
diff --git a/build/bin/check-env.sh b/build/bin/check-env.sh
index a4003c9..e446d66 100644
--- a/build/bin/check-env.sh
+++ b/build/bin/check-env.sh
@@ -47,3 +47,9 @@ if [ $? != 0 ]
 then
 quit "Failed to create $WORKING_DIR. Please make sure the user has right 
to access $WORKING_DIR"
 fi
+
+hadoop fs -mkdir -p $WORKING_DIR/spark-history
+if [ $? != 0 ]
+then
+quit "Failed to create $WORKING_DIR/spark-history. Please make sure the 
user has right to access $WORKING_DIR"
+fi
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/kylin/blob/e864cd3b/build/bin/find-spark-dependency.sh
--
diff --git a/build/bin/find-spark-dependency.sh 
b/build/bin/find-spark-dependency.sh
new file mode 100644
index 000..6f74d8a
--- /dev/null
+++ b/build/bin/find-spark-dependency.sh
@@ -0,0 +1,45 @@
+#!/bin/bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+source $(cd -P -- "$(dirname -- "$0")" && pwd -P)/header.sh
+
+spark_home=
+
+verbose Retrieving Spark dependency...
+if [ -n "$SPARK_HOME" ]
+then
+verbose "SPARK_HOME is set to: $SPARK_HOME, use it to locate Spark 
dependencies."
+spark_home=$SPARK_HOME
+fi
+
+if [ -z "$SPARK_HOME" ]
+then
+verbose "SPARK_HOME wasn't set, use $KYLIN_HOME/spark"
+spark_home=$KYLIN_HOME/spark
+fi
+
+spark_dependency=`find -L $spark_home -name 
'spark-assembly-[a-z0-9A-Z\.-]*.jar' ! -name '*doc*' ! -name '*test*' ! -name 
'*sources*' ''-printf '%p:' | sed 's/:$//'`
+if [ -z "$spark_dependency" ]
+then
+quit "spark assembly lib not found"
+else
+verbose "spark dependency: $spark_dependency"
+export spark_dependency
+fi
+

http://git-wip-us.apache.org/repos/asf/kylin/blob/e864cd3b/build/bin/kylin.sh
--
diff --git a/build/bin/kylin.sh b/build/bin/kylin.sh
index 0cdbbc6..7813b79 100644
--- a/build/bin/kylin.sh
+++ b/build/bin/kylin.sh
@@ -45,6 +45,11 @@ function retrieveDependency() {
 source ${dir}/find-kafka-dependency.sh
 export HBASE_CLASSPATH=${HBASE_CLASSPATH}:${kafka_dependency}
 fi
+
+source ${dir}/find-spark-dependency.sh
+export HBASE_CLASSPATH=${HBASE_CLASSPATH}:${spark_dependency}
+
+verbose "HBASE_CLASSPATH: ${HBASE_CLASSPATH}"
 }
 
 # start command
@@ -112,6 +117,7 @@ then
 -Dkylin.hive.dependency=${hive_dependency} \
 -Dkylin.hbase.dependency=${hbase_dependency} \
 -Dkylin.kafka.dependency=${kafka_dependency} \
+-Dkylin.spark.dependency=${spark_dependency} \
 

[7/8] kylin git commit: fix NPE in CacheDictionary in Spark cubing

[shaofengshi]

fix NPE in CacheDictionary in Spark cubing


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/66bca9a6
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/66bca9a6
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/66bca9a6

Branch: refs/heads/master
Commit: 66bca9a676fda02aa88cc25bad2e545f779f6dde
Parents: 2cf52b4
Author: shaofengshi 
Authored: Sat Jan 7 10:17:33 2017 +0800
Committer: shaofengshi 
Committed: Mon Jan 9 16:58:11 2017 +0800

--
 .../java/org/apache/kylin/dict/CacheDictionary.java   |  2 +-
 .../apache/kylin/engine/mr/BatchCubingJobBuilder.java |  5 ++---
 .../kylin/engine/mr/BatchCubingJobBuilder2.java   |  7 +++
 .../org/apache/kylin/engine/mr/JobBuilderSupport.java | 14 +-
 .../engine/spark/SparkBatchCubingJobBuilder2.java |  1 +
 .../apache/kylin/engine/spark/SparkCubingByLayer.java |  2 +-
 .../kylin/engine/spark/SparkCubingJobBuilder.java |  1 -
 7 files changed, 9 insertions(+), 23 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/66bca9a6/core-dictionary/src/main/java/org/apache/kylin/dict/CacheDictionary.java
--
diff --git 
a/core-dictionary/src/main/java/org/apache/kylin/dict/CacheDictionary.java 
b/core-dictionary/src/main/java/org/apache/kylin/dict/CacheDictionary.java
index d7ed6bd..b2bad53 100644
--- a/core-dictionary/src/main/java/org/apache/kylin/dict/CacheDictionary.java
+++ b/core-dictionary/src/main/java/org/apache/kylin/dict/CacheDictionary.java
@@ -35,7 +35,7 @@ public abstract class CacheDictionary extends 
Dictionary {
 
 protected transient int baseId;
 
-protected transient BytesConverter bytesConvert;
+protected BytesConverter bytesConvert;
 
 public CacheDictionary() {
 

http://git-wip-us.apache.org/repos/asf/kylin/blob/66bca9a6/engine-mr/src/main/java/org/apache/kylin/engine/mr/BatchCubingJobBuilder.java
--
diff --git 
a/engine-mr/src/main/java/org/apache/kylin/engine/mr/BatchCubingJobBuilder.java 
b/engine-mr/src/main/java/org/apache/kylin/engine/mr/BatchCubingJobBuilder.java
index 456f615..36c12a1 100644
--- 
a/engine-mr/src/main/java/org/apache/kylin/engine/mr/BatchCubingJobBuilder.java
+++ 
b/engine-mr/src/main/java/org/apache/kylin/engine/mr/BatchCubingJobBuilder.java
@@ -65,12 +65,11 @@ public class BatchCubingJobBuilder extends 
JobBuilderSupport {
 // Phase 3: Build Cube
 RowKeyDesc rowKeyDesc = seg.getCubeDesc().getRowkey();
 final int groupRowkeyColumnsCount = seg.getCubeDesc().getBuildLevel();
-final String[] cuboidOutputTempPath = 
getCuboidOutputPaths(cuboidRootPath, groupRowkeyColumnsCount);
 // base cuboid step
-result.addTask(createBaseCuboidStep(cuboidOutputTempPath[0], jobId));
+
result.addTask(createBaseCuboidStep(getCuboidOutputPathsByLevel(cuboidRootPath, 
0), jobId));
 // n dim cuboid steps
 for (int i = 1; i <= groupRowkeyColumnsCount; i++) {
-result.addTask(createNDimensionCuboidStep(cuboidOutputTempPath[i - 
1], cuboidOutputTempPath[i], i));
+
result.addTask(createNDimensionCuboidStep(getCuboidOutputPathsByLevel(cuboidRootPath,
 i - 1), getCuboidOutputPathsByLevel(cuboidRootPath, i), i));
 }
 outputSide.addStepPhase3_BuildCube(result, cuboidRootPath);
 

http://git-wip-us.apache.org/repos/asf/kylin/blob/66bca9a6/engine-mr/src/main/java/org/apache/kylin/engine/mr/BatchCubingJobBuilder2.java
--
diff --git 
a/engine-mr/src/main/java/org/apache/kylin/engine/mr/BatchCubingJobBuilder2.java
 
b/engine-mr/src/main/java/org/apache/kylin/engine/mr/BatchCubingJobBuilder2.java
index 700f821..dd866bd 100644
--- 
a/engine-mr/src/main/java/org/apache/kylin/engine/mr/BatchCubingJobBuilder2.java
+++ 
b/engine-mr/src/main/java/org/apache/kylin/engine/mr/BatchCubingJobBuilder2.java
@@ -76,14 +76,13 @@ public class BatchCubingJobBuilder2 extends 
JobBuilderSupport {
 return result;
 }
 
-private void addLayerCubingSteps(final CubingJob result, final String 
jobId, final String cuboidRootPath) {
+protected void addLayerCubingSteps(final CubingJob result, final String 
jobId, final String cuboidRootPath) {
 final int maxLevel = seg.getCubeDesc().getBuildLevel();
-final String[] cuboidOutputTempPath = 
getCuboidOutputPaths(cuboidRootPath, maxLevel);
 // base cuboid step
-result.addTask(createBaseCuboidStep(cuboidOutputTempPath[0], jobId));
+
result.addTask(createBaseCuboidStep(getCuboidOutputPathsByLevel(cuboidRootPath, 
0), jobId));
 // n dim cuboid steps
 

[5/8] kylin git commit: KYLIN-2331 by layer spark cubing

[shaofengshi]

KYLIN-2331 by layer spark cubing


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/b7d87bb4
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/b7d87bb4
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/b7d87bb4

Branch: refs/heads/master
Commit: b7d87bb4e6c4de484d910e9af04965392da518e5
Parents: 75625ae
Author: shaofengshi 
Authored: Sun Dec 25 15:59:16 2016 +0800
Committer: shaofengshi 
Committed: Mon Jan 9 16:58:11 2017 +0800

--
 assembly/pom.xml|   5 +-
 .../main/config/assemblies/source-assembly.xml  |   3 -
 .../java/org/apache/kylin/job/DeployUtil.java   |   1 -
 build/conf/kylin-spark-conf.properties  |  26 ++
 build/conf/kylin.properties |  20 +
 .../apache/kylin/common/KylinConfigBase.java|  64 ++-
 .../kylin/common/persistence/ResourceStore.java |  35 ++
 .../org/apache/kylin/common/util/Array.java |   2 +-
 .../apache/kylin/common/util/SplittedBytes.java |   2 +-
 .../java/org/apache/kylin/cube/CubeSegment.java |   2 +-
 .../kylin/cube/common/RowKeySplitter.java   |  15 +-
 .../org/apache/kylin/cube/cuboid/Cuboid.java|  27 +-
 .../kylin/cube/cuboid/CuboidScheduler.java  |  15 +-
 .../cube/inmemcubing/InMemCubeBuilder.java  |   2 +
 .../kylin/cube/kv/AbstractRowKeyEncoder.java|   6 +-
 .../org/apache/kylin/cube/kv/CubeDimEncMap.java |   7 +-
 .../apache/kylin/cube/kv/RowKeyColumnIO.java|   2 +-
 .../org/apache/kylin/cube/kv/RowKeyEncoder.java |  19 +-
 .../kylin/cube/kv/RowKeyEncoderProvider.java|   2 +-
 .../kylin/cube/model/AggregationGroup.java  |   2 +-
 .../org/apache/kylin/cube/model/CubeDesc.java   |   2 +-
 .../cube/model/CubeJoinedFlatTableDesc.java |   2 +-
 .../cube/model/CubeJoinedFlatTableEnrich.java   |   6 +-
 .../apache/kylin/cube/model/DictionaryDesc.java |   4 +-
 .../apache/kylin/cube/model/DimensionDesc.java  |  14 +-
 .../kylin/cube/model/HBaseColumnDesc.java   |  11 +-
 .../kylin/cube/model/HBaseColumnFamilyDesc.java |   9 +-
 .../kylin/cube/model/HBaseMappingDesc.java  |  10 +-
 .../apache/kylin/cube/model/HierarchyDesc.java  |   2 +-
 .../apache/kylin/cube/model/RowKeyColDesc.java  |   2 +-
 .../org/apache/kylin/cube/model/RowKeyDesc.java |  18 +-
 .../org/apache/kylin/cube/model/SelectRule.java |   2 +-
 .../apache/kylin/gridtable/UnitTestSupport.java |  41 +-
 .../gridtable/AggregationCacheMemSizeTest.java  |   6 +-
 .../gridtable/AggregationCacheSpillTest.java|   5 +-
 .../kylin/gridtable/SimpleGridTableTest.java|  11 +-
 .../gridtable/SimpleInvertedIndexTest.java  |   3 +-
 .../metadata/measure/MeasureCodecTest.java  |   6 +-
 .../apache/kylin/dict/AppendTrieDictionary.java |   6 +-
 .../org/apache/kylin/dict/NumberDictionary.java |   2 +-
 .../apache/kylin/dict/NumberDictionary2.java|   2 +-
 .../dict/NumberDictionaryForestBuilder.java |   4 +-
 .../apache/kylin/dict/StringBytesConverter.java |   2 +-
 .../kylin/job/execution/ExecutableManager.java  |  21 +-
 .../kylin/dimension/AbstractDateDimEnc.java |  12 +-
 .../apache/kylin/dimension/BooleanDimEnc.java   |   4 +-
 .../kylin/dimension/DictionaryDimEnc.java   |  21 +-
 .../apache/kylin/dimension/FixedLenDimEnc.java  |   4 +-
 .../kylin/dimension/FixedLenHexDimEnc.java  |   4 +-
 .../org/apache/kylin/dimension/IntDimEnc.java   |   4 +-
 .../apache/kylin/dimension/IntegerDimEnc.java   |   4 +-
 .../kylin/dimension/OneMoreByteVLongDimEnc.java |  14 +-
 .../kylin/measure/BufferedMeasureCodec.java |  10 +-
 .../apache/kylin/measure/MeasureAggregator.java |   2 +
 .../kylin/measure/MeasureAggregators.java   |  18 +
 .../org/apache/kylin/measure/MeasureCodec.java  |   8 +-
 .../apache/kylin/measure/MeasureIngester.java   |  12 +-
 .../org/apache/kylin/measure/MeasureType.java   |  12 +-
 .../measure/basic/BigDecimalMaxAggregator.java  |  14 +
 .../measure/basic/BigDecimalMinAggregator.java  |  28 +-
 .../measure/basic/BigDecimalSumAggregator.java  |  10 +
 .../kylin/measure/basic/DoubleIngester.java |  19 +-
 .../measure/basic/DoubleMaxAggregator.java  |  20 +-
 .../measure/basic/DoubleMinAggregator.java  |  20 +-
 .../measure/basic/DoubleSumAggregator.java  |  18 +-
 .../kylin/measure/basic/LongIngester.java   |  19 +-
 .../kylin/measure/basic/LongMaxAggregator.java  |  20 +-
 .../kylin/measure/basic/LongMinAggregator.java  |  20 +-
 .../kylin/measure/basic/LongSumAggregator.java  |  18 +-
 .../kylin/measure/bitmap/BitmapAggregator.java  |  13 +
 .../kylin/measure/bitmap/BitmapCounter.java |   2 +-
 .../kylin/measure/bitmap/BitmapMeasureType.java |   5 +
 .../ExtendedColumnMeasureType.java  |  15 +
 .../kylin/measure/hllc/DenseRegister.java   |   2 +-
 .../kylin/measure/hllc/HLLCAggregator.java  |   7 +
 

[2/8] kylin git commit: modify default partition setting

[shaofengshi]

modify default partition setting


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/2cf52b49
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/2cf52b49
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/2cf52b49

Branch: refs/heads/master
Commit: 2cf52b49ed9a1158cb2ec0ded6d907f90ee024fe
Parents: b9a69ef
Author: shaofengshi 
Authored: Thu Jan 5 19:56:10 2017 +0800
Committer: shaofengshi 
Committed: Mon Jan 9 16:58:11 2017 +0800

--
 build/conf/kylin.properties | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/2cf52b49/build/conf/kylin.properties
--
diff --git a/build/conf/kylin.properties b/build/conf/kylin.properties
index bd0bbd4..54430f0 100644
--- a/build/conf/kylin.properties
+++ b/build/conf/kylin.properties
@@ -137,13 +137,13 @@ kylin.engine.spark.env.hadoop-conf-dir=/etc/hadoop/conf
 #kylin.engine.spark.properties-file=
 
 # Estimate the RDD partition numbers
-kylin.engine.spark.rdd-partition-cut-mb=200
+kylin.engine.spark.rdd-partition-cut-mb=50
 
 # Minimal partition numbers of rdd
-kylin.engine.spark.min-partition=1
+kylin.engine.spark.min-partition=10
 
 # Max partition numbers of rdd
-kylin.engine.spark.max-partition=500
+kylin.engine.spark.max-partition=5000
 
 ### CUBE | DICTIONARY ###
 

[3/4] kylin git commit: KYLIN-1528 Create a branch for v1.5 with HBase 1.x API

[lidong]

KYLIN-1528 Create a branch for v1.5 with HBase 1.x API


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/081bec0b
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/081bec0b
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/081bec0b

Branch: refs/heads/yang22-hbase102
Commit: 081bec0b157911bc2d957596d43b876b0726
Parents: 2fec060
Author: shaofengshi 
Authored: Wed Mar 23 17:07:05 2016 +0800
Committer: lidongsjtu 
Committed: Mon Jan 9 17:07:05 2017 +0800

--
 examples/test_case_data/sandbox/hbase-site.xml  | 19 +---
 .../kylin/provision/BuildCubeWithEngine.java| 12 +--
 pom.xml | 12 +--
 .../kylin/rest/security/AclHBaseStorage.java|  4 +-
 .../rest/security/MockAclHBaseStorage.java  |  8 +-
 .../apache/kylin/rest/security/MockHTable.java  | 95 
 .../rest/security/RealAclHBaseStorage.java  |  9 +-
 .../apache/kylin/rest/service/AclService.java   | 25 +++---
 .../apache/kylin/rest/service/CubeService.java  | 35 +++-
 .../apache/kylin/rest/service/QueryService.java | 24 +++--
 .../apache/kylin/rest/service/UserService.java  | 17 ++--
 .../kylin/storage/hbase/HBaseConnection.java| 44 -
 .../kylin/storage/hbase/HBaseResourceStore.java | 31 +++
 .../kylin/storage/hbase/HBaseStorage.java   |  3 +-
 .../storage/hbase/cube/SimpleHBaseStore.java| 20 ++---
 .../hbase/cube/v1/CubeSegmentTupleIterator.java | 11 +--
 .../storage/hbase/cube/v1/CubeStorageQuery.java |  4 +-
 .../hbase/cube/v1/RegionScannerAdapter.java | 10 ++-
 .../cube/v1/SerializedHBaseTupleIterator.java   |  4 +-
 .../observer/AggregateRegionObserver.java   |  4 +-
 .../observer/AggregationScanner.java| 14 ++-
 .../observer/ObserverAggregationCache.java  | 10 ++-
 .../coprocessor/observer/ObserverEnabler.java   |  4 +-
 .../hbase/cube/v2/CubeHBaseEndpointRPC.java | 13 +--
 .../storage/hbase/cube/v2/CubeHBaseScanRPC.java |  9 +-
 .../coprocessor/endpoint/CubeVisitService.java  |  4 +-
 .../storage/hbase/steps/CubeHTableUtil.java | 16 ++--
 .../storage/hbase/steps/DeprecatedGCStep.java   | 26 +++---
 .../storage/hbase/steps/HBaseCuboidWriter.java  |  7 +-
 .../kylin/storage/hbase/steps/MergeGCStep.java  | 23 ++---
 .../storage/hbase/util/CleanHtableCLI.java  | 12 +--
 .../storage/hbase/util/CubeMigrationCLI.java| 37 
 .../hbase/util/CubeMigrationCheckCLI.java   | 17 ++--
 .../hbase/util/DeployCoprocessorCLI.java| 27 +++---
 .../hbase/util/ExtendCubeToHybridCLI.java   |  8 +-
 .../hbase/util/GridTableHBaseBenchmark.java | 34 +++
 .../kylin/storage/hbase/util/HBaseClean.java| 18 ++--
 .../hbase/util/HBaseRegionSizeCalculator.java   | 35 
 .../kylin/storage/hbase/util/HBaseUsage.java|  9 +-
 .../storage/hbase/util/HbaseStreamingInput.java | 30 +++
 .../hbase/util/HtableAlterMetadataCLI.java  |  9 +-
 .../storage/hbase/util/OrphanHBaseCleanJob.java | 19 ++--
 .../kylin/storage/hbase/util/PingHBaseCLI.java  | 15 ++--
 .../kylin/storage/hbase/util/RowCounterCLI.java | 11 +--
 .../storage/hbase/util/StorageCleanupJob.java   | 20 +++--
 .../storage/hbase/util/UpdateHTableHostCLI.java | 17 ++--
 .../observer/AggregateRegionObserverTest.java   | 26 ++
 .../v1/filter/TestFuzzyRowFilterV2EndToEnd.java |  5 +-
 .../org/apache/kylin/tool/CubeMigrationCLI.java | 19 ++--
 .../kylin/tool/ExtendCubeToHybridCLI.java   |  8 +-
 50 files changed, 418 insertions(+), 475 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/081bec0b/examples/test_case_data/sandbox/hbase-site.xml
--
diff --git a/examples/test_case_data/sandbox/hbase-site.xml 
b/examples/test_case_data/sandbox/hbase-site.xml
index 46d5345..734908e 100644
--- a/examples/test_case_data/sandbox/hbase-site.xml
+++ b/examples/test_case_data/sandbox/hbase-site.xml
@@ -190,22 +190,5 @@
 zookeeper.znode.parent
 /hbase-unsecure
 
-
-hbase.client.pause
-100
-General client pause value.  Used mostly as value to wait
-before running a retry of a failed get, region lookup, etc.
-See hbase.client.retries.number for description of how we backoff 
from
-this initial pause amount and how this pause works w/ 
retries.
-
-
-hbase.client.retries.number
-5
-Maximum retries.  Used as maximum for all retryable
-operations such as the getting of a cell's value, starting a row 
update,
-etc.  Retry interval is a rough function based on 
hbase.client.pause.  At
-first we retry at this interval but then with backoff, we pretty 
quickly reach
-

[1/4] kylin git commit: KYLIN-2362 Unify shell interpreter in scripts [Forced Update!]

[lidong]

Repository: kylin
Updated Branches:
  refs/heads/yang22-hbase102 04d172e32 -> 69e3ea42d (forced update)


KYLIN-2362 Unify shell interpreter in scripts

Signed-off-by: lidongsjtu 


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/2fec0603
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/2fec0603
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/2fec0603

Branch: refs/heads/yang22-hbase102
Commit: 2fec0603ed1fdfc25e5ceafec1636436ad0823e9
Parents: 3b70b1a
Author: etherge 
Authored: Fri Jan 6 14:18:47 2017 +0800
Committer: lidongsjtu 
Committed: Mon Jan 9 17:06:30 2017 +0800

--
 build/bin/check-env.sh| 2 +-
 build/bin/find-hive-dependency.sh | 4 ++--
 build/bin/kylin.sh| 2 +-
 build/bin/sample.sh   | 8 
 4 files changed, 8 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/2fec0603/build/bin/check-env.sh
--
diff --git a/build/bin/check-env.sh b/build/bin/check-env.sh
index 9cd8a64..a4003c9 100644
--- a/build/bin/check-env.sh
+++ b/build/bin/check-env.sh
@@ -41,7 +41,7 @@ then
 quit "Please make sure the user has the privilege to run hadoop shell"
 fi
 
-WORKING_DIR=`sh $KYLIN_HOME/bin/get-properties.sh kylin.env.hdfs-working-dir`
+WORKING_DIR=`bash $KYLIN_HOME/bin/get-properties.sh kylin.env.hdfs-working-dir`
 hadoop fs -mkdir -p $WORKING_DIR
 if [ $? != 0 ]
 then

http://git-wip-us.apache.org/repos/asf/kylin/blob/2fec0603/build/bin/find-hive-dependency.sh
--
diff --git a/build/bin/find-hive-dependency.sh 
b/build/bin/find-hive-dependency.sh
index fe9e0a3..453a35a 100644
--- a/build/bin/find-hive-dependency.sh
+++ b/build/bin/find-hive-dependency.sh
@@ -19,13 +19,13 @@
 
 source $(cd -P -- "$(dirname -- "$0")" && pwd -P)/header.sh
 
-client_mode=`sh ${KYLIN_HOME}/bin/get-properties.sh kylin.source.hive.client`
+client_mode=`bash ${KYLIN_HOME}/bin/get-properties.sh kylin.source.hive.client`
 hive_env=
 
 echo Retrieving hive dependency...
 if [ "${client_mode}" == "beeline" ]
 then
-beeline_params=`sh ${KYLIN_HOME}/bin/get-properties.sh 
kylin.source.hive.beeline-params`
+beeline_params=`bash ${KYLIN_HOME}/bin/get-properties.sh 
kylin.source.hive.beeline-params`
 hive_env=`beeline ${beeline_params} --outputformat=dsv -e set 2>&1 | grep 
'env:CLASSPATH' `
 else
 hive_env=`hive -e set 2>&1 | grep 'env:CLASSPATH'`

http://git-wip-us.apache.org/repos/asf/kylin/blob/2fec0603/build/bin/kylin.sh
--
diff --git a/build/bin/kylin.sh b/build/bin/kylin.sh
index fca1e5c..0cdbbc6 100644
--- a/build/bin/kylin.sh
+++ b/build/bin/kylin.sh
@@ -73,7 +73,7 @@ then
 #In this way we no longer need to explicitly configure hadoop/hbase 
related classpath for tomcat,
 #hbase command will do all the dirty tasks for us:
 
-spring_profile=`sh ${dir}/get-properties.sh kylin.security.profile`
+spring_profile=`bash ${dir}/get-properties.sh kylin.security.profile`
 if [ -z "$spring_profile" ]
 then
 quit 'please set kylin.security.profile in kylin.properties, options 
are: testing, ldap, saml.'

http://git-wip-us.apache.org/repos/asf/kylin/blob/2fec0603/build/bin/sample.sh
--
diff --git a/build/bin/sample.sh b/build/bin/sample.sh
index 1397034..1363a5d 100644
--- a/build/bin/sample.sh
+++ b/build/bin/sample.sh
@@ -29,10 +29,10 @@ echo "Loading sample data into HDFS tmp path: 
/tmp/kylin/sample_cube/data"
 hadoop fs -mkdir -p /tmp/kylin/sample_cube/data
 hadoop fs -put * /tmp/kylin/sample_cube/data/
 
-hive_client_mode=`sh ${KYLIN_HOME}/bin/get-properties.sh 
kylin.source.hive.client`
+hive_client_mode=`bash ${KYLIN_HOME}/bin/get-properties.sh 
kylin.source.hive.client`
 if [ "${hive_client_mode}" == "beeline" ]
 then
-beeline_params=`sh ${KYLIN_HOME}/bin/get-properties.sh 
kylin.source.hive.beeline-params`
+beeline_params=`bash ${KYLIN_HOME}/bin/get-properties.sh 
kylin.source.hive.beeline-params`
 beeline ${beeline_params} -f 
${KYLIN_HOME}/sample_cube/create_sample_tables.sql  || { exit 1; }
 else
 hive -f ${KYLIN_HOME}/sample_cube/create_sample_tables.sql  || { exit 1; }
@@ -42,8 +42,8 @@ echo "Sample hive tables are created successfully; Going to 
create sample cube..
 hadoop fs -rm -r /tmp/kylin/sample_cube
 
 # set engine type and storage type to cube desc
-default_engine_type=`sh ${KYLIN_HOME}/bin/get-properties.sh 
kylin.engine.default`
-default_storage_type=`sh ${KYLIN_HOME}/bin/get-properties.sh 
kylin.storage.default`
+default_engine_type=`bash 

[4/4] kylin git commit: KYLIN-2233 Support HBase 1.0.2

[lidong]

KYLIN-2233 Support HBase 1.0.2

Signed-off-by: Yang Li 


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/69e3ea42
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/69e3ea42
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/69e3ea42

Branch: refs/heads/yang22-hbase102
Commit: 69e3ea42d03300b931b0c2f9ee2fecf697db95f0
Parents: 081bec0
Author: Cheng Wang 
Authored: Wed Oct 12 14:39:24 2016 +0800
Committer: lidongsjtu 
Committed: Mon Jan 9 17:11:10 2017 +0800

--
 pom.xml   |  2 +-
 .../storage/hbase/cube/v1/RegionScannerAdapter.java   | 13 +++--
 .../coprocessor/observer/AggregateRegionObserver.java |  4 ++--
 .../v1/coprocessor/observer/AggregationScanner.java   | 14 --
 .../observer/ObserverAggregationCache.java| 10 ++
 .../observer/AggregateRegionObserverTest.java | 10 ++
 .../cube/v1/filter/TestFuzzyRowFilterV2EndToEnd.java  |  3 +--
 7 files changed, 15 insertions(+), 41 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/69e3ea42/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 6d3425e..ebc8c9d 100644
--- a/pom.xml
+++ b/pom.xml
@@ -54,7 +54,7 @@
 1.2.1
 
 
-1.1.1
+1.0.2
 0.10.0.0
 
 

http://git-wip-us.apache.org/repos/asf/kylin/blob/69e3ea42/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/cube/v1/RegionScannerAdapter.java
--
diff --git 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/cube/v1/RegionScannerAdapter.java
 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/cube/v1/RegionScannerAdapter.java
index 3d30767..c07d47f 100644
--- 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/cube/v1/RegionScannerAdapter.java
+++ 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/cube/v1/RegionScannerAdapter.java
@@ -26,7 +26,6 @@ import org.apache.hadoop.hbase.HRegionInfo;
 import org.apache.hadoop.hbase.client.Result;
 import org.apache.hadoop.hbase.client.ResultScanner;
 import org.apache.hadoop.hbase.regionserver.RegionScanner;
-import org.apache.hadoop.hbase.regionserver.ScannerContext;
 
 /**
  * @author yangli9
@@ -51,7 +50,7 @@ public class RegionScannerAdapter implements RegionScanner {
 }
 
 @Override
-public boolean next(List result, ScannerContext scannerContext) 
throws IOException {
+public boolean next(List result, int limit) throws IOException {
 return next(result);
 }
 
@@ -61,7 +60,7 @@ public class RegionScannerAdapter implements RegionScanner {
 }
 
 @Override
-public boolean nextRaw(List result, ScannerContext scannerContext) 
throws IOException {
+public boolean nextRaw(List result, int limit) throws IOException {
 return next(result);
 }
 
@@ -94,10 +93,4 @@ public class RegionScannerAdapter implements RegionScanner {
 public long getMvccReadPoint() {
 return Long.MAX_VALUE;
 }
-
-@Override
-public int getBatch() {
-return -1;
-}
-
-}
+}
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/kylin/blob/69e3ea42/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/cube/v1/coprocessor/observer/AggregateRegionObserver.java
--
diff --git 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/cube/v1/coprocessor/observer/AggregateRegionObserver.java
 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/cube/v1/coprocessor/observer/AggregateRegionObserver.java
index 7e25e4c..7139ca7 100644
--- 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/cube/v1/coprocessor/observer/AggregateRegionObserver.java
+++ 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/cube/v1/coprocessor/observer/AggregateRegionObserver.java
@@ -26,7 +26,7 @@ import org.apache.hadoop.hbase.client.Scan;
 import org.apache.hadoop.hbase.coprocessor.BaseRegionObserver;
 import org.apache.hadoop.hbase.coprocessor.ObserverContext;
 import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
-import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.regionserver.HRegion;
 import org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost;
 import org.apache.hadoop.hbase.regionserver.RegionScanner;
 import org.apache.kylin.gridtable.StorageSideBehavior;
@@ -99,7 +99,7 @@ public class AggregateRegionObserver extends 
BaseRegionObserver {
 // start/end region operation & sync on scanner is suggested by the
 // javadoc of 

[2/4] kylin git commit: KYLIN-1528 Create a branch for v1.5 with HBase 1.x API

[lidong]

http://git-wip-us.apache.org/repos/asf/kylin/blob/081bec0b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/MergeGCStep.java
--
diff --git 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/MergeGCStep.java
 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/MergeGCStep.java
index 5b2441c..2f7e164 100644
--- 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/MergeGCStep.java
+++ 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/MergeGCStep.java
@@ -24,11 +24,11 @@ import java.util.Collections;
 import java.util.List;
 
 import org.apache.commons.lang.StringUtils;
-import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.hbase.HTableDescriptor;
-import org.apache.hadoop.hbase.client.HBaseAdmin;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.Connection;
 import org.apache.kylin.common.KylinConfig;
-import org.apache.kylin.common.util.Bytes;
 import org.apache.kylin.job.exception.ExecuteException;
 import org.apache.kylin.job.execution.AbstractExecutable;
 import org.apache.kylin.job.execution.ExecutableContext;
@@ -69,19 +69,20 @@ public class MergeGCStep extends AbstractExecutable {
 List oldTables = getOldHTables();
 if (oldTables != null && oldTables.size() > 0) {
 String metadataUrlPrefix = 
KylinConfig.getInstanceFromEnv().getMetadataUrlPrefix();
-Configuration conf = 
HBaseConnection.getCurrentHBaseConfiguration();
-HBaseAdmin admin = null;
+Admin admin = null;
 try {
-admin = new HBaseAdmin(conf);
+Connection conn = 
HBaseConnection.get(KylinConfig.getInstanceFromEnv().getStorageUrl());
+admin = conn.getAdmin();
+
 for (String table : oldTables) {
-if (admin.tableExists(table)) {
-HTableDescriptor tableDescriptor = 
admin.getTableDescriptor(Bytes.toBytes(table));
+if (admin.tableExists(TableName.valueOf(table))) {
+HTableDescriptor tableDescriptor = 
admin.getTableDescriptor(TableName.valueOf((table)));
 String host = 
tableDescriptor.getValue(IRealizationConstants.HTableTag);
 if (metadataUrlPrefix.equalsIgnoreCase(host)) {
-if (admin.isTableEnabled(table)) {
-admin.disableTable(table);
+if 
(admin.isTableEnabled(TableName.valueOf(table))) {
+admin.disableTable(TableName.valueOf(table));
 }
-admin.deleteTable(table);
+admin.deleteTable(TableName.valueOf(table));
 logger.debug("Dropped htable: " + table);
 output.append("HBase table " + table + " is 
dropped. \n");
 } else {

http://git-wip-us.apache.org/repos/asf/kylin/blob/081bec0b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/util/CleanHtableCLI.java
--
diff --git 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/util/CleanHtableCLI.java
 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/util/CleanHtableCLI.java
index a150607..56f867a 100644
--- 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/util/CleanHtableCLI.java
+++ 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/util/CleanHtableCLI.java
@@ -21,9 +21,11 @@ package org.apache.kylin.storage.hbase.util;
 import java.io.IOException;
 
 import org.apache.commons.cli.Options;
-import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.hbase.HTableDescriptor;
-import org.apache.hadoop.hbase.client.HBaseAdmin;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.kylin.common.KylinConfig;
 import org.apache.kylin.common.util.AbstractApplication;
 import org.apache.kylin.common.util.OptionsHelper;
 import org.apache.kylin.metadata.realization.IRealizationConstants;
@@ -38,8 +40,8 @@ public class CleanHtableCLI extends AbstractApplication {
 protected static final Logger logger = 
LoggerFactory.getLogger(CleanHtableCLI.class);
 
 private void clean() throws IOException {
-Configuration conf = HBaseConnection.getCurrentHBaseConfiguration();
-HBaseAdmin hbaseAdmin = new HBaseAdmin(conf);
+Connection conn = 
HBaseConnection.get(KylinConfig.getInstanceFromEnv().getStorageUrl());
+Admin hbaseAdmin = conn.getAdmin();
 
 for (HTableDescriptor descriptor : hbaseAdmin.listTables()) {
 String name = 

[2/4] kylin git commit: KYLIN-1528 Create a branch for v1.5 with HBase 1.x API

[lidong]

http://git-wip-us.apache.org/repos/asf/kylin/blob/081bec0b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/MergeGCStep.java
--
diff --git 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/MergeGCStep.java
 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/MergeGCStep.java
index 5b2441c..2f7e164 100644
--- 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/MergeGCStep.java
+++ 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/MergeGCStep.java
@@ -24,11 +24,11 @@ import java.util.Collections;
 import java.util.List;
 
 import org.apache.commons.lang.StringUtils;
-import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.hbase.HTableDescriptor;
-import org.apache.hadoop.hbase.client.HBaseAdmin;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.Connection;
 import org.apache.kylin.common.KylinConfig;
-import org.apache.kylin.common.util.Bytes;
 import org.apache.kylin.job.exception.ExecuteException;
 import org.apache.kylin.job.execution.AbstractExecutable;
 import org.apache.kylin.job.execution.ExecutableContext;
@@ -69,19 +69,20 @@ public class MergeGCStep extends AbstractExecutable {
 List oldTables = getOldHTables();
 if (oldTables != null && oldTables.size() > 0) {
 String metadataUrlPrefix = 
KylinConfig.getInstanceFromEnv().getMetadataUrlPrefix();
-Configuration conf = 
HBaseConnection.getCurrentHBaseConfiguration();
-HBaseAdmin admin = null;
+Admin admin = null;
 try {
-admin = new HBaseAdmin(conf);
+Connection conn = 
HBaseConnection.get(KylinConfig.getInstanceFromEnv().getStorageUrl());
+admin = conn.getAdmin();
+
 for (String table : oldTables) {
-if (admin.tableExists(table)) {
-HTableDescriptor tableDescriptor = 
admin.getTableDescriptor(Bytes.toBytes(table));
+if (admin.tableExists(TableName.valueOf(table))) {
+HTableDescriptor tableDescriptor = 
admin.getTableDescriptor(TableName.valueOf((table)));
 String host = 
tableDescriptor.getValue(IRealizationConstants.HTableTag);
 if (metadataUrlPrefix.equalsIgnoreCase(host)) {
-if (admin.isTableEnabled(table)) {
-admin.disableTable(table);
+if 
(admin.isTableEnabled(TableName.valueOf(table))) {
+admin.disableTable(TableName.valueOf(table));
 }
-admin.deleteTable(table);
+admin.deleteTable(TableName.valueOf(table));
 logger.debug("Dropped htable: " + table);
 output.append("HBase table " + table + " is 
dropped. \n");
 } else {

http://git-wip-us.apache.org/repos/asf/kylin/blob/081bec0b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/util/CleanHtableCLI.java
--
diff --git 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/util/CleanHtableCLI.java
 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/util/CleanHtableCLI.java
index a150607..56f867a 100644
--- 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/util/CleanHtableCLI.java
+++ 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/util/CleanHtableCLI.java
@@ -21,9 +21,11 @@ package org.apache.kylin.storage.hbase.util;
 import java.io.IOException;
 
 import org.apache.commons.cli.Options;
-import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.hbase.HTableDescriptor;
-import org.apache.hadoop.hbase.client.HBaseAdmin;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.kylin.common.KylinConfig;
 import org.apache.kylin.common.util.AbstractApplication;
 import org.apache.kylin.common.util.OptionsHelper;
 import org.apache.kylin.metadata.realization.IRealizationConstants;
@@ -38,8 +40,8 @@ public class CleanHtableCLI extends AbstractApplication {
 protected static final Logger logger = 
LoggerFactory.getLogger(CleanHtableCLI.class);
 
 private void clean() throws IOException {
-Configuration conf = HBaseConnection.getCurrentHBaseConfiguration();
-HBaseAdmin hbaseAdmin = new HBaseAdmin(conf);
+Connection conn = 
HBaseConnection.get(KylinConfig.getInstanceFromEnv().getStorageUrl());
+Admin hbaseAdmin = conn.getAdmin();
 
 for (HTableDescriptor descriptor : hbaseAdmin.listTables()) {
 String name = 

[3/4] kylin git commit: KYLIN-1528 Create a branch for v1.5 with HBase 1.x API

[lidong]

KYLIN-1528 Create a branch for v1.5 with HBase 1.x API


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/081bec0b
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/081bec0b
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/081bec0b

Branch: refs/heads/yang22-cdh5.7
Commit: 081bec0b157911bc2d957596d43b876b0726
Parents: 2fec060
Author: shaofengshi 
Authored: Wed Mar 23 17:07:05 2016 +0800
Committer: lidongsjtu 
Committed: Mon Jan 9 17:07:05 2017 +0800

--
 examples/test_case_data/sandbox/hbase-site.xml  | 19 +---
 .../kylin/provision/BuildCubeWithEngine.java| 12 +--
 pom.xml | 12 +--
 .../kylin/rest/security/AclHBaseStorage.java|  4 +-
 .../rest/security/MockAclHBaseStorage.java  |  8 +-
 .../apache/kylin/rest/security/MockHTable.java  | 95 
 .../rest/security/RealAclHBaseStorage.java  |  9 +-
 .../apache/kylin/rest/service/AclService.java   | 25 +++---
 .../apache/kylin/rest/service/CubeService.java  | 35 +++-
 .../apache/kylin/rest/service/QueryService.java | 24 +++--
 .../apache/kylin/rest/service/UserService.java  | 17 ++--
 .../kylin/storage/hbase/HBaseConnection.java| 44 -
 .../kylin/storage/hbase/HBaseResourceStore.java | 31 +++
 .../kylin/storage/hbase/HBaseStorage.java   |  3 +-
 .../storage/hbase/cube/SimpleHBaseStore.java| 20 ++---
 .../hbase/cube/v1/CubeSegmentTupleIterator.java | 11 +--
 .../storage/hbase/cube/v1/CubeStorageQuery.java |  4 +-
 .../hbase/cube/v1/RegionScannerAdapter.java | 10 ++-
 .../cube/v1/SerializedHBaseTupleIterator.java   |  4 +-
 .../observer/AggregateRegionObserver.java   |  4 +-
 .../observer/AggregationScanner.java| 14 ++-
 .../observer/ObserverAggregationCache.java  | 10 ++-
 .../coprocessor/observer/ObserverEnabler.java   |  4 +-
 .../hbase/cube/v2/CubeHBaseEndpointRPC.java | 13 +--
 .../storage/hbase/cube/v2/CubeHBaseScanRPC.java |  9 +-
 .../coprocessor/endpoint/CubeVisitService.java  |  4 +-
 .../storage/hbase/steps/CubeHTableUtil.java | 16 ++--
 .../storage/hbase/steps/DeprecatedGCStep.java   | 26 +++---
 .../storage/hbase/steps/HBaseCuboidWriter.java  |  7 +-
 .../kylin/storage/hbase/steps/MergeGCStep.java  | 23 ++---
 .../storage/hbase/util/CleanHtableCLI.java  | 12 +--
 .../storage/hbase/util/CubeMigrationCLI.java| 37 
 .../hbase/util/CubeMigrationCheckCLI.java   | 17 ++--
 .../hbase/util/DeployCoprocessorCLI.java| 27 +++---
 .../hbase/util/ExtendCubeToHybridCLI.java   |  8 +-
 .../hbase/util/GridTableHBaseBenchmark.java | 34 +++
 .../kylin/storage/hbase/util/HBaseClean.java| 18 ++--
 .../hbase/util/HBaseRegionSizeCalculator.java   | 35 
 .../kylin/storage/hbase/util/HBaseUsage.java|  9 +-
 .../storage/hbase/util/HbaseStreamingInput.java | 30 +++
 .../hbase/util/HtableAlterMetadataCLI.java  |  9 +-
 .../storage/hbase/util/OrphanHBaseCleanJob.java | 19 ++--
 .../kylin/storage/hbase/util/PingHBaseCLI.java  | 15 ++--
 .../kylin/storage/hbase/util/RowCounterCLI.java | 11 +--
 .../storage/hbase/util/StorageCleanupJob.java   | 20 +++--
 .../storage/hbase/util/UpdateHTableHostCLI.java | 17 ++--
 .../observer/AggregateRegionObserverTest.java   | 26 ++
 .../v1/filter/TestFuzzyRowFilterV2EndToEnd.java |  5 +-
 .../org/apache/kylin/tool/CubeMigrationCLI.java | 19 ++--
 .../kylin/tool/ExtendCubeToHybridCLI.java   |  8 +-
 50 files changed, 418 insertions(+), 475 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/081bec0b/examples/test_case_data/sandbox/hbase-site.xml
--
diff --git a/examples/test_case_data/sandbox/hbase-site.xml 
b/examples/test_case_data/sandbox/hbase-site.xml
index 46d5345..734908e 100644
--- a/examples/test_case_data/sandbox/hbase-site.xml
+++ b/examples/test_case_data/sandbox/hbase-site.xml
@@ -190,22 +190,5 @@
 zookeeper.znode.parent
 /hbase-unsecure
 
-
-hbase.client.pause
-100
-General client pause value.  Used mostly as value to wait
-before running a retry of a failed get, region lookup, etc.
-See hbase.client.retries.number for description of how we backoff 
from
-this initial pause amount and how this pause works w/ 
retries.
-
-
-hbase.client.retries.number
-5
-Maximum retries.  Used as maximum for all retryable
-operations such as the getting of a cell's value, starting a row 
update,
-etc.  Retry interval is a rough function based on 
hbase.client.pause.  At
-first we retry at this interval but then with backoff, we pretty 
quickly reach
-  

[1/4] kylin git commit: KYLIN-2362 Unify shell interpreter in scripts [Forced Update!]

[lidong]

Repository: kylin
Updated Branches:
  refs/heads/yang22-cdh5.7 07a70eceb -> 5ff6be9a1 (forced update)


KYLIN-2362 Unify shell interpreter in scripts

Signed-off-by: lidongsjtu 


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/2fec0603
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/2fec0603
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/2fec0603

Branch: refs/heads/yang22-cdh5.7
Commit: 2fec0603ed1fdfc25e5ceafec1636436ad0823e9
Parents: 3b70b1a
Author: etherge 
Authored: Fri Jan 6 14:18:47 2017 +0800
Committer: lidongsjtu 
Committed: Mon Jan 9 17:06:30 2017 +0800

--
 build/bin/check-env.sh| 2 +-
 build/bin/find-hive-dependency.sh | 4 ++--
 build/bin/kylin.sh| 2 +-
 build/bin/sample.sh   | 8 
 4 files changed, 8 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/2fec0603/build/bin/check-env.sh
--
diff --git a/build/bin/check-env.sh b/build/bin/check-env.sh
index 9cd8a64..a4003c9 100644
--- a/build/bin/check-env.sh
+++ b/build/bin/check-env.sh
@@ -41,7 +41,7 @@ then
 quit "Please make sure the user has the privilege to run hadoop shell"
 fi
 
-WORKING_DIR=`sh $KYLIN_HOME/bin/get-properties.sh kylin.env.hdfs-working-dir`
+WORKING_DIR=`bash $KYLIN_HOME/bin/get-properties.sh kylin.env.hdfs-working-dir`
 hadoop fs -mkdir -p $WORKING_DIR
 if [ $? != 0 ]
 then

http://git-wip-us.apache.org/repos/asf/kylin/blob/2fec0603/build/bin/find-hive-dependency.sh
--
diff --git a/build/bin/find-hive-dependency.sh 
b/build/bin/find-hive-dependency.sh
index fe9e0a3..453a35a 100644
--- a/build/bin/find-hive-dependency.sh
+++ b/build/bin/find-hive-dependency.sh
@@ -19,13 +19,13 @@
 
 source $(cd -P -- "$(dirname -- "$0")" && pwd -P)/header.sh
 
-client_mode=`sh ${KYLIN_HOME}/bin/get-properties.sh kylin.source.hive.client`
+client_mode=`bash ${KYLIN_HOME}/bin/get-properties.sh kylin.source.hive.client`
 hive_env=
 
 echo Retrieving hive dependency...
 if [ "${client_mode}" == "beeline" ]
 then
-beeline_params=`sh ${KYLIN_HOME}/bin/get-properties.sh 
kylin.source.hive.beeline-params`
+beeline_params=`bash ${KYLIN_HOME}/bin/get-properties.sh 
kylin.source.hive.beeline-params`
 hive_env=`beeline ${beeline_params} --outputformat=dsv -e set 2>&1 | grep 
'env:CLASSPATH' `
 else
 hive_env=`hive -e set 2>&1 | grep 'env:CLASSPATH'`

http://git-wip-us.apache.org/repos/asf/kylin/blob/2fec0603/build/bin/kylin.sh
--
diff --git a/build/bin/kylin.sh b/build/bin/kylin.sh
index fca1e5c..0cdbbc6 100644
--- a/build/bin/kylin.sh
+++ b/build/bin/kylin.sh
@@ -73,7 +73,7 @@ then
 #In this way we no longer need to explicitly configure hadoop/hbase 
related classpath for tomcat,
 #hbase command will do all the dirty tasks for us:
 
-spring_profile=`sh ${dir}/get-properties.sh kylin.security.profile`
+spring_profile=`bash ${dir}/get-properties.sh kylin.security.profile`
 if [ -z "$spring_profile" ]
 then
 quit 'please set kylin.security.profile in kylin.properties, options 
are: testing, ldap, saml.'

http://git-wip-us.apache.org/repos/asf/kylin/blob/2fec0603/build/bin/sample.sh
--
diff --git a/build/bin/sample.sh b/build/bin/sample.sh
index 1397034..1363a5d 100644
--- a/build/bin/sample.sh
+++ b/build/bin/sample.sh
@@ -29,10 +29,10 @@ echo "Loading sample data into HDFS tmp path: 
/tmp/kylin/sample_cube/data"
 hadoop fs -mkdir -p /tmp/kylin/sample_cube/data
 hadoop fs -put * /tmp/kylin/sample_cube/data/
 
-hive_client_mode=`sh ${KYLIN_HOME}/bin/get-properties.sh 
kylin.source.hive.client`
+hive_client_mode=`bash ${KYLIN_HOME}/bin/get-properties.sh 
kylin.source.hive.client`
 if [ "${hive_client_mode}" == "beeline" ]
 then
-beeline_params=`sh ${KYLIN_HOME}/bin/get-properties.sh 
kylin.source.hive.beeline-params`
+beeline_params=`bash ${KYLIN_HOME}/bin/get-properties.sh 
kylin.source.hive.beeline-params`
 beeline ${beeline_params} -f 
${KYLIN_HOME}/sample_cube/create_sample_tables.sql  || { exit 1; }
 else
 hive -f ${KYLIN_HOME}/sample_cube/create_sample_tables.sql  || { exit 1; }
@@ -42,8 +42,8 @@ echo "Sample hive tables are created successfully; Going to 
create sample cube..
 hadoop fs -rm -r /tmp/kylin/sample_cube
 
 # set engine type and storage type to cube desc
-default_engine_type=`sh ${KYLIN_HOME}/bin/get-properties.sh 
kylin.engine.default`
-default_storage_type=`sh ${KYLIN_HOME}/bin/get-properties.sh 
kylin.storage.default`
+default_engine_type=`bash 

[4/4] kylin git commit: KYLIN-1672 support kylin on cdh 5.7

[lidong]

KYLIN-1672 support kylin on cdh 5.7

Signed-off-by: Li Yang 


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/5ff6be9a
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/5ff6be9a
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/5ff6be9a

Branch: refs/heads/yang22-cdh5.7
Commit: 5ff6be9a1d73a1f57c2e8049e46793fe50ad9eb0
Parents: 081bec0
Author: Lynne Jiang 
Authored: Mon May 16 03:33:27 2016 -0700
Committer: lidongsjtu 
Committed: Mon Jan 9 17:08:53 2017 +0800

--
 dev-support/test_all_against_hdp_2_2_4_2_2.sh   |   0
 .../kylin/engine/mr/steps/MockupMapContext.java |  15 +-
 examples/test_case_data/sandbox/core-site.xml   | 146 +++---
 examples/test_case_data/sandbox/hbase-site.xml  | 162 ++
 examples/test_case_data/sandbox/hdfs-site.xml   | 259 ++
 examples/test_case_data/sandbox/mapred-site.xml | 398 ++-
 examples/test_case_data/sandbox/yarn-site.xml   | 496 ++-
 pom.xml |  16 +-
 server/pom.xml  |  36 ++
 .../storage/hbase/steps/MockupMapContext.java   |  19 +-
 tool/pom.xml|  12 +
 11 files changed, 428 insertions(+), 1131 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/5ff6be9a/dev-support/test_all_against_hdp_2_2_4_2_2.sh
--
diff --git a/dev-support/test_all_against_hdp_2_2_4_2_2.sh 
b/dev-support/test_all_against_hdp_2_2_4_2_2.sh
old mode 100644
new mode 100755

http://git-wip-us.apache.org/repos/asf/kylin/blob/5ff6be9a/engine-mr/src/test/java/org/apache/kylin/engine/mr/steps/MockupMapContext.java
--
diff --git 
a/engine-mr/src/test/java/org/apache/kylin/engine/mr/steps/MockupMapContext.java
 
b/engine-mr/src/test/java/org/apache/kylin/engine/mr/steps/MockupMapContext.java
index 847071d..9900465 100644
--- 
a/engine-mr/src/test/java/org/apache/kylin/engine/mr/steps/MockupMapContext.java
+++ 
b/engine-mr/src/test/java/org/apache/kylin/engine/mr/steps/MockupMapContext.java
@@ -77,6 +77,7 @@ public class MockupMapContext {
 outKV[0] = key;
 outKV[1] = value;
 }
+
 }
 
 @Override
@@ -99,6 +100,7 @@ public class MockupMapContext {
 throw new NotImplementedException();
 }
 
+
 @Override
 public float getProgress() {
 throw new NotImplementedException();
@@ -195,17 +197,17 @@ public class MockupMapContext {
 }
 
 @Override
-public RawComparator getSortComparator() {
+public boolean userClassesTakesPrecedence() {
 throw new NotImplementedException();
 }
 
 @Override
-public String getJar() {
+public RawComparator getSortComparator() {
 throw new NotImplementedException();
 }
 
 @Override
-public RawComparator getGroupingComparator() {
+public String getJar() {
 throw new NotImplementedException();
 }
 
@@ -221,7 +223,7 @@ public class MockupMapContext {
 
 @Override
 public boolean getProfileEnabled() {
-throw new NotImplementedException();
+return false;
 }
 
 @Override
@@ -308,6 +310,11 @@ public class MockupMapContext {
 public RawComparator getCombinerKeyGroupingComparator() {
 throw new NotImplementedException();
 }
+
+@Override
+public RawComparator getGroupingComparator() {
+return null;
+}
 });
 }
 }

http://git-wip-us.apache.org/repos/asf/kylin/blob/5ff6be9a/examples/test_case_data/sandbox/core-site.xml
--
diff --git a/examples/test_case_data/sandbox/core-site.xml 
b/examples/test_case_data/sandbox/core-site.xml
index 9aa588c..6162406 100644
--- a/examples/test_case_data/sandbox/core-site.xml
+++ b/examples/test_case_data/sandbox/core-site.xml
@@ -14,152 +14,146 @@
   See the License for the specific language governing permissions and
   limitations under the License.
 -->
+
 
-
 
 fs.defaultFS
-hdfs://sandbox.hortonworks.com:8020
-true
+hdfs://quickstart.cloudera:8020
 
-
 
 fs.trash.interval
-360
+1
 
-
 
-
ha.failover-controller.active-standby-elector.zk.op.retries
-120
+io.compression.codecs
+

[3/3] kylin git commit: KYLIN-1528 Create a branch for v1.5 with HBase 1.x API

[lidong]

KYLIN-1528 Create a branch for v1.5 with HBase 1.x API


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/081bec0b
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/081bec0b
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/081bec0b

Branch: refs/heads/yang22-hbase1.x
Commit: 081bec0b157911bc2d957596d43b876b0726
Parents: 2fec060
Author: shaofengshi 
Authored: Wed Mar 23 17:07:05 2016 +0800
Committer: lidongsjtu 
Committed: Mon Jan 9 17:07:05 2017 +0800

--
 examples/test_case_data/sandbox/hbase-site.xml  | 19 +---
 .../kylin/provision/BuildCubeWithEngine.java| 12 +--
 pom.xml | 12 +--
 .../kylin/rest/security/AclHBaseStorage.java|  4 +-
 .../rest/security/MockAclHBaseStorage.java  |  8 +-
 .../apache/kylin/rest/security/MockHTable.java  | 95 
 .../rest/security/RealAclHBaseStorage.java  |  9 +-
 .../apache/kylin/rest/service/AclService.java   | 25 +++---
 .../apache/kylin/rest/service/CubeService.java  | 35 +++-
 .../apache/kylin/rest/service/QueryService.java | 24 +++--
 .../apache/kylin/rest/service/UserService.java  | 17 ++--
 .../kylin/storage/hbase/HBaseConnection.java| 44 -
 .../kylin/storage/hbase/HBaseResourceStore.java | 31 +++
 .../kylin/storage/hbase/HBaseStorage.java   |  3 +-
 .../storage/hbase/cube/SimpleHBaseStore.java| 20 ++---
 .../hbase/cube/v1/CubeSegmentTupleIterator.java | 11 +--
 .../storage/hbase/cube/v1/CubeStorageQuery.java |  4 +-
 .../hbase/cube/v1/RegionScannerAdapter.java | 10 ++-
 .../cube/v1/SerializedHBaseTupleIterator.java   |  4 +-
 .../observer/AggregateRegionObserver.java   |  4 +-
 .../observer/AggregationScanner.java| 14 ++-
 .../observer/ObserverAggregationCache.java  | 10 ++-
 .../coprocessor/observer/ObserverEnabler.java   |  4 +-
 .../hbase/cube/v2/CubeHBaseEndpointRPC.java | 13 +--
 .../storage/hbase/cube/v2/CubeHBaseScanRPC.java |  9 +-
 .../coprocessor/endpoint/CubeVisitService.java  |  4 +-
 .../storage/hbase/steps/CubeHTableUtil.java | 16 ++--
 .../storage/hbase/steps/DeprecatedGCStep.java   | 26 +++---
 .../storage/hbase/steps/HBaseCuboidWriter.java  |  7 +-
 .../kylin/storage/hbase/steps/MergeGCStep.java  | 23 ++---
 .../storage/hbase/util/CleanHtableCLI.java  | 12 +--
 .../storage/hbase/util/CubeMigrationCLI.java| 37 
 .../hbase/util/CubeMigrationCheckCLI.java   | 17 ++--
 .../hbase/util/DeployCoprocessorCLI.java| 27 +++---
 .../hbase/util/ExtendCubeToHybridCLI.java   |  8 +-
 .../hbase/util/GridTableHBaseBenchmark.java | 34 +++
 .../kylin/storage/hbase/util/HBaseClean.java| 18 ++--
 .../hbase/util/HBaseRegionSizeCalculator.java   | 35 
 .../kylin/storage/hbase/util/HBaseUsage.java|  9 +-
 .../storage/hbase/util/HbaseStreamingInput.java | 30 +++
 .../hbase/util/HtableAlterMetadataCLI.java  |  9 +-
 .../storage/hbase/util/OrphanHBaseCleanJob.java | 19 ++--
 .../kylin/storage/hbase/util/PingHBaseCLI.java  | 15 ++--
 .../kylin/storage/hbase/util/RowCounterCLI.java | 11 +--
 .../storage/hbase/util/StorageCleanupJob.java   | 20 +++--
 .../storage/hbase/util/UpdateHTableHostCLI.java | 17 ++--
 .../observer/AggregateRegionObserverTest.java   | 26 ++
 .../v1/filter/TestFuzzyRowFilterV2EndToEnd.java |  5 +-
 .../org/apache/kylin/tool/CubeMigrationCLI.java | 19 ++--
 .../kylin/tool/ExtendCubeToHybridCLI.java   |  8 +-
 50 files changed, 418 insertions(+), 475 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/081bec0b/examples/test_case_data/sandbox/hbase-site.xml
--
diff --git a/examples/test_case_data/sandbox/hbase-site.xml 
b/examples/test_case_data/sandbox/hbase-site.xml
index 46d5345..734908e 100644
--- a/examples/test_case_data/sandbox/hbase-site.xml
+++ b/examples/test_case_data/sandbox/hbase-site.xml
@@ -190,22 +190,5 @@
 zookeeper.znode.parent
 /hbase-unsecure
 
-
-hbase.client.pause
-100
-General client pause value.  Used mostly as value to wait
-before running a retry of a failed get, region lookup, etc.
-See hbase.client.retries.number for description of how we backoff 
from
-this initial pause amount and how this pause works w/ 
retries.
-
-
-hbase.client.retries.number
-5
-Maximum retries.  Used as maximum for all retryable
-operations such as the getting of a cell's value, starting a row 
update,
-etc.  Retry interval is a rough function based on 
hbase.client.pause.  At
-first we retry at this interval but then with backoff, we pretty 
quickly reach
-

[1/3] kylin git commit: KYLIN-2362 Unify shell interpreter in scripts [Forced Update!]

[lidong]

Repository: kylin
Updated Branches:
  refs/heads/yang22-hbase1.x ebe4b5bb0 -> 081bec0b1 (forced update)


KYLIN-2362 Unify shell interpreter in scripts

Signed-off-by: lidongsjtu 


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/2fec0603
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/2fec0603
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/2fec0603

Branch: refs/heads/yang22-hbase1.x
Commit: 2fec0603ed1fdfc25e5ceafec1636436ad0823e9
Parents: 3b70b1a
Author: etherge 
Authored: Fri Jan 6 14:18:47 2017 +0800
Committer: lidongsjtu 
Committed: Mon Jan 9 17:06:30 2017 +0800

--
 build/bin/check-env.sh| 2 +-
 build/bin/find-hive-dependency.sh | 4 ++--
 build/bin/kylin.sh| 2 +-
 build/bin/sample.sh   | 8 
 4 files changed, 8 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/2fec0603/build/bin/check-env.sh
--
diff --git a/build/bin/check-env.sh b/build/bin/check-env.sh
index 9cd8a64..a4003c9 100644
--- a/build/bin/check-env.sh
+++ b/build/bin/check-env.sh
@@ -41,7 +41,7 @@ then
 quit "Please make sure the user has the privilege to run hadoop shell"
 fi
 
-WORKING_DIR=`sh $KYLIN_HOME/bin/get-properties.sh kylin.env.hdfs-working-dir`
+WORKING_DIR=`bash $KYLIN_HOME/bin/get-properties.sh kylin.env.hdfs-working-dir`
 hadoop fs -mkdir -p $WORKING_DIR
 if [ $? != 0 ]
 then

http://git-wip-us.apache.org/repos/asf/kylin/blob/2fec0603/build/bin/find-hive-dependency.sh
--
diff --git a/build/bin/find-hive-dependency.sh 
b/build/bin/find-hive-dependency.sh
index fe9e0a3..453a35a 100644
--- a/build/bin/find-hive-dependency.sh
+++ b/build/bin/find-hive-dependency.sh
@@ -19,13 +19,13 @@
 
 source $(cd -P -- "$(dirname -- "$0")" && pwd -P)/header.sh
 
-client_mode=`sh ${KYLIN_HOME}/bin/get-properties.sh kylin.source.hive.client`
+client_mode=`bash ${KYLIN_HOME}/bin/get-properties.sh kylin.source.hive.client`
 hive_env=
 
 echo Retrieving hive dependency...
 if [ "${client_mode}" == "beeline" ]
 then
-beeline_params=`sh ${KYLIN_HOME}/bin/get-properties.sh 
kylin.source.hive.beeline-params`
+beeline_params=`bash ${KYLIN_HOME}/bin/get-properties.sh 
kylin.source.hive.beeline-params`
 hive_env=`beeline ${beeline_params} --outputformat=dsv -e set 2>&1 | grep 
'env:CLASSPATH' `
 else
 hive_env=`hive -e set 2>&1 | grep 'env:CLASSPATH'`

http://git-wip-us.apache.org/repos/asf/kylin/blob/2fec0603/build/bin/kylin.sh
--
diff --git a/build/bin/kylin.sh b/build/bin/kylin.sh
index fca1e5c..0cdbbc6 100644
--- a/build/bin/kylin.sh
+++ b/build/bin/kylin.sh
@@ -73,7 +73,7 @@ then
 #In this way we no longer need to explicitly configure hadoop/hbase 
related classpath for tomcat,
 #hbase command will do all the dirty tasks for us:
 
-spring_profile=`sh ${dir}/get-properties.sh kylin.security.profile`
+spring_profile=`bash ${dir}/get-properties.sh kylin.security.profile`
 if [ -z "$spring_profile" ]
 then
 quit 'please set kylin.security.profile in kylin.properties, options 
are: testing, ldap, saml.'

http://git-wip-us.apache.org/repos/asf/kylin/blob/2fec0603/build/bin/sample.sh
--
diff --git a/build/bin/sample.sh b/build/bin/sample.sh
index 1397034..1363a5d 100644
--- a/build/bin/sample.sh
+++ b/build/bin/sample.sh
@@ -29,10 +29,10 @@ echo "Loading sample data into HDFS tmp path: 
/tmp/kylin/sample_cube/data"
 hadoop fs -mkdir -p /tmp/kylin/sample_cube/data
 hadoop fs -put * /tmp/kylin/sample_cube/data/
 
-hive_client_mode=`sh ${KYLIN_HOME}/bin/get-properties.sh 
kylin.source.hive.client`
+hive_client_mode=`bash ${KYLIN_HOME}/bin/get-properties.sh 
kylin.source.hive.client`
 if [ "${hive_client_mode}" == "beeline" ]
 then
-beeline_params=`sh ${KYLIN_HOME}/bin/get-properties.sh 
kylin.source.hive.beeline-params`
+beeline_params=`bash ${KYLIN_HOME}/bin/get-properties.sh 
kylin.source.hive.beeline-params`
 beeline ${beeline_params} -f 
${KYLIN_HOME}/sample_cube/create_sample_tables.sql  || { exit 1; }
 else
 hive -f ${KYLIN_HOME}/sample_cube/create_sample_tables.sql  || { exit 1; }
@@ -42,8 +42,8 @@ echo "Sample hive tables are created successfully; Going to 
create sample cube..
 hadoop fs -rm -r /tmp/kylin/sample_cube
 
 # set engine type and storage type to cube desc
-default_engine_type=`sh ${KYLIN_HOME}/bin/get-properties.sh 
kylin.engine.default`
-default_storage_type=`sh ${KYLIN_HOME}/bin/get-properties.sh 
kylin.storage.default`
+default_engine_type=`bash 

[2/3] kylin git commit: KYLIN-1528 Create a branch for v1.5 with HBase 1.x API

[lidong]

http://git-wip-us.apache.org/repos/asf/kylin/blob/081bec0b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/MergeGCStep.java
--
diff --git 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/MergeGCStep.java
 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/MergeGCStep.java
index 5b2441c..2f7e164 100644
--- 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/MergeGCStep.java
+++ 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/MergeGCStep.java
@@ -24,11 +24,11 @@ import java.util.Collections;
 import java.util.List;
 
 import org.apache.commons.lang.StringUtils;
-import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.hbase.HTableDescriptor;
-import org.apache.hadoop.hbase.client.HBaseAdmin;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.Connection;
 import org.apache.kylin.common.KylinConfig;
-import org.apache.kylin.common.util.Bytes;
 import org.apache.kylin.job.exception.ExecuteException;
 import org.apache.kylin.job.execution.AbstractExecutable;
 import org.apache.kylin.job.execution.ExecutableContext;
@@ -69,19 +69,20 @@ public class MergeGCStep extends AbstractExecutable {
 List oldTables = getOldHTables();
 if (oldTables != null && oldTables.size() > 0) {
 String metadataUrlPrefix = 
KylinConfig.getInstanceFromEnv().getMetadataUrlPrefix();
-Configuration conf = 
HBaseConnection.getCurrentHBaseConfiguration();
-HBaseAdmin admin = null;
+Admin admin = null;
 try {
-admin = new HBaseAdmin(conf);
+Connection conn = 
HBaseConnection.get(KylinConfig.getInstanceFromEnv().getStorageUrl());
+admin = conn.getAdmin();
+
 for (String table : oldTables) {
-if (admin.tableExists(table)) {
-HTableDescriptor tableDescriptor = 
admin.getTableDescriptor(Bytes.toBytes(table));
+if (admin.tableExists(TableName.valueOf(table))) {
+HTableDescriptor tableDescriptor = 
admin.getTableDescriptor(TableName.valueOf((table)));
 String host = 
tableDescriptor.getValue(IRealizationConstants.HTableTag);
 if (metadataUrlPrefix.equalsIgnoreCase(host)) {
-if (admin.isTableEnabled(table)) {
-admin.disableTable(table);
+if 
(admin.isTableEnabled(TableName.valueOf(table))) {
+admin.disableTable(TableName.valueOf(table));
 }
-admin.deleteTable(table);
+admin.deleteTable(TableName.valueOf(table));
 logger.debug("Dropped htable: " + table);
 output.append("HBase table " + table + " is 
dropped. \n");
 } else {

http://git-wip-us.apache.org/repos/asf/kylin/blob/081bec0b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/util/CleanHtableCLI.java
--
diff --git 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/util/CleanHtableCLI.java
 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/util/CleanHtableCLI.java
index a150607..56f867a 100644
--- 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/util/CleanHtableCLI.java
+++ 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/util/CleanHtableCLI.java
@@ -21,9 +21,11 @@ package org.apache.kylin.storage.hbase.util;
 import java.io.IOException;
 
 import org.apache.commons.cli.Options;
-import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.hbase.HTableDescriptor;
-import org.apache.hadoop.hbase.client.HBaseAdmin;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.kylin.common.KylinConfig;
 import org.apache.kylin.common.util.AbstractApplication;
 import org.apache.kylin.common.util.OptionsHelper;
 import org.apache.kylin.metadata.realization.IRealizationConstants;
@@ -38,8 +40,8 @@ public class CleanHtableCLI extends AbstractApplication {
 protected static final Logger logger = 
LoggerFactory.getLogger(CleanHtableCLI.class);
 
 private void clean() throws IOException {
-Configuration conf = HBaseConnection.getCurrentHBaseConfiguration();
-HBaseAdmin hbaseAdmin = new HBaseAdmin(conf);
+Connection conn = 
HBaseConnection.get(KylinConfig.getInstanceFromEnv().getStorageUrl());
+Admin hbaseAdmin = conn.getAdmin();
 
 for (HTableDescriptor descriptor : hbaseAdmin.listTables()) {
 String name = 

kylin git commit: KYLIN-2362 Unify shell interpreter in scripts

[lidong]

Repository: kylin
Updated Branches:
  refs/heads/yang22 3b70b1a8b -> 2fec0603e


KYLIN-2362 Unify shell interpreter in scripts

Signed-off-by: lidongsjtu 


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/2fec0603
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/2fec0603
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/2fec0603

Branch: refs/heads/yang22
Commit: 2fec0603ed1fdfc25e5ceafec1636436ad0823e9
Parents: 3b70b1a
Author: etherge 
Authored: Fri Jan 6 14:18:47 2017 +0800
Committer: lidongsjtu 
Committed: Mon Jan 9 17:06:30 2017 +0800

--
 build/bin/check-env.sh| 2 +-
 build/bin/find-hive-dependency.sh | 4 ++--
 build/bin/kylin.sh| 2 +-
 build/bin/sample.sh   | 8 
 4 files changed, 8 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/2fec0603/build/bin/check-env.sh
--
diff --git a/build/bin/check-env.sh b/build/bin/check-env.sh
index 9cd8a64..a4003c9 100644
--- a/build/bin/check-env.sh
+++ b/build/bin/check-env.sh
@@ -41,7 +41,7 @@ then
 quit "Please make sure the user has the privilege to run hadoop shell"
 fi
 
-WORKING_DIR=`sh $KYLIN_HOME/bin/get-properties.sh kylin.env.hdfs-working-dir`
+WORKING_DIR=`bash $KYLIN_HOME/bin/get-properties.sh kylin.env.hdfs-working-dir`
 hadoop fs -mkdir -p $WORKING_DIR
 if [ $? != 0 ]
 then

http://git-wip-us.apache.org/repos/asf/kylin/blob/2fec0603/build/bin/find-hive-dependency.sh
--
diff --git a/build/bin/find-hive-dependency.sh 
b/build/bin/find-hive-dependency.sh
index fe9e0a3..453a35a 100644
--- a/build/bin/find-hive-dependency.sh
+++ b/build/bin/find-hive-dependency.sh
@@ -19,13 +19,13 @@
 
 source $(cd -P -- "$(dirname -- "$0")" && pwd -P)/header.sh
 
-client_mode=`sh ${KYLIN_HOME}/bin/get-properties.sh kylin.source.hive.client`
+client_mode=`bash ${KYLIN_HOME}/bin/get-properties.sh kylin.source.hive.client`
 hive_env=
 
 echo Retrieving hive dependency...
 if [ "${client_mode}" == "beeline" ]
 then
-beeline_params=`sh ${KYLIN_HOME}/bin/get-properties.sh 
kylin.source.hive.beeline-params`
+beeline_params=`bash ${KYLIN_HOME}/bin/get-properties.sh 
kylin.source.hive.beeline-params`
 hive_env=`beeline ${beeline_params} --outputformat=dsv -e set 2>&1 | grep 
'env:CLASSPATH' `
 else
 hive_env=`hive -e set 2>&1 | grep 'env:CLASSPATH'`

http://git-wip-us.apache.org/repos/asf/kylin/blob/2fec0603/build/bin/kylin.sh
--
diff --git a/build/bin/kylin.sh b/build/bin/kylin.sh
index fca1e5c..0cdbbc6 100644
--- a/build/bin/kylin.sh
+++ b/build/bin/kylin.sh
@@ -73,7 +73,7 @@ then
 #In this way we no longer need to explicitly configure hadoop/hbase 
related classpath for tomcat,
 #hbase command will do all the dirty tasks for us:
 
-spring_profile=`sh ${dir}/get-properties.sh kylin.security.profile`
+spring_profile=`bash ${dir}/get-properties.sh kylin.security.profile`
 if [ -z "$spring_profile" ]
 then
 quit 'please set kylin.security.profile in kylin.properties, options 
are: testing, ldap, saml.'

http://git-wip-us.apache.org/repos/asf/kylin/blob/2fec0603/build/bin/sample.sh
--
diff --git a/build/bin/sample.sh b/build/bin/sample.sh
index 1397034..1363a5d 100644
--- a/build/bin/sample.sh
+++ b/build/bin/sample.sh
@@ -29,10 +29,10 @@ echo "Loading sample data into HDFS tmp path: 
/tmp/kylin/sample_cube/data"
 hadoop fs -mkdir -p /tmp/kylin/sample_cube/data
 hadoop fs -put * /tmp/kylin/sample_cube/data/
 
-hive_client_mode=`sh ${KYLIN_HOME}/bin/get-properties.sh 
kylin.source.hive.client`
+hive_client_mode=`bash ${KYLIN_HOME}/bin/get-properties.sh 
kylin.source.hive.client`
 if [ "${hive_client_mode}" == "beeline" ]
 then
-beeline_params=`sh ${KYLIN_HOME}/bin/get-properties.sh 
kylin.source.hive.beeline-params`
+beeline_params=`bash ${KYLIN_HOME}/bin/get-properties.sh 
kylin.source.hive.beeline-params`
 beeline ${beeline_params} -f 
${KYLIN_HOME}/sample_cube/create_sample_tables.sql  || { exit 1; }
 else
 hive -f ${KYLIN_HOME}/sample_cube/create_sample_tables.sql  || { exit 1; }
@@ -42,8 +42,8 @@ echo "Sample hive tables are created successfully; Going to 
create sample cube..
 hadoop fs -rm -r /tmp/kylin/sample_cube
 
 # set engine type and storage type to cube desc
-default_engine_type=`sh ${KYLIN_HOME}/bin/get-properties.sh 
kylin.engine.default`
-default_storage_type=`sh ${KYLIN_HOME}/bin/get-properties.sh 
kylin.storage.default`
+default_engine_type=`bash 

[13/25] kylin git commit: KYLIN-2353 minor code review

[shaofengshi]

KYLIN-2353 minor code review


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/56daf6d5
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/56daf6d5
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/56daf6d5

Branch: refs/heads/sparkcubing-rebase
Commit: 56daf6d592ae454b3bca97c4a7a948781485befa
Parents: 731a53a
Author: Yang Li 
Authored: Sat Jan 7 19:30:27 2017 +0800
Committer: Yang Li 
Committed: Sat Jan 7 19:30:27 2017 +0800

--
 .../org/apache/kylin/measure/bitmap/BitmapCounter.java| 10 ++
 1 file changed, 6 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/56daf6d5/core-metadata/src/main/java/org/apache/kylin/measure/bitmap/BitmapCounter.java
--
diff --git 
a/core-metadata/src/main/java/org/apache/kylin/measure/bitmap/BitmapCounter.java
 
b/core-metadata/src/main/java/org/apache/kylin/measure/bitmap/BitmapCounter.java
index a18ac4a..a632b0d 100644
--- 
a/core-metadata/src/main/java/org/apache/kylin/measure/bitmap/BitmapCounter.java
+++ 
b/core-metadata/src/main/java/org/apache/kylin/measure/bitmap/BitmapCounter.java
@@ -51,7 +51,9 @@ public class BitmapCounter implements 
Comparable {
 }
 
 if (buffer != null) {
+@SuppressWarnings("unused")
 int version = buffer.getInt();
+@SuppressWarnings("unused")
 int size = buffer.getInt();
 count = buffer.getInt();
 
@@ -79,7 +81,7 @@ public class BitmapCounter implements 
Comparable {
 
 public void add(int value) {
 getBitmap().add(value);
-count = getBitmap().getCardinality();
+count = null;
 }
 
 public void add(byte[] value) {
@@ -103,12 +105,12 @@ public class BitmapCounter implements 
Comparable {
 
 public void merge(BitmapCounter another) {
 getBitmap().or(another.getBitmap());
-count = getBitmap().getCardinality();
+count = null;
 }
 
 public void intersect(BitmapCounter another) {
 getBitmap().and(another.getBitmap());
-count = getBitmap().getCardinality();
+count = null;
 }
 
 public int getCount() {
@@ -148,7 +150,6 @@ public class BitmapCounter implements 
Comparable {
 
 // keep forward compatibility
 if (version == VERSION) {
-@SuppressWarnings("unused")
 int size = in.getInt();
 count = in.getInt();
 in.position(mark);
@@ -258,4 +259,5 @@ public class BitmapCounter implements 
Comparable {
 return len;
 }
 }
+
 }

[14/25] kylin git commit: KYLIN-2364 Output table name to error info in LookupTable

[shaofengshi]

KYLIN-2364 Output table name to error info in LookupTable

Signed-off-by: Billy Liu 


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/e46d699e
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/e46d699e
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/e46d699e

Branch: refs/heads/sparkcubing-rebase
Commit: e46d699e05100db084db354d7efb3786575d5c54
Parents: 56daf6d
Author: kangkaisen 
Authored: Sat Jan 7 19:43:43 2017 +0800
Committer: Billy Liu 
Committed: Sun Jan 8 12:14:05 2017 +0800

--
 .../src/main/java/org/apache/kylin/dict/lookup/LookupTable.java| 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/e46d699e/core-dictionary/src/main/java/org/apache/kylin/dict/lookup/LookupTable.java
--
diff --git 
a/core-dictionary/src/main/java/org/apache/kylin/dict/lookup/LookupTable.java 
b/core-dictionary/src/main/java/org/apache/kylin/dict/lookup/LookupTable.java
index 21221f4..90ca500 100644
--- 
a/core-dictionary/src/main/java/org/apache/kylin/dict/lookup/LookupTable.java
+++ 
b/core-dictionary/src/main/java/org/apache/kylin/dict/lookup/LookupTable.java
@@ -83,7 +83,7 @@ abstract public class LookupTable {
 Array key = new Array(keyCols);
 
 if (data.containsKey(key))
-throw new IllegalStateException("Dup key found, key=" + 
toString(keyCols) + ", value1=" + toString(data.get(key)) + ", value2=" + 
toString(value));
+throw new IllegalStateException("The table: " + 
tableDesc.getName() + " Dup key found, key=" + toString(keyCols) + ", value1=" 
+ toString(data.get(key)) + ", value2=" + toString(value));
 
 data.put(key, value);
 }

[25/25] kylin git commit: KYLIN-2331 use org.apache.htrace.Trace to find htrace-core.jar

[shaofengshi]

KYLIN-2331 use org.apache.htrace.Trace to find htrace-core.jar


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/b9a69efa
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/b9a69efa
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/b9a69efa

Branch: refs/heads/sparkcubing-rebase
Commit: b9a69efaea27d915ae44f75db436b6c9b6b7a4ec
Parents: e864cd3
Author: shaofengshi 
Authored: Tue Jan 3 13:36:49 2017 +0800
Committer: shaofengshi 
Committed: Mon Jan 9 16:58:11 2017 +0800

--
 .../apache/kylin/engine/spark/SparkBatchCubingJobBuilder2.java   | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/b9a69efa/engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkBatchCubingJobBuilder2.java
--
diff --git 
a/engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkBatchCubingJobBuilder2.java
 
b/engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkBatchCubingJobBuilder2.java
index c5d47e7..9431468 100644
--- 
a/engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkBatchCubingJobBuilder2.java
+++ 
b/engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkBatchCubingJobBuilder2.java
@@ -59,10 +59,12 @@ public class SparkBatchCubingJobBuilder2 extends 
BatchCubingJobBuilder2 {
 StringBuilder jars = new StringBuilder();
 
 StringUtil.appendWithSeparator(jars, 
findJar("org.htrace.HTraceConfiguration")); // htrace-core.jar
-StringUtil.appendWithSeparator(jars, 
findJar("org.cloudera.htrace.HTraceConfiguration"));
+StringUtil.appendWithSeparator(jars, 
findJar("org.apache.htrace.Trace")); // htrace-core.jar
+StringUtil.appendWithSeparator(jars, 
findJar("org.cloudera.htrace.HTraceConfiguration")); // htrace-core.jar
 StringUtil.appendWithSeparator(jars, 
findJar("org.apache.hadoop.hbase.client.HConnection")); // hbase-client.jar
 StringUtil.appendWithSeparator(jars, 
findJar("org.apache.hadoop.hbase.HBaseConfiguration")); // hbase-common.jar
 StringUtil.appendWithSeparator(jars, 
findJar("org.apache.hadoop.hbase.util.ByteStringer")); // hbase-protocol.jar
+StringUtil.appendWithSeparator(jars, 
findJar("com.yammer.metrics.core.Gauge")); // metrics-core.jar
 
 StringUtil.appendWithSeparator(jars, 
seg.getConfig().getSparkAdditionalJars());
 sparkExecutable.setJars(jars.toString());

[12/25] kylin git commit: KYLIN-2353 Serialize BitmapCounter with distinct count

[shaofengshi]

KYLIN-2353 Serialize BitmapCounter with distinct count

Signed-off-by: Yang Li 


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/731a53a5
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/731a53a5
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/731a53a5

Branch: refs/heads/sparkcubing-rebase
Commit: 731a53a51a32fb1314b4e1282f158e3b7ed819a5
Parents: aa57446
Author: kangkaisen 
Authored: Sat Dec 31 20:28:30 2016 +0800
Committer: Yang Li 
Committed: Sat Jan 7 19:08:37 2017 +0800

--
 .../kylin/measure/bitmap/BitmapCounter.java | 105 ---
 .../bitmap/BitmapDistinctCountAggFunc.java  |   2 +-
 .../kylin/measure/bitmap/BitmapSerializer.java  |  15 +--
 3 files changed, 70 insertions(+), 52 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/731a53a5/core-metadata/src/main/java/org/apache/kylin/measure/bitmap/BitmapCounter.java
--
diff --git 
a/core-metadata/src/main/java/org/apache/kylin/measure/bitmap/BitmapCounter.java
 
b/core-metadata/src/main/java/org/apache/kylin/measure/bitmap/BitmapCounter.java
index aeb14ba..a18ac4a 100644
--- 
a/core-metadata/src/main/java/org/apache/kylin/measure/bitmap/BitmapCounter.java
+++ 
b/core-metadata/src/main/java/org/apache/kylin/measure/bitmap/BitmapCounter.java
@@ -35,6 +35,8 @@ public class BitmapCounter implements 
Comparable {
 
 private MutableRoaringBitmap bitmap = new MutableRoaringBitmap();
 private final int VERSION = 2;
+private Integer count;
+private ByteBuffer buffer;
 
 public BitmapCounter() {
 }
@@ -43,18 +45,41 @@ public class BitmapCounter implements 
Comparable {
 merge(another);
 }
 
+private MutableRoaringBitmap getBitmap() {
+if (!bitmap.isEmpty()) {
+return bitmap;
+}
+
+if (buffer != null) {
+int version = buffer.getInt();
+int size = buffer.getInt();
+count = buffer.getInt();
+
+try (DataInputStream is = new DataInputStream(new 
ByteBufferBackedInputStream(buffer))) {
+bitmap.deserialize(is);
+} catch (IOException e) {
+throw new RuntimeException("deserialize bitmap failed!");
+}
+
+buffer = null;
+}
+
+return bitmap;
+}
+
 public void clear() {
-bitmap.clear();
+getBitmap().clear();
 }
 
 public BitmapCounter clone() {
 BitmapCounter newCounter = new BitmapCounter();
-newCounter.bitmap = bitmap.clone();
+newCounter.bitmap = getBitmap().clone();
 return newCounter;
 }
 
 public void add(int value) {
-bitmap.add(value);
+getBitmap().add(value);
+count = getBitmap().getCardinality();
 }
 
 public void add(byte[] value) {
@@ -77,34 +102,43 @@ public class BitmapCounter implements 
Comparable {
 }
 
 public void merge(BitmapCounter another) {
-this.bitmap.or(another.bitmap);
+getBitmap().or(another.getBitmap());
+count = getBitmap().getCardinality();
 }
 
 public void intersect(BitmapCounter another) {
-this.bitmap.and(another.bitmap);
+getBitmap().and(another.getBitmap());
+count = getBitmap().getCardinality();
 }
 
-public long getCount() {
-return this.bitmap.getCardinality();
+public int getCount() {
+if (count != null) {
+return count;
+}
+
+return getBitmap().getCardinality();
 }
 
 public int getMemBytes() {
-return this.bitmap.getSizeInBytes();
+return getBitmap().getSizeInBytes();
 }
 
 public Iterator iterator() {
-return bitmap.iterator();
+return getBitmap().iterator();
 }
 
 public void writeRegisters(ByteBuffer out) throws IOException {
 ByteArrayOutputStream bos = new ByteArrayOutputStream();
 DataOutputStream dos = new DataOutputStream(bos);
+MutableRoaringBitmap bitmap = getBitmap();
 bitmap.runOptimize();
 bitmap.serialize(dos);
 dos.close();
 ByteBuffer bb = ByteBuffer.wrap(bos.toByteArray());
+
 out.putInt(VERSION);
-out.putInt(bos.size() + 4 + 4);
+out.putInt(bos.size() + 4 + 4 + 4);
+out.putInt(getCount());
 out.put(bb);
 }
 
@@ -116,13 +150,30 @@ public class BitmapCounter implements 
Comparable {
 if (version == VERSION) {
 @SuppressWarnings("unused")
 int size = in.getInt();
+count = in.getInt();
+in.position(mark);
+buffer = cloneBuffer(in, size);
 } else {
 in.position(mark);
+  

[05/25] kylin git commit: minor, skip JarScan in DebugTomcat

[shaofengshi]

minor, skip JarScan in DebugTomcat


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/1b85f710
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/1b85f710
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/1b85f710

Branch: refs/heads/sparkcubing-rebase
Commit: 1b85f710d91c7f7f5a81790d665a1ac617c7a591
Parents: fbacb71
Author: Li Yang 
Authored: Fri Jan 6 16:28:10 2017 +0800
Committer: Li Yang 
Committed: Fri Jan 6 16:28:20 2017 +0800

--
 server/src/main/java/org/apache/kylin/rest/DebugTomcat.java | 8 
 1 file changed, 8 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/1b85f710/server/src/main/java/org/apache/kylin/rest/DebugTomcat.java
--
diff --git a/server/src/main/java/org/apache/kylin/rest/DebugTomcat.java 
b/server/src/main/java/org/apache/kylin/rest/DebugTomcat.java
index 4f49402..2cac82e 100644
--- a/server/src/main/java/org/apache/kylin/rest/DebugTomcat.java
+++ b/server/src/main/java/org/apache/kylin/rest/DebugTomcat.java
@@ -25,6 +25,8 @@ import org.apache.catalina.startup.Tomcat;
 import org.apache.commons.lang3.StringUtils;
 import org.apache.hadoop.util.Shell;
 import org.apache.kylin.common.KylinConfig;
+import org.apache.tomcat.JarScanFilter;
+import org.apache.tomcat.JarScanType;
 import org.apache.tomcat.util.descriptor.web.ErrorPage;
 
 import java.io.File;
@@ -127,6 +129,12 @@ public class DebugTomcat {
 notFound.setLocation("/index.html");
 webContext.addErrorPage(notFound);
 webContext.addWelcomeFile("index.html");
+webContext.getJarScanner().setJarScanFilter(new JarScanFilter() {
+@Override
+public boolean check(JarScanType arg0, String arg1) {
+return false;
+}
+});
 
 // tomcat start
 tomcat.start();

[02/25] kylin git commit: KYLIN-2287 fix the button display issue with js

[shaofengshi]

KYLIN-2287 fix the button display issue with js

Signed-off-by: zhongjian 


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/49de556f
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/49de556f
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/49de556f

Branch: refs/heads/sparkcubing-rebase
Commit: 49de556f00ff8fd854654daf1bc7e1d26c1fb879
Parents: f267ba9
Author: ShaoChinYu 
Authored: Fri Jan 6 12:22:59 2017 +0800
Committer: zhongjian 
Committed: Fri Jan 6 12:29:35 2017 +0800

--
 webapp/app/js/controllers/models.js | 36 
 webapp/app/partials/models/models_tree.html |  6 ++--
 2 files changed, 39 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/49de556f/webapp/app/js/controllers/models.js
--
diff --git a/webapp/app/js/controllers/models.js 
b/webapp/app/js/controllers/models.js
index e3ee038..fd21845 100644
--- a/webapp/app/js/controllers/models.js
+++ b/webapp/app/js/controllers/models.js
@@ -183,7 +183,41 @@ KylinApp.controller('ModelsCtrl', function ($scope, $q, 
$routeParams, $location,
 });
   };
 
+  function changePositionOfScrollBar(){
+//get which button be clicked
+var btn = window.event.srcElement || window.event.target;
+//get current position of scroll bar
+var scrollTop =$("#cube_model_trees").scrollTop();
+//get total length of scroll bar
+var scrollHeight  = 
document.getElementById('cube_model_trees').scrollHeight;
+//get the position of clicked button relative to the top of window
+var offsetTop  =$(btn).offset().top;
+//get the position of the container relative to the top of window
+var treeOffsetTop = $("#cube_model_trees").offset().top;
+
+//distance from button to the top of tree model container
+var minor = offsetTop - treeOffsetTop;
+//height of tree model container
+var  viewH =$("#cube_model_trees").height();
+
+//change scroll bar to show the dropdown menu
+if(minor + 100 > viewH){//100 is the height of dropdowm menu
+  if((scrollHeight - scrollTop - viewH)>=minor+100-viewH){
+
document.getElementById('cube_model_trees').scrollTop+=(minor+120-viewH);
+  }else{
+var node=document.createElement("LI");
+node.style.height = (minor+120-viewH)+"px";
+document.getElementById("models-tree").appendChild(node);
+var  viewH =$("#cube_model_trees").height();//可见高度
+
document.getElementById('cube_model_trees').scrollTop+=(minor+120-viewH);
+
+  }
+}
+  }
+
   $scope.listModelAccess = function (model) {
+changePositionOfScrollBar();
+
 if(model.uuid){
   AccessService.list({type: "DataModelDesc", uuid: model.uuid}, function 
(accessEntities) {
 model.accessEntities = accessEntities;
@@ -196,6 +230,8 @@ KylinApp.controller('ModelsCtrl', function ($scope, $q, 
$routeParams, $location,
 }
   })
 }
+
+
   };
 
   var ModelDetailModalCtrl = function ($scope, $location, $modalInstance, 
scope) {

http://git-wip-us.apache.org/repos/asf/kylin/blob/49de556f/webapp/app/partials/models/models_tree.html
--
diff --git a/webapp/app/partials/models/models_tree.html 
b/webapp/app/partials/models/models_tree.html
index 944d2dc..eba7d29 100644
--- a/webapp/app/partials/models/models_tree.html
+++ b/webapp/app/partials/models/models_tree.html
@@ -41,10 +41,10 @@
   
 Models
   
+
+
 
-
-
-
+
   
 
 

[18/25] kylin git commit: modify default partition setting

[shaofengshi]

modify default partition setting


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/2cf52b49
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/2cf52b49
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/2cf52b49

Branch: refs/heads/sparkcubing-rebase
Commit: 2cf52b49ed9a1158cb2ec0ded6d907f90ee024fe
Parents: b9a69ef
Author: shaofengshi 
Authored: Thu Jan 5 19:56:10 2017 +0800
Committer: shaofengshi 
Committed: Mon Jan 9 16:58:11 2017 +0800

--
 build/conf/kylin.properties | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/2cf52b49/build/conf/kylin.properties
--
diff --git a/build/conf/kylin.properties b/build/conf/kylin.properties
index bd0bbd4..54430f0 100644
--- a/build/conf/kylin.properties
+++ b/build/conf/kylin.properties
@@ -137,13 +137,13 @@ kylin.engine.spark.env.hadoop-conf-dir=/etc/hadoop/conf
 #kylin.engine.spark.properties-file=
 
 # Estimate the RDD partition numbers
-kylin.engine.spark.rdd-partition-cut-mb=200
+kylin.engine.spark.rdd-partition-cut-mb=50
 
 # Minimal partition numbers of rdd
-kylin.engine.spark.min-partition=1
+kylin.engine.spark.min-partition=10
 
 # Max partition numbers of rdd
-kylin.engine.spark.max-partition=500
+kylin.engine.spark.max-partition=5000
 
 ### CUBE | DICTIONARY ###
 

[09/25] kylin git commit: fix NPE bug

[shaofengshi]

fix NPE bug

Signed-off-by: shaofengshi 


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/5a18af04
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/5a18af04
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/5a18af04

Branch: refs/heads/sparkcubing-rebase
Commit: 5a18af04ca40aaa6509cdf7c39b71b17dc5eb4a3
Parents: 122b15a
Author: xiefan46 <958034...@qq.com>
Authored: Thu Jan 5 23:29:43 2017 +0800
Committer: shaofengshi 
Committed: Sat Jan 7 10:18:20 2017 +0800

--
 .../src/main/java/org/apache/kylin/dict/CacheDictionary.java | 8 ++--
 1 file changed, 2 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/5a18af04/core-dictionary/src/main/java/org/apache/kylin/dict/CacheDictionary.java
--
diff --git 
a/core-dictionary/src/main/java/org/apache/kylin/dict/CacheDictionary.java 
b/core-dictionary/src/main/java/org/apache/kylin/dict/CacheDictionary.java
index 1e260b2..d7ed6bd 100644
--- a/core-dictionary/src/main/java/org/apache/kylin/dict/CacheDictionary.java
+++ b/core-dictionary/src/main/java/org/apache/kylin/dict/CacheDictionary.java
@@ -29,8 +29,6 @@ import java.util.concurrent.ConcurrentHashMap;
 public abstract class CacheDictionary extends Dictionary {
 private static final long serialVersionUID = 1L;
 
-protected transient boolean enableValueCache = false;
-
 private transient SoftReference valueToIdCache;
 
 private transient SoftReference idToValueCache;
@@ -46,7 +44,7 @@ public abstract class CacheDictionary extends 
Dictionary {
 //value --> id
 @Override
 protected final int getIdFromValueImpl(T value, int roundingFlag) {
-if (enableValueCache && roundingFlag == 0) {
+if (this.valueToIdCache != null && roundingFlag == 0) {
 Map cache = valueToIdCache.get(); // SoftReference to skip cache 
gracefully when short of memory
 if (cache != null) {
 Integer id;
@@ -66,7 +64,7 @@ public abstract class CacheDictionary extends 
Dictionary {
 //id --> value
 @Override
 protected final T getValueFromIdImpl(int id) {
-if (enableValueCache) {
+if (this.idToValueCache != null) {
 Object[] cache = idToValueCache.get();
 if (cache != null) {
 int seq = calcSeqNoFromId(id);
@@ -91,7 +89,6 @@ public abstract class CacheDictionary extends 
Dictionary {
 }
 
 public final void enableCache() {
-this.enableValueCache = true;
 if (this.valueToIdCache == null)
 this.valueToIdCache = new SoftReference<>(new ConcurrentHashMap());
 if (this.idToValueCache == null)
@@ -99,7 +96,6 @@ public abstract class CacheDictionary extends 
Dictionary {
 }
 
 public final void disableCache() {
-this.enableValueCache = false;
 this.valueToIdCache = null;
 this.idToValueCache = null;
 }

[07/25] kylin git commit: KYLIN-2337 Remove expensive toString in SortedIteratorMergerWithLimit

[shaofengshi]

KYLIN-2337 Remove expensive toString in SortedIteratorMergerWithLimit

Signed-off-by: Yang Li 


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/7509beee
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/7509beee
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/7509beee

Branch: refs/heads/sparkcubing-rebase
Commit: 7509beeeaaf1d4a50bb82fd83dff5516276b9b9e
Parents: 1087f7a
Author: kangkaisen 
Authored: Thu Dec 29 21:37:26 2016 +0800
Committer: Yang Li 
Committed: Sat Jan 7 09:35:00 2017 +0800

--
 core-cube/src/main/java/org/apache/kylin/gridtable/GTRecord.java   | 1 +
 .../kylin/storage/gtrecord/SortedIteratorMergerWithLimit.java  | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/7509beee/core-cube/src/main/java/org/apache/kylin/gridtable/GTRecord.java
--
diff --git a/core-cube/src/main/java/org/apache/kylin/gridtable/GTRecord.java 
b/core-cube/src/main/java/org/apache/kylin/gridtable/GTRecord.java
index 4d26029..f4480c8 100644
--- a/core-cube/src/main/java/org/apache/kylin/gridtable/GTRecord.java
+++ b/core-cube/src/main/java/org/apache/kylin/gridtable/GTRecord.java
@@ -234,6 +234,7 @@ public class GTRecord implements Comparable, 
Cloneable {
 return toString(info.colAll);
 }
 
+/** toString for MemoryHungry Measure is expensive, please invoke 
carefully */
 public String toString(ImmutableBitSet selectedColumns) {
 Object[] values = new Object[selectedColumns.cardinality()];
 getValues(selectedColumns, values);

http://git-wip-us.apache.org/repos/asf/kylin/blob/7509beee/core-storage/src/main/java/org/apache/kylin/storage/gtrecord/SortedIteratorMergerWithLimit.java
--
diff --git 
a/core-storage/src/main/java/org/apache/kylin/storage/gtrecord/SortedIteratorMergerWithLimit.java
 
b/core-storage/src/main/java/org/apache/kylin/storage/gtrecord/SortedIteratorMergerWithLimit.java
index 2f2691e..6a7e713 100644
--- 
a/core-storage/src/main/java/org/apache/kylin/storage/gtrecord/SortedIteratorMergerWithLimit.java
+++ 
b/core-storage/src/main/java/org/apache/kylin/storage/gtrecord/SortedIteratorMergerWithLimit.java
@@ -126,7 +126,7 @@ public class SortedIteratorMergerWithLimit extends SortedIt
 
 //TODO: remove this check when validated
 if (last != null) {
-Preconditions.checkState(comparator.compare(last, fetched) <= 
0, "Not sorted! last: " + last + " fetched: " + fetched);
+Preconditions.checkState(comparator.compare(last, fetched) <= 
0, "Not sorted!");
 }
 
 last = fetched;

[06/25] kylin git commit: minor, update jetty to 9.2.20

[shaofengshi]

minor, update jetty to 9.2.20


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/1087f7ad
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/1087f7ad
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/1087f7ad

Branch: refs/heads/sparkcubing-rebase
Commit: 1087f7ad49f324b6ee9c1d48b4b6b0489236fe9c
Parents: 1b85f71
Author: Billy Liu 
Authored: Fri Jan 6 16:42:46 2017 +0800
Committer: Billy Liu 
Committed: Fri Jan 6 16:43:04 2017 +0800

--
 pom.xml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/1087f7ad/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 1d47361..22681e8 100644
--- a/pom.xml
+++ b/pom.xml
@@ -71,7 +71,7 @@
 1.1.0
 2.5.2
 1.4.192
-9.3.10.v20160621
+9.2.20.v20161216
 0.3.1
 
 

[17/25] kylin git commit: KYLIN-2368 Enable Findbugs plugin

[shaofengshi]

KYLIN-2368 Enable Findbugs plugin


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/75625ae5
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/75625ae5
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/75625ae5

Branch: refs/heads/sparkcubing-rebase
Commit: 75625ae512b3d6cc080053e4e8871f3fff816ef0
Parents: 6f8cdeb
Author: Billy Liu 
Authored: Mon Jan 9 15:02:06 2017 +0800
Committer: Billy Liu 
Committed: Mon Jan 9 15:02:13 2017 +0800

--
 atopcalcite/pom.xml |  4 +--
 core-common/pom.xml |  5 +--
 core-cube/pom.xml   |  2 +-
 core-dictionary/pom.xml |  5 +--
 core-job/pom.xml|  5 +--
 core-metadata/pom.xml   |  5 +--
 core-storage/pom.xml|  5 +--
 engine-mr/pom.xml   |  6 +---
 engine-spark/pom.xml| 16 --
 jdbc/pom.xml|  3 +-
 kylin-it/pom.xml| 11 ---
 pom.xml | 73 +++-
 query/pom.xml   |  1 +
 server-base/pom.xml |  5 +--
 server/pom.xml  |  2 +-
 source-hive/pom.xml |  5 +--
 source-kafka/pom.xml|  2 +-
 storage-hbase/pom.xml   |  5 +--
 tomcat-ext/pom.xml  |  8 +++--
 tool/pom.xml| 11 ---
 20 files changed, 92 insertions(+), 87 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/75625ae5/atopcalcite/pom.xml
--
diff --git a/atopcalcite/pom.xml b/atopcalcite/pom.xml
index b1bc171..b916df2 100644
--- a/atopcalcite/pom.xml
+++ b/atopcalcite/pom.xml
@@ -24,6 +24,7 @@
 atopcalcite
 jar
 Apache Kylin - Calcite Overrides
+Apache Kylin - Calcite Overrides
 
 
 org.apache.kylin
@@ -31,9 +32,6 @@
 2.0.0-SNAPSHOT
 
 
-
-
-
 
 
 org.apache.calcite

http://git-wip-us.apache.org/repos/asf/kylin/blob/75625ae5/core-common/pom.xml
--
diff --git a/core-common/pom.xml b/core-common/pom.xml
index 2265651..25b10a7 100644
--- a/core-common/pom.xml
+++ b/core-common/pom.xml
@@ -24,17 +24,14 @@
 kylin-core-common
 jar
 Apache Kylin - Core Common
+Apache Kylin - Core Common
 
 
 org.apache.kylin
 kylin
 2.0.0-SNAPSHOT
-
 
 
-
-
-
 
 
 

http://git-wip-us.apache.org/repos/asf/kylin/blob/75625ae5/core-cube/pom.xml
--
diff --git a/core-cube/pom.xml b/core-cube/pom.xml
index b7db363..0a30432 100644
--- a/core-cube/pom.xml
+++ b/core-cube/pom.xml
@@ -24,12 +24,12 @@
 kylin-core-cube
 jar
 Apache Kylin - Core Cube
+Apache Kylin - Core Cube
 
 
 org.apache.kylin
 kylin
 2.0.0-SNAPSHOT
-
 
 
 

http://git-wip-us.apache.org/repos/asf/kylin/blob/75625ae5/core-dictionary/pom.xml
--
diff --git a/core-dictionary/pom.xml b/core-dictionary/pom.xml
index a74eccb..5d839dd 100644
--- a/core-dictionary/pom.xml
+++ b/core-dictionary/pom.xml
@@ -24,17 +24,14 @@
 kylin-core-dictionary
 jar
 Apache Kylin - Core Dictionary
+Apache Kylin - Core Dictionary
 
 
 org.apache.kylin
 kylin
 2.0.0-SNAPSHOT
-
 
 
-
-
-
 
 
 

http://git-wip-us.apache.org/repos/asf/kylin/blob/75625ae5/core-job/pom.xml
--
diff --git a/core-job/pom.xml b/core-job/pom.xml
index dad0428..36d34c8 100644
--- a/core-job/pom.xml
+++ b/core-job/pom.xml
@@ -24,17 +24,14 @@
 kylin-core-job
 jar
 Apache Kylin - Core Job
+Apache Kylin - Core Job
 
 
 org.apache.kylin
 kylin
 2.0.0-SNAPSHOT
-
 
 
-
-
-
 
 
 

http://git-wip-us.apache.org/repos/asf/kylin/blob/75625ae5/core-metadata/pom.xml
--
diff --git a/core-metadata/pom.xml b/core-metadata/pom.xml
index f5c8f8a..b3fe885 100644
--- a/core-metadata/pom.xml
+++ b/core-metadata/pom.xml
@@ -24,17 +24,14 @@
 kylin-core-metadata
 jar
 Apache Kylin - Core Metadata
+Apache Kylin - Core Metadata
 
 
 org.apache.kylin
 kylin
 2.0.0-SNAPSHOT
-
 
 
-
-
-
 
 
 

http://git-wip-us.apache.org/repos/asf/kylin/blob/75625ae5/core-storage/pom.xml
--
diff --git a/core-storage/pom.xml b/core-storage/pom.xml
index 8987608..9bd3f04 100644
--- a/core-storage/pom.xml
+++ b/core-storage/pom.xml
@@ -24,17 +24,14 @@
 kylin-core-storage
 jar
 Apache Kylin - 

[08/25] kylin git commit: KYLIN-2337 minor improvement, still give good error info

[shaofengshi]

KYLIN-2337 minor improvement, still give good error info


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/122b15a2
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/122b15a2
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/122b15a2

Branch: refs/heads/sparkcubing-rebase
Commit: 122b15a2a5fb7385e64839edfb9cbdaec5fde841
Parents: 7509bee
Author: Yang Li 
Authored: Sat Jan 7 09:38:06 2017 +0800
Committer: Yang Li 
Committed: Sat Jan 7 09:38:06 2017 +0800

--
 .../kylin/storage/gtrecord/SortedIteratorMergerWithLimit.java | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/122b15a2/core-storage/src/main/java/org/apache/kylin/storage/gtrecord/SortedIteratorMergerWithLimit.java
--
diff --git 
a/core-storage/src/main/java/org/apache/kylin/storage/gtrecord/SortedIteratorMergerWithLimit.java
 
b/core-storage/src/main/java/org/apache/kylin/storage/gtrecord/SortedIteratorMergerWithLimit.java
index 6a7e713..c5c95a6 100644
--- 
a/core-storage/src/main/java/org/apache/kylin/storage/gtrecord/SortedIteratorMergerWithLimit.java
+++ 
b/core-storage/src/main/java/org/apache/kylin/storage/gtrecord/SortedIteratorMergerWithLimit.java
@@ -126,7 +126,8 @@ public class SortedIteratorMergerWithLimit extends SortedIt
 
 //TODO: remove this check when validated
 if (last != null) {
-Preconditions.checkState(comparator.compare(last, fetched) <= 
0, "Not sorted!");
+if (comparator.compare(last, fetched) > 0)
+throw new IllegalStateException("Not sorted! last: " + 
last + " fetched: " + fetched);
 }
 
 last = fetched;

[24/25] kylin git commit: KYLIN-2344 Package spark into Kylin binary package

[shaofengshi]

KYLIN-2344 Package spark into Kylin binary package


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/e864cd3b
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/e864cd3b
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/e864cd3b

Branch: refs/heads/sparkcubing-rebase
Commit: e864cd3b1c06700b1a1054d1f520eaabedc25d82
Parents: b7d87bb
Author: shaofengshi 
Authored: Tue Jan 3 10:28:38 2017 +0800
Committer: shaofengshi 
Committed: Mon Jan 9 16:58:11 2017 +0800

--
 build/bin/check-env.sh  |  6 ++
 build/bin/find-spark-dependency.sh  | 45 +++
 build/bin/kylin.sh  |  6 ++
 build/conf/kylin-spark-conf.properties  |  2 +-
 build/conf/kylin.properties |  3 -
 build/script/compress.sh|  5 +-
 build/script/download-spark.sh  | 52 +
 build/script/functions.sh   | 60 
 build/script/package.sh |  1 +
 .../org/apache/kylin/common/KylinConfig.java| 14 +
 .../apache/kylin/common/KylinConfigBase.java| 27 -
 .../spark/SparkBatchCubingJobBuilder2.java  |  4 +-
 12 files changed, 200 insertions(+), 25 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/e864cd3b/build/bin/check-env.sh
--
diff --git a/build/bin/check-env.sh b/build/bin/check-env.sh
index a4003c9..e446d66 100644
--- a/build/bin/check-env.sh
+++ b/build/bin/check-env.sh
@@ -47,3 +47,9 @@ if [ $? != 0 ]
 then
 quit "Failed to create $WORKING_DIR. Please make sure the user has right 
to access $WORKING_DIR"
 fi
+
+hadoop fs -mkdir -p $WORKING_DIR/spark-history
+if [ $? != 0 ]
+then
+quit "Failed to create $WORKING_DIR/spark-history. Please make sure the 
user has right to access $WORKING_DIR"
+fi
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/kylin/blob/e864cd3b/build/bin/find-spark-dependency.sh
--
diff --git a/build/bin/find-spark-dependency.sh 
b/build/bin/find-spark-dependency.sh
new file mode 100644
index 000..6f74d8a
--- /dev/null
+++ b/build/bin/find-spark-dependency.sh
@@ -0,0 +1,45 @@
+#!/bin/bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+source $(cd -P -- "$(dirname -- "$0")" && pwd -P)/header.sh
+
+spark_home=
+
+verbose Retrieving Spark dependency...
+if [ -n "$SPARK_HOME" ]
+then
+verbose "SPARK_HOME is set to: $SPARK_HOME, use it to locate Spark 
dependencies."
+spark_home=$SPARK_HOME
+fi
+
+if [ -z "$SPARK_HOME" ]
+then
+verbose "SPARK_HOME wasn't set, use $KYLIN_HOME/spark"
+spark_home=$KYLIN_HOME/spark
+fi
+
+spark_dependency=`find -L $spark_home -name 
'spark-assembly-[a-z0-9A-Z\.-]*.jar' ! -name '*doc*' ! -name '*test*' ! -name 
'*sources*' ''-printf '%p:' | sed 's/:$//'`
+if [ -z "$spark_dependency" ]
+then
+quit "spark assembly lib not found"
+else
+verbose "spark dependency: $spark_dependency"
+export spark_dependency
+fi
+

http://git-wip-us.apache.org/repos/asf/kylin/blob/e864cd3b/build/bin/kylin.sh
--
diff --git a/build/bin/kylin.sh b/build/bin/kylin.sh
index 0cdbbc6..7813b79 100644
--- a/build/bin/kylin.sh
+++ b/build/bin/kylin.sh
@@ -45,6 +45,11 @@ function retrieveDependency() {
 source ${dir}/find-kafka-dependency.sh
 export HBASE_CLASSPATH=${HBASE_CLASSPATH}:${kafka_dependency}
 fi
+
+source ${dir}/find-spark-dependency.sh
+export HBASE_CLASSPATH=${HBASE_CLASSPATH}:${spark_dependency}
+
+verbose "HBASE_CLASSPATH: ${HBASE_CLASSPATH}"
 }
 
 # start command
@@ -112,6 +117,7 @@ then
 -Dkylin.hive.dependency=${hive_dependency} \
 -Dkylin.hbase.dependency=${hbase_dependency} \
 -Dkylin.kafka.dependency=${kafka_dependency} \
+

[01/25] kylin git commit: KYLIN-2361 Upgrade to Tomcat 8.5.9 [Forced Update!]

[shaofengshi]

Repository: kylin
Updated Branches:
  refs/heads/sparkcubing-rebase d94434095 -> ee74a74e4 (forced update)


KYLIN-2361 Upgrade to Tomcat 8.5.9


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/f267ba9d
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/f267ba9d
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/f267ba9d

Branch: refs/heads/sparkcubing-rebase
Commit: f267ba9dcb3db317d07eb367ddff668e03291519
Parents: f573486
Author: Billy Liu 
Authored: Fri Jan 6 10:42:48 2017 +0800
Committer: Billy Liu 
Committed: Fri Jan 6 10:42:48 2017 +0800

--
 build/deploy/context.xml| 47 ++---
 build/deploy/server.xml | 72 +---
 build/script/download-tomcat.sh | 14 ++--
 pom.xml |  2 +-
 .../java/org/apache/kylin/rest/DebugTomcat.java | 10 +--
 .../kylin/ext/CustomizedWebappClassloader.java  |  5 +-
 6 files changed, 82 insertions(+), 68 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/f267ba9d/build/deploy/context.xml
--
diff --git a/build/deploy/context.xml b/build/deploy/context.xml
index 5d1bedf..e8b4cf1 100644
--- a/build/deploy/context.xml
+++ b/build/deploy/context.xml
@@ -1,38 +1,31 @@
-
+
 
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
 
 
 
-
+
+
 WEB-INF/web.xml
+${catalina.base}/conf/web.xml
 
 
 
-
-
-
-
 
-
-
\ No newline at end of file
+

http://git-wip-us.apache.org/repos/asf/kylin/blob/f267ba9d/build/deploy/server.xml
--
diff --git a/build/deploy/server.xml b/build/deploy/server.xml
index 72a87d0..bea6146 100644
--- a/build/deploy/server.xml
+++ b/build/deploy/server.xml
@@ -1,4 +1,4 @@
-
+
 
 
+
 
 
-
-
-
+
 
-
-
-
+
+
+
 
 
 
 -->
-
+
 
+
+
 
 
-
+
 
 
 
 
+   prefix="localhost_access_log" suffix=".txt"
+   pattern="%h %l %u %t %r %s %b" />
 
 
 

http://git-wip-us.apache.org/repos/asf/kylin/blob/f267ba9d/build/script/download-tomcat.sh
--
diff --git a/build/script/download-tomcat.sh b/build/script/download-tomcat.sh
index 9dd26f0..c97c3ce 100755
--- a/build/script/download-tomcat.sh
+++ b/build/script/download-tomcat.sh
@@ -27,22 +27,22 @@ if [[ `uname -a` =~ "Darwin" ]]; then
 alias md5cmd="md5 -q"
 fi
 
-if [ ! -f "build/apache-tomcat-7.0.69.tar.gz" ]
+if [ ! -f "build/apache-tomcat-8.5.9.tar.gz" ]
 then
 echo "no binary file found"
-wget --directory-prefix=build/ 
http://archive.apache.org/dist/tomcat/tomcat-7/v7.0.69/bin/apache-tomcat-7.0.69.tar.gz
 || echo "download tomcat failed"
+wget --directory-prefix=build/ 
http://archive.apache.org/dist/tomcat/tomcat-8/v8.5.9/bin/apache-tomcat-8.5.9.tar.gz
 || echo "download tomcat failed"
 else
-if [ `md5cmd build/apache-tomcat-7.0.69.tar.gz | awk '{print $1}'` != 
"10a071e5169a1a8b14ff35a0ad181052" ]
+if [ `md5cmd build/apache-tomcat-8.5.9.tar.gz | awk '{print $1}'` != 
"b41270a64b7774c964e4bec813eea2ed" ]
 then
 echo "md5 check failed"
-rm build/apache-tomcat-7.0.69.tar.gz
-wget --directory-prefix=build/ 
http://archive.apache.org/dist/tomcat/tomcat-7/v7.0.69/bin/apache-tomcat-7.0.69.tar.gz
 || echo "download tomcat failed"
+rm build/apache-tomcat-8.5.9.tar.gz
+wget --directory-prefix=build/ 
http://archive.apache.org/dist/tomcat/tomcat-8/v8.5.9/bin/apache-tomcat-8.5.9.tar.g
 || echo "download tomcat failed"
 fi
 fi
 unalias md5cmd
 
-tar -zxvf build/apache-tomcat-7.0.69.tar.gz -C build/
-mv build/apache-tomcat-7.0.69 build/tomcat
+tar -zxvf build/apache-tomcat-8.5.9.tar.gz -C 

[23/25] kylin git commit: fix NPE in CacheDictionary in Spark cubing

[shaofengshi]

fix NPE in CacheDictionary in Spark cubing


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/66bca9a6
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/66bca9a6
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/66bca9a6

Branch: refs/heads/sparkcubing-rebase
Commit: 66bca9a676fda02aa88cc25bad2e545f779f6dde
Parents: 2cf52b4
Author: shaofengshi 
Authored: Sat Jan 7 10:17:33 2017 +0800
Committer: shaofengshi 
Committed: Mon Jan 9 16:58:11 2017 +0800

--
 .../java/org/apache/kylin/dict/CacheDictionary.java   |  2 +-
 .../apache/kylin/engine/mr/BatchCubingJobBuilder.java |  5 ++---
 .../kylin/engine/mr/BatchCubingJobBuilder2.java   |  7 +++
 .../org/apache/kylin/engine/mr/JobBuilderSupport.java | 14 +-
 .../engine/spark/SparkBatchCubingJobBuilder2.java |  1 +
 .../apache/kylin/engine/spark/SparkCubingByLayer.java |  2 +-
 .../kylin/engine/spark/SparkCubingJobBuilder.java |  1 -
 7 files changed, 9 insertions(+), 23 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/66bca9a6/core-dictionary/src/main/java/org/apache/kylin/dict/CacheDictionary.java
--
diff --git 
a/core-dictionary/src/main/java/org/apache/kylin/dict/CacheDictionary.java 
b/core-dictionary/src/main/java/org/apache/kylin/dict/CacheDictionary.java
index d7ed6bd..b2bad53 100644
--- a/core-dictionary/src/main/java/org/apache/kylin/dict/CacheDictionary.java
+++ b/core-dictionary/src/main/java/org/apache/kylin/dict/CacheDictionary.java
@@ -35,7 +35,7 @@ public abstract class CacheDictionary extends 
Dictionary {
 
 protected transient int baseId;
 
-protected transient BytesConverter bytesConvert;
+protected BytesConverter bytesConvert;
 
 public CacheDictionary() {
 

http://git-wip-us.apache.org/repos/asf/kylin/blob/66bca9a6/engine-mr/src/main/java/org/apache/kylin/engine/mr/BatchCubingJobBuilder.java
--
diff --git 
a/engine-mr/src/main/java/org/apache/kylin/engine/mr/BatchCubingJobBuilder.java 
b/engine-mr/src/main/java/org/apache/kylin/engine/mr/BatchCubingJobBuilder.java
index 456f615..36c12a1 100644
--- 
a/engine-mr/src/main/java/org/apache/kylin/engine/mr/BatchCubingJobBuilder.java
+++ 
b/engine-mr/src/main/java/org/apache/kylin/engine/mr/BatchCubingJobBuilder.java
@@ -65,12 +65,11 @@ public class BatchCubingJobBuilder extends 
JobBuilderSupport {
 // Phase 3: Build Cube
 RowKeyDesc rowKeyDesc = seg.getCubeDesc().getRowkey();
 final int groupRowkeyColumnsCount = seg.getCubeDesc().getBuildLevel();
-final String[] cuboidOutputTempPath = 
getCuboidOutputPaths(cuboidRootPath, groupRowkeyColumnsCount);
 // base cuboid step
-result.addTask(createBaseCuboidStep(cuboidOutputTempPath[0], jobId));
+
result.addTask(createBaseCuboidStep(getCuboidOutputPathsByLevel(cuboidRootPath, 
0), jobId));
 // n dim cuboid steps
 for (int i = 1; i <= groupRowkeyColumnsCount; i++) {
-result.addTask(createNDimensionCuboidStep(cuboidOutputTempPath[i - 
1], cuboidOutputTempPath[i], i));
+
result.addTask(createNDimensionCuboidStep(getCuboidOutputPathsByLevel(cuboidRootPath,
 i - 1), getCuboidOutputPathsByLevel(cuboidRootPath, i), i));
 }
 outputSide.addStepPhase3_BuildCube(result, cuboidRootPath);
 

http://git-wip-us.apache.org/repos/asf/kylin/blob/66bca9a6/engine-mr/src/main/java/org/apache/kylin/engine/mr/BatchCubingJobBuilder2.java
--
diff --git 
a/engine-mr/src/main/java/org/apache/kylin/engine/mr/BatchCubingJobBuilder2.java
 
b/engine-mr/src/main/java/org/apache/kylin/engine/mr/BatchCubingJobBuilder2.java
index 700f821..dd866bd 100644
--- 
a/engine-mr/src/main/java/org/apache/kylin/engine/mr/BatchCubingJobBuilder2.java
+++ 
b/engine-mr/src/main/java/org/apache/kylin/engine/mr/BatchCubingJobBuilder2.java
@@ -76,14 +76,13 @@ public class BatchCubingJobBuilder2 extends 
JobBuilderSupport {
 return result;
 }
 
-private void addLayerCubingSteps(final CubingJob result, final String 
jobId, final String cuboidRootPath) {
+protected void addLayerCubingSteps(final CubingJob result, final String 
jobId, final String cuboidRootPath) {
 final int maxLevel = seg.getCubeDesc().getBuildLevel();
-final String[] cuboidOutputTempPath = 
getCuboidOutputPaths(cuboidRootPath, maxLevel);
 // base cuboid step
-result.addTask(createBaseCuboidStep(cuboidOutputTempPath[0], jobId));
+
result.addTask(createBaseCuboidStep(getCuboidOutputPathsByLevel(cuboidRootPath, 
0), jobId));
 // n dim cuboid 

[16/25] kylin git commit: KYLIN-2367 fix select * where ...

[shaofengshi]

KYLIN-2367 fix select * where ...


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/6f8cdeb5
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/6f8cdeb5
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/6f8cdeb5

Branch: refs/heads/sparkcubing-rebase
Commit: 6f8cdeb55cb4b41a63871aed2b3f61c7eddd107d
Parents: 68ff75d
Author: Li Yang 
Authored: Mon Jan 9 14:41:05 2017 +0800
Committer: Li Yang 
Committed: Mon Jan 9 14:41:05 2017 +0800

--
 .../org/apache/calcite/sql2rel/SqlToRelConverter.java | 14 +-
 1 file changed, 9 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/6f8cdeb5/atopcalcite/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java
--
diff --git 
a/atopcalcite/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java 
b/atopcalcite/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java
index 7da3040..e0782ce 100644
--- 
a/atopcalcite/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java
+++ 
b/atopcalcite/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java
@@ -677,7 +677,8 @@ public class SqlToRelConverter {
  * 
  *   LogicalSort (optional)
  *|- LogicalProject
- *|- OLAPTableScan
+ *|- LogicalFilter (optional)
+ *|- OLAPTableScan
  */
 LogicalProject rootPrj = null;
 LogicalSort rootSort = null;
@@ -690,11 +691,14 @@ public class SqlToRelConverter {
 return root;
 }
 
-if 
(!rootPrj.getInput().getClass().getSimpleName().equals("OLAPTableScan"))
+RelNode input = rootPrj.getInput();
+if (!(//
+input.getClass().getSimpleName().equals("OLAPTableScan")//
+|| (input.getClass().getSimpleName().equals("LogicalFilter") 
&& input.getInput(0).getClass().getSimpleName().equals("OLAPTableScan"))//
+ ))
 return root;
 
-RelNode scan = rootPrj.getInput();
-if (rootPrj.getRowType().getFieldCount() < 
scan.getRowType().getFieldCount())
+if (rootPrj.getRowType().getFieldCount() < 
input.getRowType().getFieldCount())
 return root;
 
 RelDataType inType = rootPrj.getRowType();
@@ -713,7 +717,7 @@ public class SqlToRelConverter {
 }
 
 RelDataType projRowType = 
getCluster().getTypeFactory().createStructType(projTypeBuilder);
-rootPrj = LogicalProject.create(scan, projExp, projRowType);
+rootPrj = LogicalProject.create(input, projExp, projRowType);
 if (rootSort != null) {
 rootSort = (LogicalSort) rootSort.copy(rootSort.getTraitSet(), 
rootPrj, rootSort.collation, rootSort.offset, rootSort.fetch);
 }

[21/25] kylin git commit: KYLIN-2331 by layer spark cubing

[shaofengshi]

KYLIN-2331 by layer spark cubing


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/b7d87bb4
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/b7d87bb4
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/b7d87bb4

Branch: refs/heads/sparkcubing-rebase
Commit: b7d87bb4e6c4de484d910e9af04965392da518e5
Parents: 75625ae
Author: shaofengshi 
Authored: Sun Dec 25 15:59:16 2016 +0800
Committer: shaofengshi 
Committed: Mon Jan 9 16:58:11 2017 +0800

--
 assembly/pom.xml|   5 +-
 .../main/config/assemblies/source-assembly.xml  |   3 -
 .../java/org/apache/kylin/job/DeployUtil.java   |   1 -
 build/conf/kylin-spark-conf.properties  |  26 ++
 build/conf/kylin.properties |  20 +
 .../apache/kylin/common/KylinConfigBase.java|  64 ++-
 .../kylin/common/persistence/ResourceStore.java |  35 ++
 .../org/apache/kylin/common/util/Array.java |   2 +-
 .../apache/kylin/common/util/SplittedBytes.java |   2 +-
 .../java/org/apache/kylin/cube/CubeSegment.java |   2 +-
 .../kylin/cube/common/RowKeySplitter.java   |  15 +-
 .../org/apache/kylin/cube/cuboid/Cuboid.java|  27 +-
 .../kylin/cube/cuboid/CuboidScheduler.java  |  15 +-
 .../cube/inmemcubing/InMemCubeBuilder.java  |   2 +
 .../kylin/cube/kv/AbstractRowKeyEncoder.java|   6 +-
 .../org/apache/kylin/cube/kv/CubeDimEncMap.java |   7 +-
 .../apache/kylin/cube/kv/RowKeyColumnIO.java|   2 +-
 .../org/apache/kylin/cube/kv/RowKeyEncoder.java |  19 +-
 .../kylin/cube/kv/RowKeyEncoderProvider.java|   2 +-
 .../kylin/cube/model/AggregationGroup.java  |   2 +-
 .../org/apache/kylin/cube/model/CubeDesc.java   |   2 +-
 .../cube/model/CubeJoinedFlatTableDesc.java |   2 +-
 .../cube/model/CubeJoinedFlatTableEnrich.java   |   6 +-
 .../apache/kylin/cube/model/DictionaryDesc.java |   4 +-
 .../apache/kylin/cube/model/DimensionDesc.java  |  14 +-
 .../kylin/cube/model/HBaseColumnDesc.java   |  11 +-
 .../kylin/cube/model/HBaseColumnFamilyDesc.java |   9 +-
 .../kylin/cube/model/HBaseMappingDesc.java  |  10 +-
 .../apache/kylin/cube/model/HierarchyDesc.java  |   2 +-
 .../apache/kylin/cube/model/RowKeyColDesc.java  |   2 +-
 .../org/apache/kylin/cube/model/RowKeyDesc.java |  18 +-
 .../org/apache/kylin/cube/model/SelectRule.java |   2 +-
 .../apache/kylin/gridtable/UnitTestSupport.java |  41 +-
 .../gridtable/AggregationCacheMemSizeTest.java  |   6 +-
 .../gridtable/AggregationCacheSpillTest.java|   5 +-
 .../kylin/gridtable/SimpleGridTableTest.java|  11 +-
 .../gridtable/SimpleInvertedIndexTest.java  |   3 +-
 .../metadata/measure/MeasureCodecTest.java  |   6 +-
 .../apache/kylin/dict/AppendTrieDictionary.java |   6 +-
 .../org/apache/kylin/dict/NumberDictionary.java |   2 +-
 .../apache/kylin/dict/NumberDictionary2.java|   2 +-
 .../dict/NumberDictionaryForestBuilder.java |   4 +-
 .../apache/kylin/dict/StringBytesConverter.java |   2 +-
 .../kylin/job/execution/ExecutableManager.java  |  21 +-
 .../kylin/dimension/AbstractDateDimEnc.java |  12 +-
 .../apache/kylin/dimension/BooleanDimEnc.java   |   4 +-
 .../kylin/dimension/DictionaryDimEnc.java   |  21 +-
 .../apache/kylin/dimension/FixedLenDimEnc.java  |   4 +-
 .../kylin/dimension/FixedLenHexDimEnc.java  |   4 +-
 .../org/apache/kylin/dimension/IntDimEnc.java   |   4 +-
 .../apache/kylin/dimension/IntegerDimEnc.java   |   4 +-
 .../kylin/dimension/OneMoreByteVLongDimEnc.java |  14 +-
 .../kylin/measure/BufferedMeasureCodec.java |  10 +-
 .../apache/kylin/measure/MeasureAggregator.java |   2 +
 .../kylin/measure/MeasureAggregators.java   |  18 +
 .../org/apache/kylin/measure/MeasureCodec.java  |   8 +-
 .../apache/kylin/measure/MeasureIngester.java   |  12 +-
 .../org/apache/kylin/measure/MeasureType.java   |  12 +-
 .../measure/basic/BigDecimalMaxAggregator.java  |  14 +
 .../measure/basic/BigDecimalMinAggregator.java  |  28 +-
 .../measure/basic/BigDecimalSumAggregator.java  |  10 +
 .../kylin/measure/basic/DoubleIngester.java |  19 +-
 .../measure/basic/DoubleMaxAggregator.java  |  20 +-
 .../measure/basic/DoubleMinAggregator.java  |  20 +-
 .../measure/basic/DoubleSumAggregator.java  |  18 +-
 .../kylin/measure/basic/LongIngester.java   |  19 +-
 .../kylin/measure/basic/LongMaxAggregator.java  |  20 +-
 .../kylin/measure/basic/LongMinAggregator.java  |  20 +-
 .../kylin/measure/basic/LongSumAggregator.java  |  18 +-
 .../kylin/measure/bitmap/BitmapAggregator.java  |  13 +
 .../kylin/measure/bitmap/BitmapCounter.java |   2 +-
 .../kylin/measure/bitmap/BitmapMeasureType.java |   5 +
 .../ExtendedColumnMeasureType.java  |  15 +
 .../kylin/measure/hllc/DenseRegister.java   |   2 +-
 .../kylin/measure/hllc/HLLCAggregator.java  |   7 +
 

[10/25] kylin git commit: KYLIN-2338 refactor BitmapCounter.DataInputByteBuffer

[shaofengshi]

KYLIN-2338 refactor BitmapCounter.DataInputByteBuffer

Signed-off-by: Yang Li 


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/746e3801
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/746e3801
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/746e3801

Branch: refs/heads/sparkcubing-rebase
Commit: 746e3801faec58f0e10e619f0697ffbe37c08c63
Parents: 5a18af0
Author: kangkaisen 
Authored: Mon Dec 26 20:12:30 2016 +0800
Committer: Yang Li 
Committed: Sat Jan 7 19:04:52 2017 +0800

--
 .../kylin/measure/bitmap/BitmapCounter.java | 111 +++
 1 file changed, 18 insertions(+), 93 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/746e3801/core-metadata/src/main/java/org/apache/kylin/measure/bitmap/BitmapCounter.java
--
diff --git 
a/core-metadata/src/main/java/org/apache/kylin/measure/bitmap/BitmapCounter.java
 
b/core-metadata/src/main/java/org/apache/kylin/measure/bitmap/BitmapCounter.java
index 827390d..a7f277e 100644
--- 
a/core-metadata/src/main/java/org/apache/kylin/measure/bitmap/BitmapCounter.java
+++ 
b/core-metadata/src/main/java/org/apache/kylin/measure/bitmap/BitmapCounter.java
@@ -26,7 +26,6 @@ import java.io.InputStream;
 import java.nio.ByteBuffer;
 import java.util.Iterator;
 
-import org.apache.commons.io.IOUtils;
 import org.roaringbitmap.buffer.MutableRoaringBitmap;
 
 /**
@@ -107,12 +106,8 @@ public class BitmapCounter implements 
Comparable {
 }
 
 public void readRegisters(ByteBuffer in) throws IOException {
-DataInputByteBuffer input = new DataInputByteBuffer();
-try {
-input.reset(new ByteBuffer[] { in });
-bitmap.deserialize(input);
-} finally {
-IOUtils.closeQuietly(input);
+try (DataInputStream is = new DataInputStream(new 
ByteBufferBackedInputStream(in))) {
+bitmap.deserialize(is);
 }
 }
 
@@ -178,15 +173,11 @@ public class BitmapCounter implements 
Comparable {
 int mark = in.position();
 int len;
 
-DataInputByteBuffer input = new DataInputByteBuffer();
-input.reset(new ByteBuffer[] { in });
 MutableRoaringBitmap bitmap = new MutableRoaringBitmap();
-try {
-bitmap.deserialize(input);
+try (DataInputStream is = new DataInputStream(new 
ByteBufferBackedInputStream(in))) {
+bitmap.deserialize(is);
 } catch (IOException e) {
 throw new IllegalStateException(e);
-} finally {
-IOUtils.closeQuietly(input);
 }
 
 len = in.position() - mark;
@@ -194,93 +185,27 @@ public class BitmapCounter implements 
Comparable {
 return len;
 }
 
-static class DataInputByteBuffer extends DataInputStream {
-private DataInputByteBuffer.Buffer buffers;
+private class ByteBufferBackedInputStream extends InputStream {
+private final ByteBuffer buffer;
 
-public DataInputByteBuffer() {
-this(new DataInputByteBuffer.Buffer());
+private ByteBufferBackedInputStream(ByteBuffer buf) {
+buffer = buf;
 }
 
-private DataInputByteBuffer(DataInputByteBuffer.Buffer buffers) {
-super(buffers);
-this.buffers = buffers;
+@Override
+public int read() throws IOException {
+return buffer.hasRemaining() ? (buffer.get() & 0xFF) : -1;
 }
 
-public void reset(ByteBuffer... input) {
-this.buffers.reset(input);
-}
-
-public ByteBuffer[] getData() {
-return this.buffers.getData();
-}
-
-public int getPosition() {
-return this.buffers.getPosition();
-}
-
-public int getLength() {
-return this.buffers.getLength();
-}
-
-private static class Buffer extends InputStream {
-private final byte[] scratch;
-ByteBuffer[] buffers;
-int bidx;
-int pos;
-int length;
-
-private Buffer() {
-this.scratch = new byte[1];
-this.buffers = new ByteBuffer[0];
-}
-
-public int read() {
-return -1 == this.read(this.scratch, 0, 1) ? -1 : 
this.scratch[0] & 255;
+@Override
+public int read(byte[] bytes, int off, int len) throws IOException {
+if (!buffer.hasRemaining()) {
+return -1;
 }
 
-public int read(byte[] b, int off, int len) {
-if (this.bidx >= this.buffers.length) {
-return -1;
-} else {
-int cur = 

[22/25] kylin git commit: KYLIN-2371 Allow overwrite default spark conf at cube level

[shaofengshi]

KYLIN-2371 Allow overwrite default spark conf at cube level


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/ee74a74e
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/ee74a74e
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/ee74a74e

Branch: refs/heads/sparkcubing-rebase
Commit: ee74a74e4d17a45b3832912381220af39e71f3d5
Parents: 66bca9a
Author: shaofengshi 
Authored: Mon Jan 9 15:23:22 2017 +0800
Committer: shaofengshi 
Committed: Mon Jan 9 16:58:11 2017 +0800

--
 build/conf/kylin-spark-conf.properties  |  5 +--
 build/conf/kylin.properties |  6 ++--
 .../apache/kylin/common/KylinConfigBase.java| 12 ---
 .../kylin/job/constant/ExecutableConstants.java |  1 +
 .../spark/SparkBatchCubingJobBuilder2.java  |  2 +-
 .../kylin/engine/spark/SparkCubingByLayer.java  |  7 ++--
 .../kylin/engine/spark/SparkExecutable.java | 38 +---
 .../test_case_data/sandbox/kylin.properties |  3 +-
 8 files changed, 55 insertions(+), 19 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/ee74a74e/build/conf/kylin-spark-conf.properties
--
diff --git a/build/conf/kylin-spark-conf.properties 
b/build/conf/kylin-spark-conf.properties
index 5e6dafe..143e7e4 100644
--- a/build/conf/kylin-spark-conf.properties
+++ b/build/conf/kylin-spark-conf.properties
@@ -1,5 +1,5 @@
 spark.yarn.submit.file.replication=1
-spark.yarn.executor.memoryOverhead=200
+spark.yarn.executor.memoryOverhead=1024
 spark.yarn.driver.memoryOverhead=384
 spark.master=yarn
 spark.submit.deployMode=cluster
@@ -20,7 +20,8 @@ spark.executor.cores=4
 spark.executor.instances=8
 spark.history.kerberos.keytab=none
 spark.history.kerberos.principal=none
-#spark.yarn.jar=hdfs://namenode:8020/apps/spark/spark-assembly-1.6.3-hadoop2.6.0.jar
 spark.driver.extraJavaOptions=-Dhdp.version=current
 spark.yarn.am.extraJavaOptions=-Dhdp.version=current
 spark.executor.extraJavaOptions=-Dhdp.version=current
+#spark.yarn.jar=hdfs://namenode:8020/kylin/spark/spark-assembly-1.6.3-hadoop2.6.0.jar
+#spark.io.compression.codec=org.apache.spark.io.SnappyCompressionCodec

http://git-wip-us.apache.org/repos/asf/kylin/blob/ee74a74e/build/conf/kylin.properties
--
diff --git a/build/conf/kylin.properties b/build/conf/kylin.properties
index 54430f0..6efa423 100644
--- a/build/conf/kylin.properties
+++ b/build/conf/kylin.properties
@@ -131,16 +131,16 @@ kylin.engine.mr.mapper-input-rows=100
 ### Spark Engine ###
 
 # Hadoop conf folder, will export this as "HADOOP_CONF_DIR" before run 
spark-submit
-kylin.engine.spark.env.hadoop-conf-dir=/etc/hadoop/conf
+#kylin.engine.spark.env.hadoop-conf-dir=/etc/hive/conf
 
 # Spark job submission properties file, default be 
$KYLIN_HOME/conf/kylin-spark-conf.properties
 #kylin.engine.spark.properties-file=
 
 # Estimate the RDD partition numbers
-kylin.engine.spark.rdd-partition-cut-mb=50
+kylin.engine.spark.rdd-partition-cut-mb=10
 
 # Minimal partition numbers of rdd
-kylin.engine.spark.min-partition=10
+kylin.engine.spark.min-partition=1
 
 # Max partition numbers of rdd
 kylin.engine.spark.max-partition=5000

http://git-wip-us.apache.org/repos/asf/kylin/blob/ee74a74e/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
--
diff --git 
a/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java 
b/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
index 7d6ac2b..36ddbf4 100644
--- a/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
+++ b/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
@@ -695,6 +695,10 @@ abstract public class KylinConfigBase implements 
Serializable {
 return getPropertiesByPrefix("kylin.engine.mr.config-override.");
 }
 
+public Map getSparkConfigOverride() {
+return getPropertiesByPrefix("kylin.engine.spark.config-override.");
+}
+
 public double getDefaultHadoopJobReducerInputMB() {
 return 
Double.parseDouble(getOptional("kylin.engine.mr.reduce-input-mb", "500"));
 }
@@ -739,8 +743,8 @@ abstract public class KylinConfigBase implements 
Serializable {
 // ENGINE.SPARK
 // 

 
-public String getSparkHadoopConfDir() {
-return getRequired("kylin.engine.spark.env.hadoop-conf-dir");
+public String getHadoopConfDir() {
+return getOptional("kylin.engine.spark.env.hadoop-conf-dir", "");
 }
 
 public String getSparkConfFile() {
@@ 

[04/25] kylin git commit: KYLIN-2362 Unify shell interpreter in scripts

[shaofengshi]

KYLIN-2362 Unify shell interpreter in scripts

Signed-off-by: lidongsjtu 


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/fbacb71e
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/fbacb71e
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/fbacb71e

Branch: refs/heads/sparkcubing-rebase
Commit: fbacb71ed7aa982b04dc4556c253a9bcfee576fa
Parents: 40ede31
Author: etherge 
Authored: Fri Jan 6 14:18:47 2017 +0800
Committer: lidongsjtu 
Committed: Fri Jan 6 15:53:52 2017 +0800

--
 build/bin/check-env.sh| 2 +-
 build/bin/find-hive-dependency.sh | 4 ++--
 build/bin/kylin.sh| 2 +-
 build/bin/sample.sh   | 8 
 4 files changed, 8 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/fbacb71e/build/bin/check-env.sh
--
diff --git a/build/bin/check-env.sh b/build/bin/check-env.sh
index 9cd8a64..a4003c9 100644
--- a/build/bin/check-env.sh
+++ b/build/bin/check-env.sh
@@ -41,7 +41,7 @@ then
 quit "Please make sure the user has the privilege to run hadoop shell"
 fi
 
-WORKING_DIR=`sh $KYLIN_HOME/bin/get-properties.sh kylin.env.hdfs-working-dir`
+WORKING_DIR=`bash $KYLIN_HOME/bin/get-properties.sh kylin.env.hdfs-working-dir`
 hadoop fs -mkdir -p $WORKING_DIR
 if [ $? != 0 ]
 then

http://git-wip-us.apache.org/repos/asf/kylin/blob/fbacb71e/build/bin/find-hive-dependency.sh
--
diff --git a/build/bin/find-hive-dependency.sh 
b/build/bin/find-hive-dependency.sh
index fe9e0a3..453a35a 100644
--- a/build/bin/find-hive-dependency.sh
+++ b/build/bin/find-hive-dependency.sh
@@ -19,13 +19,13 @@
 
 source $(cd -P -- "$(dirname -- "$0")" && pwd -P)/header.sh
 
-client_mode=`sh ${KYLIN_HOME}/bin/get-properties.sh kylin.source.hive.client`
+client_mode=`bash ${KYLIN_HOME}/bin/get-properties.sh kylin.source.hive.client`
 hive_env=
 
 echo Retrieving hive dependency...
 if [ "${client_mode}" == "beeline" ]
 then
-beeline_params=`sh ${KYLIN_HOME}/bin/get-properties.sh 
kylin.source.hive.beeline-params`
+beeline_params=`bash ${KYLIN_HOME}/bin/get-properties.sh 
kylin.source.hive.beeline-params`
 hive_env=`beeline ${beeline_params} --outputformat=dsv -e set 2>&1 | grep 
'env:CLASSPATH' `
 else
 hive_env=`hive -e set 2>&1 | grep 'env:CLASSPATH'`

http://git-wip-us.apache.org/repos/asf/kylin/blob/fbacb71e/build/bin/kylin.sh
--
diff --git a/build/bin/kylin.sh b/build/bin/kylin.sh
index fca1e5c..0cdbbc6 100644
--- a/build/bin/kylin.sh
+++ b/build/bin/kylin.sh
@@ -73,7 +73,7 @@ then
 #In this way we no longer need to explicitly configure hadoop/hbase 
related classpath for tomcat,
 #hbase command will do all the dirty tasks for us:
 
-spring_profile=`sh ${dir}/get-properties.sh kylin.security.profile`
+spring_profile=`bash ${dir}/get-properties.sh kylin.security.profile`
 if [ -z "$spring_profile" ]
 then
 quit 'please set kylin.security.profile in kylin.properties, options 
are: testing, ldap, saml.'

http://git-wip-us.apache.org/repos/asf/kylin/blob/fbacb71e/build/bin/sample.sh
--
diff --git a/build/bin/sample.sh b/build/bin/sample.sh
index 1397034..1363a5d 100644
--- a/build/bin/sample.sh
+++ b/build/bin/sample.sh
@@ -29,10 +29,10 @@ echo "Loading sample data into HDFS tmp path: 
/tmp/kylin/sample_cube/data"
 hadoop fs -mkdir -p /tmp/kylin/sample_cube/data
 hadoop fs -put * /tmp/kylin/sample_cube/data/
 
-hive_client_mode=`sh ${KYLIN_HOME}/bin/get-properties.sh 
kylin.source.hive.client`
+hive_client_mode=`bash ${KYLIN_HOME}/bin/get-properties.sh 
kylin.source.hive.client`
 if [ "${hive_client_mode}" == "beeline" ]
 then
-beeline_params=`sh ${KYLIN_HOME}/bin/get-properties.sh 
kylin.source.hive.beeline-params`
+beeline_params=`bash ${KYLIN_HOME}/bin/get-properties.sh 
kylin.source.hive.beeline-params`
 beeline ${beeline_params} -f 
${KYLIN_HOME}/sample_cube/create_sample_tables.sql  || { exit 1; }
 else
 hive -f ${KYLIN_HOME}/sample_cube/create_sample_tables.sql  || { exit 1; }
@@ -42,8 +42,8 @@ echo "Sample hive tables are created successfully; Going to 
create sample cube..
 hadoop fs -rm -r /tmp/kylin/sample_cube
 
 # set engine type and storage type to cube desc
-default_engine_type=`sh ${KYLIN_HOME}/bin/get-properties.sh 
kylin.engine.default`
-default_storage_type=`sh ${KYLIN_HOME}/bin/get-properties.sh 
kylin.storage.default`
+default_engine_type=`bash ${KYLIN_HOME}/bin/get-properties.sh 
kylin.engine.default`
+default_storage_type=`bash 

[11/25] kylin git commit: KYLIN-2349 Serialize BitmapCounter with peekLength

[shaofengshi]

KYLIN-2349 Serialize BitmapCounter with peekLength

Signed-off-by: Yang Li 


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/aa574462
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/aa574462
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/aa574462

Branch: refs/heads/sparkcubing-rebase
Commit: aa574462aad0620fe10343613dc4c8d761c0d6a0
Parents: 746e380
Author: kangkaisen 
Authored: Sat Dec 31 15:41:07 2016 +0800
Committer: Yang Li 
Committed: Sat Jan 7 19:08:15 2017 +0800

--
 .../kylin/measure/bitmap/BitmapCounter.java | 53 ++--
 1 file changed, 37 insertions(+), 16 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/aa574462/core-metadata/src/main/java/org/apache/kylin/measure/bitmap/BitmapCounter.java
--
diff --git 
a/core-metadata/src/main/java/org/apache/kylin/measure/bitmap/BitmapCounter.java
 
b/core-metadata/src/main/java/org/apache/kylin/measure/bitmap/BitmapCounter.java
index a7f277e..aeb14ba 100644
--- 
a/core-metadata/src/main/java/org/apache/kylin/measure/bitmap/BitmapCounter.java
+++ 
b/core-metadata/src/main/java/org/apache/kylin/measure/bitmap/BitmapCounter.java
@@ -34,6 +34,7 @@ import org.roaringbitmap.buffer.MutableRoaringBitmap;
 public class BitmapCounter implements Comparable {
 
 private MutableRoaringBitmap bitmap = new MutableRoaringBitmap();
+private final int VERSION = 2;
 
 public BitmapCounter() {
 }
@@ -102,15 +103,51 @@ public class BitmapCounter implements 
Comparable {
 bitmap.serialize(dos);
 dos.close();
 ByteBuffer bb = ByteBuffer.wrap(bos.toByteArray());
+out.putInt(VERSION);
+out.putInt(bos.size() + 4 + 4);
 out.put(bb);
 }
 
 public void readRegisters(ByteBuffer in) throws IOException {
+int mark = in.position();
+int version = in.getInt();
+
+// keep forward compatibility
+if (version == VERSION) {
+@SuppressWarnings("unused")
+int size = in.getInt();
+} else {
+in.position(mark);
+}
+
 try (DataInputStream is = new DataInputStream(new 
ByteBufferBackedInputStream(in))) {
 bitmap.deserialize(is);
 }
 }
 
+public int peekLength(ByteBuffer in) {
+int mark = in.position();
+int len;
+int version = in.getInt();
+
+// keep forward compatibility
+if (version == VERSION) {
+len = in.getInt() ;
+} else {
+in.position(mark);
+try (DataInputStream is = new DataInputStream(new 
ByteBufferBackedInputStream(in))) {
+MutableRoaringBitmap bitmap = new MutableRoaringBitmap();
+bitmap.deserialize(is);
+len = in.position() - mark;
+} catch (IOException e) {
+throw new IllegalStateException(e);
+}
+}
+
+in.position(mark);
+return len;
+}
+
 @Override
 public String toString() {
 long count = getCount();
@@ -169,22 +206,6 @@ public class BitmapCounter implements 
Comparable {
 return -1;
 }
 
-public int peekLength(ByteBuffer in) {
-int mark = in.position();
-int len;
-
-MutableRoaringBitmap bitmap = new MutableRoaringBitmap();
-try (DataInputStream is = new DataInputStream(new 
ByteBufferBackedInputStream(in))) {
-bitmap.deserialize(is);
-} catch (IOException e) {
-throw new IllegalStateException(e);
-}
-
-len = in.position() - mark;
-in.position(mark);
-return len;
-}
-
 private class ByteBufferBackedInputStream extends InputStream {
 private final ByteBuffer buffer;
 

kylin git commit: fix protobuf version

[billyliu]

Repository: kylin
Updated Branches:
  refs/heads/calcite-11 3a2117d4c -> a99f5d331


fix protobuf version


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/a99f5d33
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/a99f5d33
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/a99f5d33

Branch: refs/heads/calcite-11
Commit: a99f5d3310dcaed5523549bfe3fd9e0a9b23969e
Parents: 3a2117d
Author: Billy Liu 
Authored: Mon Jan 9 16:24:25 2017 +0800
Committer: Billy Liu 
Committed: Mon Jan 9 16:24:25 2017 +0800

--
 jdbc/pom.xml   | 5 +
 pom.xml| 2 ++
 server/pom.xml | 6 ++
 3 files changed, 13 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/a99f5d33/jdbc/pom.xml
--
diff --git a/jdbc/pom.xml b/jdbc/pom.xml
index 45c212a..97e73c6 100644
--- a/jdbc/pom.xml
+++ b/jdbc/pom.xml
@@ -47,6 +47,11 @@
 avatica
 
 
+com.google.protobuf
+protobuf-java
+${avatica.protobuf.version}
+
+
 junit
 junit
 test

http://git-wip-us.apache.org/repos/asf/kylin/blob/a99f5d33/pom.xml
--
diff --git a/pom.xml b/pom.xml
index b358bb5..d64a058 100644
--- a/pom.xml
+++ b/pom.xml
@@ -114,6 +114,8 @@
 
 1.11.0
 1.9.0
+
+3.1.0
 
 
 jacoco

http://git-wip-us.apache.org/repos/asf/kylin/blob/a99f5d33/server/pom.xml
--
diff --git a/server/pom.xml b/server/pom.xml
index ed96442..b40883a 100644
--- a/server/pom.xml
+++ b/server/pom.xml
@@ -50,6 +50,12 @@
 
 
 
+com.google.protobuf
+protobuf-java
+${avatica.protobuf.version}
+
+
+
 com.thetransactioncompany
 cors-filter