(incubator-gluten) branch main updated (71d50f697 -> 598727793)

[mahongbin]

This is an automated email from the ASF dual-hosted git repository.

mahongbin pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-gluten.git


from 71d50f697 [GLUTEN-5203][VL] Support url_encode function (#5204)
 add 598727793 [Gluten-5152][CH]Support Optimize and VACUUM command for 
clickhouse tables (#5153)

No new revisions were added by this update.

Summary of changes:
 .../delta/commands/OptimizeTableCommandBase.scala  | 453 
 .../commands/OptimizeTableCommandOverwrites.scala  | 323 ++
 .../spark/sql/delta/commands/VacuumCommand.scala   | 324 ++
 .../sql/delta/commands/OptimizeTableCommand.scala  | 446 
 .../commands/OptimizeTableCommandOverwrites.scala  | 323 ++
 .../spark/sql/delta/commands/VacuumCommand.scala   | 464 +
 .../datasources/CHDatasourceJniWrapper.java|   8 +
 .../datasources/v1/CHMergeTreeWriterInjects.scala  |  12 +-
 .../v1/clickhouse/MergeTreeFileFormatWriter.scala  |  37 --
 .../GlutenClickHouseMergeTreeOptimizeSuite.scala   | 336 +++
 cpp-ch/local-engine/Operator/ExpandTransform.cpp   |   2 +
 cpp-ch/local-engine/Parser/MergeTreeRelParser.cpp  |  11 +-
 cpp-ch/local-engine/Parser/MergeTreeRelParser.h|   4 +-
 .../Storages/CustomStorageMergeTree.cpp|   2 +
 .../local-engine/Storages/CustomStorageMergeTree.h |   4 +
 .../Storages/Mergetree/MergeSparkMergeTreeTask.cpp | 188 +
 .../Storages/Mergetree/MergeSparkMergeTreeTask.h   | 126 ++
 .../Storages/StorageMergeTreeFactory.cpp   |  18 +-
 .../Storages/StorageMergeTreeFactory.h |  10 +
 cpp-ch/local-engine/local_engine_jni.cpp   | 112 +
 20 files changed, 3153 insertions(+), 50 deletions(-)
 create mode 100644 
backends-clickhouse/src/main/delta-20/org/apache/spark/sql/delta/commands/OptimizeTableCommandBase.scala
 create mode 100644 
backends-clickhouse/src/main/delta-20/org/apache/spark/sql/delta/commands/OptimizeTableCommandOverwrites.scala
 create mode 100644 
backends-clickhouse/src/main/delta-20/org/apache/spark/sql/delta/commands/VacuumCommand.scala
 create mode 100644 
backends-clickhouse/src/main/delta-22/org/apache/spark/sql/delta/commands/OptimizeTableCommand.scala
 create mode 100644 
backends-clickhouse/src/main/delta-22/org/apache/spark/sql/delta/commands/OptimizeTableCommandOverwrites.scala
 create mode 100644 
backends-clickhouse/src/main/delta-22/org/apache/spark/sql/delta/commands/VacuumCommand.scala
 create mode 100644 
backends-clickhouse/src/test/scala/io/glutenproject/execution/GlutenClickHouseMergeTreeOptimizeSuite.scala
 create mode 100644 
cpp-ch/local-engine/Storages/Mergetree/MergeSparkMergeTreeTask.cpp
 create mode 100644 
cpp-ch/local-engine/Storages/Mergetree/MergeSparkMergeTreeTask.h


-
To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org
For additional commands, e-mail: commits-h...@gluten.apache.org

(incubator-gluten) branch main updated: [Gluten-5018][CH] support minmax/bloomfilter/set skip index (#5019)

[mahongbin]

This is an automated email from the ASF dual-hosted git repository.

mahongbin pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-gluten.git


The following commit(s) were added to refs/heads/main by this push:
 new 972597184 [Gluten-5018][CH] support minmax/bloomfilter/set skip index 
(#5019)
972597184 is described below

commit 972597184e147fcf488fd6cda4b447356d61136d
Author: Hongbin Ma 
AuthorDate: Wed Mar 27 09:33:40 2024 +0800

[Gluten-5018][CH] support minmax/bloomfilter/set skip index (#5019)

* temp, by defualt all cols minmax index

basically works, dealing with nullable

nullable/not-null ok

remove unneceesary change

fix compile

* add ut

* remove dataschema

* fix spark32 bug
---
 .../source/DeltaMergeTreeFileFormat.scala  |  17 +-
 .../source/DeltaMergeTreeFileFormat.scala  |  17 +-
 .../java/io/glutenproject/metrics/MetricsStep.java |  11 +
 .../backendsapi/clickhouse/CHIteratorApi.scala |   3 +
 .../backendsapi/clickhouse/CHMetricsApi.scala  |   1 +
 .../execution/GlutenMergeTreePartition.scala   |   3 +
 .../metrics/FileSourceScanMetricsUpdater.scala |   2 +
 .../delta/ClickhouseOptimisticTransaction.scala|   7 +-
 .../sql/delta/catalog/ClickHouseTableV2.scala  |  35 ++-
 .../utils/MergeTreePartsPartitionsUtil.scala   |  33 +++
 .../datasources/v1/CHMergeTreeWriterInjects.scala  |  29 ++-
 .../v1/clickhouse/MergeTreeFileFormatWriter.scala  |   9 +
 ...GlutenClickHouseTPCHNotNullSkipIndexSuite.scala | 271 
 ...lutenClickHouseTPCHNullableSkipIndexSuite.scala | 277 +
 .../apache/spark/affinity/MixedAffinitySuite.scala |   3 +
 cpp-ch/local-engine/Common/MergeTreeTool.cpp   |  84 ++-
 cpp-ch/local-engine/Common/MergeTreeTool.h |   3 +
 cpp-ch/local-engine/Parser/MergeTreeRelParser.cpp  |  18 +-
 cpp-ch/local-engine/Parser/RelMetric.cpp   |   3 +
 cpp-ch/local-engine/Parser/TypeParser.cpp  |   4 +-
 cpp-ch/local-engine/Parser/TypeParser.h|  50 ++--
 .../substrait/rel/ExtensionTableBuilder.java   |   6 +
 .../substrait/rel/ExtensionTableNode.java  |  12 +
 .../datasource/GlutenFormatWriterInjects.scala |   4 +-
 24 files changed, 843 insertions(+), 59 deletions(-)

diff --git 
a/backends-clickhouse/src/main/delta-20/org/apache/spark/sql/execution/datasources/v2/clickhouse/source/DeltaMergeTreeFileFormat.scala
 
b/backends-clickhouse/src/main/delta-20/org/apache/spark/sql/execution/datasources/v2/clickhouse/source/DeltaMergeTreeFileFormat.scala
index fef109d35..d4ca321a9 100644
--- 
a/backends-clickhouse/src/main/delta-20/org/apache/spark/sql/execution/datasources/v2/clickhouse/source/DeltaMergeTreeFileFormat.scala
+++ 
b/backends-clickhouse/src/main/delta-20/org/apache/spark/sql/execution/datasources/v2/clickhouse/source/DeltaMergeTreeFileFormat.scala
@@ -17,7 +17,6 @@
 package org.apache.spark.sql.execution.datasources.v2.clickhouse.source
 
 import org.apache.spark.sql.SparkSession
-import org.apache.spark.sql.catalyst.expressions.Attribute
 import org.apache.spark.sql.delta.DeltaParquetFileFormat
 import org.apache.spark.sql.delta.actions.Metadata
 import org.apache.spark.sql.execution.datasources.{OutputWriter, 
OutputWriterFactory}
@@ -31,9 +30,11 @@ class DeltaMergeTreeFileFormat(metadata: Metadata)
 
   protected var database = ""
   protected var tableName = ""
-  protected var dataSchemas = Seq.empty[Attribute]
   protected var orderByKeyOption: Option[Seq[String]] = None
   protected var lowCardKeyOption: Option[Seq[String]] = None
+  protected var minmaxIndexKeyOption: Option[Seq[String]] = None
+  protected var bfIndexKeyOption: Option[Seq[String]] = None
+  protected var setIndexKeyOption: Option[Seq[String]] = None
   protected var primaryKeyOption: Option[Seq[String]] = None
   protected var partitionColumns: Seq[String] = Seq.empty[String]
   protected var clickhouseTableConfigs: Map[String, String] = Map.empty
@@ -42,18 +43,22 @@ class DeltaMergeTreeFileFormat(metadata: Metadata)
   metadata: Metadata,
   database: String,
   tableName: String,
-  schemas: Seq[Attribute],
   orderByKeyOption: Option[Seq[String]],
   lowCardKeyOption: Option[Seq[String]],
+  minmaxIndexKeyOption: Option[Seq[String]],
+  bfIndexKeyOption: Option[Seq[String]],
+  setIndexKeyOption: Option[Seq[String]],
   primaryKeyOption: Option[Seq[String]],
   clickhouseTableConfigs: Map[String, String],
   partitionColumns: Seq[String]) {
 this(metadata)
 this.database = database
 this.tableName = tableName
-this.dataSchemas = schemas
 this.orderByKeyOption = orderByKeyOption
 this.lowCardKeyOption = lowCardKeyOption
+this.minmaxIndexKeyOption = minmaxIndexKeyOption
+this.bfIndexKeyOption = bfIndexKeyOption
+this.setIndexKeyOption

(incubator-gluten) branch main updated: [Gluten-4912][CH] fix bug when a query has no shuffle (#5081)

[mahongbin]

This is an automated email from the ASF dual-hosted git repository.

mahongbin pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-gluten.git


The following commit(s) were added to refs/heads/main by this push:
 new 4a953a222 [Gluten-4912][CH] fix bug when a query has no shuffle (#5081)
4a953a222 is described below

commit 4a953a22214f0d74757d5a412218d93dda03aca6
Author: Hongbin Ma 
AuthorDate: Fri Mar 22 18:18:59 2024 +0800

[Gluten-4912][CH] fix bug when a query has no shuffle (#5081)
---
 .../execution/GlutenClickHouseMergeTreeWriteSuite.scala   | 15 +--
 cpp-ch/local-engine/Parser/CHColumnToSparkRow.cpp | 13 +++--
 cpp-ch/local-engine/Shuffle/ShuffleSplitter.cpp   |  2 +-
 cpp-ch/local-engine/tests/benchmark_local_engine.cpp  |  2 +-
 4 files changed, 22 insertions(+), 10 deletions(-)

diff --git 
a/backends-clickhouse/src/test/scala/io/glutenproject/execution/GlutenClickHouseMergeTreeWriteSuite.scala
 
b/backends-clickhouse/src/test/scala/io/glutenproject/execution/GlutenClickHouseMergeTreeWriteSuite.scala
index 6750e251c..0862fd41a 100644
--- 
a/backends-clickhouse/src/test/scala/io/glutenproject/execution/GlutenClickHouseMergeTreeWriteSuite.scala
+++ 
b/backends-clickhouse/src/test/scala/io/glutenproject/execution/GlutenClickHouseMergeTreeWriteSuite.scala
@@ -1239,7 +1239,7 @@ class GlutenClickHouseMergeTreeWriteSuite
  |)
  |USING clickhouse
  |LOCATION '$basePath/lineitem_mergetree_lowcard'
- |TBLPROPERTIES('lowCardKey'='l_returnflag,L_LINESTATUS')
+ 
|TBLPROPERTIES('lowCardKey'='l_returnflag,L_LINESTATUS,l_quantity')
  |""".stripMargin)
 
 spark.sql(s"""
@@ -1285,7 +1285,7 @@ class GlutenClickHouseMergeTreeWriteSuite
 val sqlStr2 =
   s"""
  |SELECT
- |  max(l_returnflag)
+ |  max(l_returnflag), min(l_quantity)
  |FROM
  |lineitem_mergetree_lowcard
  |GROUP BY
@@ -1298,6 +1298,17 @@ class GlutenClickHouseMergeTreeWriteSuite
   // total rows should remain unchanged
   spark.sql(sqlStr2).collect().apply(0).get(0) == "R"
 )
+
+// test select *
+val sqlStr3 =
+  s"""
+ |SELECT
+ |  *
+ |FROM
+ |lineitem_mergetree_lowcard limit 1
+ |
+ |""".stripMargin
+spark.sql(sqlStr3).collect()
   }
 
   test("test mergetree with primary keys filter") {
diff --git a/cpp-ch/local-engine/Parser/CHColumnToSparkRow.cpp 
b/cpp-ch/local-engine/Parser/CHColumnToSparkRow.cpp
index c3ff32a97..cb5f0111c 100644
--- a/cpp-ch/local-engine/Parser/CHColumnToSparkRow.cpp
+++ b/cpp-ch/local-engine/Parser/CHColumnToSparkRow.cpp
@@ -325,12 +325,12 @@ SparkRowInfo::SparkRowInfo(
 {
 const auto & col = cols[col_idx];
 /// No need to calculate backing data length for fixed length types
-const auto type_without_nullable = removeNullable(col.type);
+const auto type_without_nullable = 
removeLowCardinalityAndNullable(col.type);
 if 
(BackingDataLengthCalculator::isVariableLengthDataType(type_without_nullable))
 {
 if 
(BackingDataLengthCalculator::isDataTypeSupportRawData(type_without_nullable))
 {
-auto column = col.column->convertToFullColumnIfConst();
+auto column = col.column->convertToFullIfNeeded();
 const auto * nullable_column = 
checkAndGetColumn(*column);
 if (nullable_column)
 {
@@ -348,13 +348,13 @@ SparkRowInfo::SparkRowInfo(
 for (size_t i = 0; i < num_rows; ++i)
 {
 size_t row_idx = masks == nullptr ? i : masks->at(i);
-lengths[i] += 
roundNumberOfBytesToNearestWord(col.column->getDataAt(row_idx).size);
+lengths[i] += 
roundNumberOfBytesToNearestWord(column->getDataAt(row_idx).size);
 }
 }
 }
 else
 {
-BackingDataLengthCalculator calculator(col.type);
+BackingDataLengthCalculator calculator(type_without_nullable);
 for (size_t i = 0; i < num_rows; ++i)
 {
 size_t row_idx = masks == nullptr ? i : masks->at(i);
@@ -462,11 +462,12 @@ std::unique_ptr 
CHColumnToSparkRow::convertCHColumnToSparkRow(cons
 const auto & col = block.getByPosition(col_idx);
 int64_t field_offset = spark_row_info->getFieldOffset(col_idx);
 
-ColumnWithTypeAndName 
col_not_const{col.column->convertToFullColumnIfConst(), col.type, col.name};
+ColumnWithTypeAndName col_full{col.column->convertToFullIfNeeded(),
+re

(incubator-gluten) branch main updated: [GLUTEN-5060][CH] Remove unnecessary FilterExec execution when querying from MergeTree with the prewhere (#5067)

[mahongbin]

This is an automated email from the ASF dual-hosted git repository.

mahongbin pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-gluten.git


The following commit(s) were added to refs/heads/main by this push:
 new fa0079dca [GLUTEN-5060][CH] Remove unnecessary FilterExec execution 
when querying from MergeTree with the prewhere (#5067)
fa0079dca is described below

commit fa0079dca26e755ad4760839d4230885baf3ce5a
Author: Zhichao Zhang 
AuthorDate: Thu Mar 21 16:32:41 2024 +0800

[GLUTEN-5060][CH] Remove unnecessary FilterExec execution when querying 
from MergeTree with the prewhere (#5067)

When querying from MergeTree with the prewhere, all the filters will be 
pushdowned to the ScanExec, so it does not need to execute the FilterExec again.

Close #5060.
---
 .../backendsapi/clickhouse/CHSparkPlanExecApi.scala   | 19 +--
 .../extension/FallbackBroadcaseHashJoinRules.scala|  3 ++-
 .../GlutenClickHouseMergeTreeWriteSuite.scala | 16 +---
 3 files changed, 28 insertions(+), 10 deletions(-)

diff --git 
a/backends-clickhouse/src/main/scala/io/glutenproject/backendsapi/clickhouse/CHSparkPlanExecApi.scala
 
b/backends-clickhouse/src/main/scala/io/glutenproject/backendsapi/clickhouse/CHSparkPlanExecApi.scala
index cea88266e..29af5a0e5 100644
--- 
a/backends-clickhouse/src/main/scala/io/glutenproject/backendsapi/clickhouse/CHSparkPlanExecApi.scala
+++ 
b/backends-clickhouse/src/main/scala/io/glutenproject/backendsapi/clickhouse/CHSparkPlanExecApi.scala
@@ -22,8 +22,7 @@ import io.glutenproject.exception.GlutenNotSupportException
 import io.glutenproject.execution._
 import io.glutenproject.expression._
 import io.glutenproject.expression.ConverterUtils.FunctionConfig
-import io.glutenproject.extension.{FallbackBroadcastHashJoin, 
FallbackBroadcastHashJoinPrepQueryStage}
-import io.glutenproject.extension.CountDistinctWithoutExpand
+import io.glutenproject.extension.{CountDistinctWithoutExpand, 
FallbackBroadcastHashJoin, FallbackBroadcastHashJoinPrepQueryStage}
 import io.glutenproject.extension.columnar.AddTransformHintRule
 import 
io.glutenproject.extension.columnar.MiscColumnarRules.TransformPreOverrides
 import io.glutenproject.substrait.expression.{ExpressionBuilder, 
ExpressionNode, WindowFunctionNode}
@@ -48,7 +47,7 @@ import org.apache.spark.sql.catalyst.rules.Rule
 import org.apache.spark.sql.delta.files.TahoeFileIndex
 import org.apache.spark.sql.execution._
 import org.apache.spark.sql.execution.adaptive.AQEShuffleReadExec
-import org.apache.spark.sql.execution.datasources.{FileFormat, WriteFilesExec}
+import org.apache.spark.sql.execution.datasources.{FileFormat, 
HadoopFsRelation, WriteFilesExec}
 import 
org.apache.spark.sql.execution.datasources.GlutenWriterColumnarRules.NativeWritePostRule
 import org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat
 import 
org.apache.spark.sql.execution.datasources.v2.clickhouse.source.DeltaMergeTreeFileFormat
@@ -121,10 +120,18 @@ class CHSparkPlanExecApi extends SparkPlanExecApi {
   override def genFilterExecTransformer(
   condition: Expression,
   child: SparkPlan): FilterExecTransformerBase = {
+
+def checkMergeTreeFileFormat(relation: HadoopFsRelation): Boolean = {
+  relation.location.isInstanceOf[TahoeFileIndex] &&
+  relation.fileFormat.isInstanceOf[DeltaMergeTreeFileFormat]
+}
+
 child match {
-  case scan: FileSourceScanExec
-  if (scan.relation.location.isInstanceOf[TahoeFileIndex] &&
-scan.relation.fileFormat.isInstanceOf[DeltaMergeTreeFileFormat]) =>
+  case scan: FileSourceScanExec if 
(checkMergeTreeFileFormat(scan.relation)) =>
+// For the validation phase of the AddTransformHintRule
+CHFilterExecTransformer(condition, child)
+  case scan: FileSourceScanExecTransformerBase if 
(checkMergeTreeFileFormat(scan.relation)) =>
+// For the transform phase, the FileSourceScanExec is already 
transformed
 CHFilterExecTransformer(condition, child)
   case _ =>
 FilterExecTransformer(condition, child)
diff --git 
a/backends-clickhouse/src/main/java/io/glutenproject/extension/FallbackBroadcaseHashJoinRules.scala
 
b/backends-clickhouse/src/main/scala/io/glutenproject/extension/FallbackBroadcaseHashJoinRules.scala
similarity index 98%
rename from 
backends-clickhouse/src/main/java/io/glutenproject/extension/FallbackBroadcaseHashJoinRules.scala
rename to 
backends-clickhouse/src/main/scala/io/glutenproject/extension/FallbackBroadcaseHashJoinRules.scala
index ccfa16501..52ec8ce47 100644
--- 
a/backends-clickhouse/src/main/java/io/glutenproject/extension/FallbackBroadcaseHashJoinRules.scala
+++ 
b/backends-clickhouse/src/main/scala/io/glutenproject/extension/FallbackBroadcaseHashJoinRules.scala
@@ -100,7 +100,8 @@ case class FallbackBroadcastHashJoinPrepQueryStage(session: 
SparkSession) extend
 case cla

(incubator-gluten) branch main updated: [GLUTEN-5061][CH] Fix assert error when writing mergetree data with select * from table limit n (#5068)

[mahongbin]

This is an automated email from the ASF dual-hosted git repository.

mahongbin pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-gluten.git


The following commit(s) were added to refs/heads/main by this push:
 new 98015999e [GLUTEN-5061][CH] Fix assert error when writing mergetree 
data with select * from table limit n (#5068)
98015999e is described below

commit 98015999ee093a0db6bbf151c354e13ba6637bda
Author: Zhichao Zhang 
AuthorDate: Thu Mar 21 16:31:10 2024 +0800

[GLUTEN-5061][CH] Fix assert error when writing mergetree data with select 
* from table limit n (#5068)

The below writing mergetree data sql will throw the assert error:
```
insert into table lineitem_mergetree
select * from lineitem **limit 10**
```

RC:
with limit n, there is a wrong wrapper logical with the FakeRowAdaptor for 
the query plan.

Close #5061.
---
 .../delta/ClickhouseOptimisticTransaction.scala|  2 +-
 .../GlutenClickHouseMergeTreeWriteSuite.scala  | 63 ++
 2 files changed, 64 insertions(+), 1 deletion(-)

diff --git 
a/backends-clickhouse/src/main/scala/org/apache/spark/sql/delta/ClickhouseOptimisticTransaction.scala
 
b/backends-clickhouse/src/main/scala/org/apache/spark/sql/delta/ClickhouseOptimisticTransaction.scala
index e4786168e..9111bea7f 100644
--- 
a/backends-clickhouse/src/main/scala/org/apache/spark/sql/delta/ClickhouseOptimisticTransaction.scala
+++ 
b/backends-clickhouse/src/main/scala/org/apache/spark/sql/delta/ClickhouseOptimisticTransaction.scala
@@ -94,7 +94,7 @@ class ClickhouseOptimisticTransaction(
   aqe.isSubquery,
   supportsColumnar = true
 ))
-case other => queryPlan.withNewChildren(Array(FakeRowAdaptor(other)))
+case other => FakeRowAdaptor(other)
   }
 
   val statsTrackers: ListBuffer[WriteJobStatsTracker] = ListBuffer()
diff --git 
a/backends-clickhouse/src/test/scala/io/glutenproject/execution/GlutenClickHouseMergeTreeWriteSuite.scala
 
b/backends-clickhouse/src/test/scala/io/glutenproject/execution/GlutenClickHouseMergeTreeWriteSuite.scala
index f8aa2cfa4..ca192cb89 100644
--- 
a/backends-clickhouse/src/test/scala/io/glutenproject/execution/GlutenClickHouseMergeTreeWriteSuite.scala
+++ 
b/backends-clickhouse/src/test/scala/io/glutenproject/execution/GlutenClickHouseMergeTreeWriteSuite.scala
@@ -1379,5 +1379,68 @@ class GlutenClickHouseMergeTreeWriteSuite
 assert(plans(0).metrics("totalMarksPk").value === 74)
 }
   }
+
+  test(
+"GLUTEN-5061: Fix assert error when writing mergetree data with select * 
from table limit n") {
+spark.sql(s"""
+ |DROP TABLE IF EXISTS lineitem_mergetree_5061;
+ |""".stripMargin)
+
+spark.sql(s"""
+ |CREATE TABLE IF NOT EXISTS lineitem_mergetree_5061
+ |(
+ | l_orderkey  bigint,
+ | l_partkey   bigint,
+ | l_suppkey   bigint,
+ | l_linenumberbigint,
+ | l_quantity  double,
+ | l_extendedprice double,
+ | l_discount  double,
+ | l_tax   double,
+ | l_returnflagstring,
+ | l_linestatusstring,
+ | l_shipdate  date,
+ | l_commitdatedate,
+ | l_receiptdate   date,
+ | l_shipinstruct  string,
+ | l_shipmode  string,
+ | l_comment   string
+ |)
+ |USING clickhouse
+ |LOCATION '$basePath/lineitem_mergetree_5061'
+ |""".stripMargin)
+
+spark.sql(s"""
+ | insert into table lineitem_mergetree_5061
+ | select * from lineitem limit 10
+ |""".stripMargin)
+
+val sqlStr =
+  s"""
+ |SELECT
+ |count(1)
+ |FROM
+ |lineitem_mergetree_5061
+ |""".stripMargin
+runSql(sqlStr)(
+  df => {
+val result = df.collect()
+assert(result.size == 1)
+assert(result(0).getLong(0) == 10)
+
+val scanExec = collect(df.queryExecution.executedPlan) {
+  case f: FileSourceScanExecTransformer => f
+}
+assert(scanExec.size == 1)
+
+val mergetreeScan = scanExec(0)
+assert(mergetreeScan.nodeName.startsWith("Scan mergetree"))
+
+val fileIndex = 
mergetreeScan.relation.location.asInstanceOf[TahoeFileIndex]
+val addFiles = fileIndex.matchingFiles(Nil, Nil).map(f => 
f.asInstanceOf[AddMergeTreeParts])
+assert(addFiles.size == 1)
+assert(addFiles(0).rows == 10)
+  })
+  }
 }
 // scalastyle:off

(incubator-gluten) branch main updated (d762deb9b -> e1f0c01a7)

[mahongbin]

This is an automated email from the ASF dual-hosted git repository.

mahongbin pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-gluten.git


from d762deb9b [GLUTEN-5041][CH] Fix primary not used when query with 
filter (#5045)
 add e1f0c01a7 [GLUTEN-4989][CH] Support function timestamp_add (#5012)

No new revisions were added by this update.

Summary of changes:
 cpp-ch/clickhouse.version  |   2 +-
 .../local-engine/Parser/SerializedPlanParser.cpp   | 168 ++---
 cpp-ch/local-engine/Parser/SerializedPlanParser.h  |   2 +
 .../Parser/scalar_function_parser/timestampAdd.cpp | 102 +
 .../expression/ExpressionConverter.scala   |  17 +++
 ...ansformer.scala => TimestampAddTransform.scala} |  35 +++--
 .../utils/velox/VeloxTestSettings.scala|   1 +
 .../expressions/GlutenDateExpressionsSuite.scala   |  22 +++
 .../glutenproject/expression/ExpressionNames.scala |   1 +
 .../io/glutenproject/sql/shims/SparkShims.scala|   3 +
 .../sql/shims/spark33/Spark33Shims.scala   |  18 ++-
 11 files changed, 268 insertions(+), 103 deletions(-)
 create mode 100644 
cpp-ch/local-engine/Parser/scalar_function_parser/timestampAdd.cpp
 copy 
gluten-core/src/main/scala/io/glutenproject/expression/{GenericExpressionTransformer.scala
 => TimestampAddTransform.scala} (60%)


-
To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org
For additional commands, e-mail: commits-h...@gluten.apache.org

(incubator-gluten) branch main updated: [Gluten-4706] Add a mode to execute count distinct directly instead of Expand+Count (#4708)

[mahongbin]

This is an automated email from the ASF dual-hosted git repository.

mahongbin pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-gluten.git


The following commit(s) were added to refs/heads/main by this push:
 new b07b36960 [Gluten-4706] Add a mode to execute count distinct directly 
instead of Expand+Count (#4708)
b07b36960 is described below

commit b07b36960238db62647b6aeefc930b7fc3f271c7
Author: Hongbin Ma 
AuthorDate: Tue Mar 12 16:38:47 2024 +0800

[Gluten-4706] Add a mode to execute count distinct directly instead of 
Expand+Count (#4708)
---
 .../clickhouse/CHSparkPlanExecApi.scala|   6 +-
 .../execution/CHHashAggregateExecTransformer.scala |   6 ++
 .../execution/GlutenClickHouseHiveTableSuite.scala |  71 +++--
 .../GlutenClickHouseNativeWriteTableSuite.scala|  38 +--
 ...lutenClickHouseWholeStageTransformerSuite.scala |  17 ++-
 .../GlutenClickhouseCountDistinctSuite.scala   | 118 +
 .../extension/CustomAggExpressionTransformer.scala |   2 +-
 .../AggregateFunctionPartialMerge.h|   4 +-
 .../local-engine/Parser/AggregateFunctionParser.h  |  14 ++-
 .../CommonAggregateFunctionParser.cpp  |   2 +-
 .../expression/ExpressionMappings.scala|   1 +
 .../extension/CountDistinctWithoutExpand.scala |  49 +
 .../expressions/aggregate/CountDistinct.scala  |  59 +++
 .../main/scala/io/glutenproject/GlutenConfig.scala |  13 +++
 .../glutenproject/expression/ExpressionNames.scala |   1 +
 15 files changed, 322 insertions(+), 79 deletions(-)

diff --git 
a/backends-clickhouse/src/main/scala/io/glutenproject/backendsapi/clickhouse/CHSparkPlanExecApi.scala
 
b/backends-clickhouse/src/main/scala/io/glutenproject/backendsapi/clickhouse/CHSparkPlanExecApi.scala
index 43d8ed1bc..f32116728 100644
--- 
a/backends-clickhouse/src/main/scala/io/glutenproject/backendsapi/clickhouse/CHSparkPlanExecApi.scala
+++ 
b/backends-clickhouse/src/main/scala/io/glutenproject/backendsapi/clickhouse/CHSparkPlanExecApi.scala
@@ -22,6 +22,7 @@ import io.glutenproject.execution._
 import io.glutenproject.expression._
 import io.glutenproject.expression.ConverterUtils.FunctionConfig
 import io.glutenproject.extension.{FallbackBroadcastHashJoin, 
FallbackBroadcastHashJoinPrepQueryStage}
+import io.glutenproject.extension.CountDistinctWithoutExpand
 import io.glutenproject.extension.columnar.AddTransformHintRule
 import 
io.glutenproject.extension.columnar.MiscColumnarRules.TransformPreOverrides
 import io.glutenproject.substrait.expression.{ExpressionBuilder, 
ExpressionNode, WindowFunctionNode}
@@ -503,7 +504,10 @@ class CHSparkPlanExecApi extends SparkPlanExecApi {
* @return
*/
   override def genExtendedOptimizers(): List[SparkSession => 
Rule[LogicalPlan]] = {
-List(spark => new CommonSubexpressionEliminateRule(spark, 
spark.sessionState.conf))
+List(
+  spark => new CommonSubexpressionEliminateRule(spark, 
spark.sessionState.conf),
+  _ => CountDistinctWithoutExpand
+)
   }
 
   /**
diff --git 
a/backends-clickhouse/src/main/scala/io/glutenproject/execution/CHHashAggregateExecTransformer.scala
 
b/backends-clickhouse/src/main/scala/io/glutenproject/execution/CHHashAggregateExecTransformer.scala
index c6b99a6bc..85b967446 100644
--- 
a/backends-clickhouse/src/main/scala/io/glutenproject/execution/CHHashAggregateExecTransformer.scala
+++ 
b/backends-clickhouse/src/main/scala/io/glutenproject/execution/CHHashAggregateExecTransformer.scala
@@ -358,6 +358,12 @@ case class CHHashAggregateExecTransformer(
   (makeStructType(fields), attr.nullable)
 case expr if "bloom_filter_agg".equals(expr.prettyName) =>
   (makeStructTypeSingleOne(expr.children.head.dataType, 
attr.nullable), attr.nullable)
+case cd: CountDistinct =>
+  var fields = Seq[(DataType, Boolean)]()
+  for (child <- cd.children) {
+fields = fields :+ (child.dataType, child.nullable)
+  }
+  (makeStructType(fields), false)
 case _ =>
   (makeStructTypeSingleOne(attr.dataType, attr.nullable), 
attr.nullable)
   }
diff --git 
a/backends-clickhouse/src/test/scala/io/glutenproject/execution/GlutenClickHouseHiveTableSuite.scala
 
b/backends-clickhouse/src/test/scala/io/glutenproject/execution/GlutenClickHouseHiveTableSuite.scala
index 938f8e6d1..b40a0fe0d 100644
--- 
a/backends-clickhouse/src/test/scala/io/glutenproject/execution/GlutenClickHouseHiveTableSuite.scala
+++ 
b/backends-clickhouse/src/test/scala/io/glutenproject/execution/GlutenClickHouseHiveTableSuite.scala
@@ -19,8 +19,10 @@ package io.glutenproject.execution
 import io.glutenproject.GlutenConfig
 import io.glutenproject.utils.UTSystemParameters
 
-import org.apache.spark.{SPARK_VERSION_SHORT, SparkConf}
-import org.apache.spark.sql

(incubator-gluten) branch main updated: [Gluten-4912][CH]Support Specifying columns in clickhouse tables to be Low Cardinality (#4925)

[mahongbin]

This is an automated email from the ASF dual-hosted git repository.

mahongbin pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-gluten.git


The following commit(s) were added to refs/heads/main by this push:
 new 7ff18ee85 [Gluten-4912][CH]Support Specifying columns in clickhouse 
tables to be Low Cardinality (#4925)
7ff18ee85 is described below

commit 7ff18ee85d2c66d3ca8c3936bea457070fb450ac
Author: Hongbin Ma 
AuthorDate: Tue Mar 12 15:30:44 2024 +0800

[Gluten-4912][CH]Support Specifying columns in clickhouse tables to be Low 
Cardinality (#4925)
---
 .../source/DeltaMergeTreeFileFormat.scala  |   4 +
 .../source/DeltaMergeTreeFileFormat.scala  |   4 +
 .../backendsapi/clickhouse/CHIteratorApi.scala |   1 +
 .../execution/GlutenMergeTreePartition.scala   |   1 +
 .../delta/ClickhouseOptimisticTransaction.scala|   2 +
 .../sql/delta/catalog/ClickHouseTableV2.scala  |  22 +++
 .../utils/MergeTreePartsPartitionsUtil.scala   |  11 ++
 .../datasources/v1/CHMergeTreeWriterInjects.scala  |   9 +
 .../v1/clickhouse/MergeTreeFileFormatWriter.scala  |   3 +
 .../GlutenClickHouseMergeTreeWriteSuite.scala  |  90 ++
 .../apache/spark/affinity/MixedAffinitySuite.scala |   1 +
 .../local-engine/Builder/SerializedPlanBuilder.cpp |   8 +-
 cpp-ch/local-engine/Common/MergeTreeTool.cpp   |   2 +
 cpp-ch/local-engine/Common/MergeTreeTool.h |   1 +
 cpp-ch/local-engine/Parser/MergeTreeRelParser.cpp  |   6 +-
 .../local-engine/Parser/SerializedPlanParser.cpp   | 190 +++--
 cpp-ch/local-engine/Parser/TypeParser.cpp  |  24 ++-
 cpp-ch/local-engine/Parser/TypeParser.h|   3 +-
 cpp-ch/local-engine/Shuffle/ShuffleSplitter.cpp|   4 +-
 cpp-ch/local-engine/Storages/IO/NativeWriter.cpp   |   5 +-
 .../substrait/rel/ExtensionTableBuilder.java   |   2 +
 .../substrait/rel/ExtensionTableNode.java  |   5 +
 .../datasource/GlutenFormatWriterInjects.scala |   1 +
 23 files changed, 328 insertions(+), 71 deletions(-)

diff --git 
a/backends-clickhouse/src/main/delta-20/org/apache/spark/sql/execution/datasources/v2/clickhouse/source/DeltaMergeTreeFileFormat.scala
 
b/backends-clickhouse/src/main/delta-20/org/apache/spark/sql/execution/datasources/v2/clickhouse/source/DeltaMergeTreeFileFormat.scala
index 09f17c468..fef109d35 100644
--- 
a/backends-clickhouse/src/main/delta-20/org/apache/spark/sql/execution/datasources/v2/clickhouse/source/DeltaMergeTreeFileFormat.scala
+++ 
b/backends-clickhouse/src/main/delta-20/org/apache/spark/sql/execution/datasources/v2/clickhouse/source/DeltaMergeTreeFileFormat.scala
@@ -33,6 +33,7 @@ class DeltaMergeTreeFileFormat(metadata: Metadata)
   protected var tableName = ""
   protected var dataSchemas = Seq.empty[Attribute]
   protected var orderByKeyOption: Option[Seq[String]] = None
+  protected var lowCardKeyOption: Option[Seq[String]] = None
   protected var primaryKeyOption: Option[Seq[String]] = None
   protected var partitionColumns: Seq[String] = Seq.empty[String]
   protected var clickhouseTableConfigs: Map[String, String] = Map.empty
@@ -43,6 +44,7 @@ class DeltaMergeTreeFileFormat(metadata: Metadata)
   tableName: String,
   schemas: Seq[Attribute],
   orderByKeyOption: Option[Seq[String]],
+  lowCardKeyOption: Option[Seq[String]],
   primaryKeyOption: Option[Seq[String]],
   clickhouseTableConfigs: Map[String, String],
   partitionColumns: Seq[String]) {
@@ -51,6 +53,7 @@ class DeltaMergeTreeFileFormat(metadata: Metadata)
 this.tableName = tableName
 this.dataSchemas = schemas
 this.orderByKeyOption = orderByKeyOption
+this.lowCardKeyOption = lowCardKeyOption
 this.primaryKeyOption = primaryKeyOption
 this.clickhouseTableConfigs = clickhouseTableConfigs
 this.partitionColumns = partitionColumns
@@ -98,6 +101,7 @@ class DeltaMergeTreeFileFormat(metadata: Metadata)
 database,
 tableName,
 orderByKeyOption,
+lowCardKeyOption,
 primaryKeyOption,
 partitionColumns,
 metadata.schema,
diff --git 
a/backends-clickhouse/src/main/delta-22/org/apache/spark/sql/execution/datasources/v2/clickhouse/source/DeltaMergeTreeFileFormat.scala
 
b/backends-clickhouse/src/main/delta-22/org/apache/spark/sql/execution/datasources/v2/clickhouse/source/DeltaMergeTreeFileFormat.scala
index 1dd341d2e..b87420787 100644
--- 
a/backends-clickhouse/src/main/delta-22/org/apache/spark/sql/execution/datasources/v2/clickhouse/source/DeltaMergeTreeFileFormat.scala
+++ 
b/backends-clickhouse/src/main/delta-22/org/apache/spark/sql/execution/datasources/v2/clickhouse/source/DeltaMergeTreeFileFormat.scala
@@ -32,6 +32,7 @@ class DeltaMergeTreeFileFormat(metadata: Metadata) extends 
DeltaParquetFileForma
   protected var tableName = ""
   protected var dataSchemas = Seq.empty[Attribute]
   protected var orderByK

[8/8] kylin git commit: KYLIN-1672 support kylin on cdh 5.7

[mahongbin]

KYLIN-1672 support kylin on cdh 5.7

Signed-off-by: Li Yang 


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/b0c90d4a
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/b0c90d4a
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/b0c90d4a

Branch: refs/heads/yang22-cdh5.7
Commit: b0c90d4af2ceaa3dd9549c196089d3b0d9b919f4
Parents: 7f82104
Author: Lynne Jiang 
Authored: Mon May 16 03:33:27 2016 -0700
Committer: Hongbin Ma 
Committed: Mon Dec 19 10:14:07 2016 +0800

--
 dev-support/test_all_against_hdp_2_2_4_2_2.sh   |   0
 .../kylin/engine/mr/steps/MockupMapContext.java |  15 +-
 examples/test_case_data/sandbox/core-site.xml   | 146 +++---
 examples/test_case_data/sandbox/hbase-site.xml  | 162 ++
 examples/test_case_data/sandbox/hdfs-site.xml   | 259 ++
 examples/test_case_data/sandbox/mapred-site.xml | 398 ++-
 examples/test_case_data/sandbox/yarn-site.xml   | 496 ++-
 pom.xml |  16 +-
 server/pom.xml  |  36 ++
 .../storage/hbase/steps/MockupMapContext.java   |  19 +-
 tool/pom.xml|  12 +
 11 files changed, 428 insertions(+), 1131 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/b0c90d4a/dev-support/test_all_against_hdp_2_2_4_2_2.sh
--
diff --git a/dev-support/test_all_against_hdp_2_2_4_2_2.sh 
b/dev-support/test_all_against_hdp_2_2_4_2_2.sh
old mode 100644
new mode 100755

http://git-wip-us.apache.org/repos/asf/kylin/blob/b0c90d4a/engine-mr/src/test/java/org/apache/kylin/engine/mr/steps/MockupMapContext.java
--
diff --git 
a/engine-mr/src/test/java/org/apache/kylin/engine/mr/steps/MockupMapContext.java
 
b/engine-mr/src/test/java/org/apache/kylin/engine/mr/steps/MockupMapContext.java
index 847071d..9900465 100644
--- 
a/engine-mr/src/test/java/org/apache/kylin/engine/mr/steps/MockupMapContext.java
+++ 
b/engine-mr/src/test/java/org/apache/kylin/engine/mr/steps/MockupMapContext.java
@@ -77,6 +77,7 @@ public class MockupMapContext {
 outKV[0] = key;
 outKV[1] = value;
 }
+
 }
 
 @Override
@@ -99,6 +100,7 @@ public class MockupMapContext {
 throw new NotImplementedException();
 }
 
+
 @Override
 public float getProgress() {
 throw new NotImplementedException();
@@ -195,17 +197,17 @@ public class MockupMapContext {
 }
 
 @Override
-public RawComparator getSortComparator() {
+public boolean userClassesTakesPrecedence() {
 throw new NotImplementedException();
 }
 
 @Override
-public String getJar() {
+public RawComparator getSortComparator() {
 throw new NotImplementedException();
 }
 
 @Override
-public RawComparator getGroupingComparator() {
+public String getJar() {
 throw new NotImplementedException();
 }
 
@@ -221,7 +223,7 @@ public class MockupMapContext {
 
 @Override
 public boolean getProfileEnabled() {
-throw new NotImplementedException();
+return false;
 }
 
 @Override
@@ -308,6 +310,11 @@ public class MockupMapContext {
 public RawComparator getCombinerKeyGroupingComparator() {
 throw new NotImplementedException();
 }
+
+@Override
+public RawComparator getGroupingComparator() {
+return null;
+}
 });
 }
 }

http://git-wip-us.apache.org/repos/asf/kylin/blob/b0c90d4a/examples/test_case_data/sandbox/core-site.xml
--
diff --git a/examples/test_case_data/sandbox/core-site.xml 
b/examples/test_case_data/sandbox/core-site.xml
index 9aa588c..6162406 100644
--- a/examples/test_case_data/sandbox/core-site.xml
+++ b/examples/test_case_data/sandbox/core-site.xml
@@ -14,152 +14,146 @@
   See the License for the specific language governing permissions and
   limitations under the License.
 -->
+
 
-
 
 fs.defaultFS
-hdfs://sandbox.hortonworks.com:8020
-true
+hdfs://quickstart.cloudera:8020
 
-
 
 fs.trash.interval
-360
+1
 
-
 
-
ha.failover-controller.active-standby-elector.zk.op.retries
-120
+io.compression.codecs
+

[5/8] kylin git commit: KYLIN-2291 Collect hive table property skip-header-line-count

[mahongbin]

KYLIN-2291 Collect hive table property skip-header-line-count

Signed-off-by: Li Yang 


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/bcf2d0ef
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/bcf2d0ef
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/bcf2d0ef

Branch: refs/heads/yang22-cdh5.7
Commit: bcf2d0ef25013306798c54bb4e485bdfbe42f35c
Parents: 82f6c55
Author: Cheng Wang 
Authored: Fri Dec 16 17:23:15 2016 +0800
Committer: Li Yang 
Committed: Fri Dec 16 17:52:26 2016 +0800

--
 .../org/apache/kylin/source/hive/BeelineHiveClient.java  |  3 +++
 .../java/org/apache/kylin/source/hive/CLIHiveClient.java |  2 +-
 .../apache/kylin/source/hive/HiveSourceTableLoader.java  |  1 +
 .../java/org/apache/kylin/source/hive/HiveTableMeta.java |  4 +++-
 .../apache/kylin/source/hive/HiveTableMetaBuilder.java   | 11 ++-
 5 files changed, 18 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/bcf2d0ef/source-hive/src/main/java/org/apache/kylin/source/hive/BeelineHiveClient.java
--
diff --git 
a/source-hive/src/main/java/org/apache/kylin/source/hive/BeelineHiveClient.java 
b/source-hive/src/main/java/org/apache/kylin/source/hive/BeelineHiveClient.java
index 47b551b..468ccb1 100644
--- 
a/source-hive/src/main/java/org/apache/kylin/source/hive/BeelineHiveClient.java
+++ 
b/source-hive/src/main/java/org/apache/kylin/source/hive/BeelineHiveClient.java
@@ -193,6 +193,9 @@ public class BeelineHiveClient implements IHiveClient {
 if ("numFiles".equals(resultSet.getString(2).trim())) {
 
builder.setFileNum(Long.parseLong(resultSet.getString(3).trim()));
 }
+if 
("skip.header.line.count".equals(resultSet.getString(2).trim())) {
+
builder.setSkipHeaderLineCount(resultSet.getString(3).trim());
+}
 }
 }
 if ("InputFormat:".equals(resultSet.getString(1).trim())) {

http://git-wip-us.apache.org/repos/asf/kylin/blob/bcf2d0ef/source-hive/src/main/java/org/apache/kylin/source/hive/CLIHiveClient.java
--
diff --git 
a/source-hive/src/main/java/org/apache/kylin/source/hive/CLIHiveClient.java 
b/source-hive/src/main/java/org/apache/kylin/source/hive/CLIHiveClient.java
index 5a17f1f..e8a93bd 100644
--- a/source-hive/src/main/java/org/apache/kylin/source/hive/CLIHiveClient.java
+++ b/source-hive/src/main/java/org/apache/kylin/source/hive/CLIHiveClient.java
@@ -42,7 +42,6 @@ import com.google.common.collect.Lists;
  *
  */
 public class CLIHiveClient implements IHiveClient {
-
 protected HiveConf hiveConf = null;
 protected Driver driver = null;
 protected HiveMetaStoreClient metaStoreClient = null;
@@ -109,6 +108,7 @@ public class CLIHiveClient implements IHiveClient {
 builder.setOwner(table.getOwner());
 builder.setLastAccessTime(table.getLastAccessTime());
 builder.setTableType(table.getTableType());
+
builder.setSkipHeaderLineCount(table.getParameters().get("skip.header.line.count"));
 
 return builder.createHiveTableMeta();
 }

http://git-wip-us.apache.org/repos/asf/kylin/blob/bcf2d0ef/source-hive/src/main/java/org/apache/kylin/source/hive/HiveSourceTableLoader.java
--
diff --git 
a/source-hive/src/main/java/org/apache/kylin/source/hive/HiveSourceTableLoader.java
 
b/source-hive/src/main/java/org/apache/kylin/source/hive/HiveSourceTableLoader.java
index 57292dc..77e1084 100644
--- 
a/source-hive/src/main/java/org/apache/kylin/source/hive/HiveSourceTableLoader.java
+++ 
b/source-hive/src/main/java/org/apache/kylin/source/hive/HiveSourceTableLoader.java
@@ -140,6 +140,7 @@ public class HiveSourceTableLoader {
 tableExtDesc.addDataSourceProp("total_file_number", 
String.valueOf(hiveTableMeta.fileNum));
 tableExtDesc.addDataSourceProp("hive_inputFormat", 
hiveTableMeta.sdInputFormat);
 tableExtDesc.addDataSourceProp("hive_outputFormat", 
hiveTableMeta.sdOutputFormat);
+tableExtDesc.addDataSourceProp("skip_header_line_count", 
String.valueOf(hiveTableMeta.skipHeaderLineCount));
 
 metaMgr.saveTableExt(tableExtDesc);
 metaMgr.saveSourceTable(tableDesc);

http://git-wip-us.apache.org/repos/asf/kylin/blob/bcf2d0ef/source-hive/src/main/java/org/apache/kylin/source/hive/HiveTableMeta.java
--
diff --git 

[1/8] kylin git commit: KYLIN-2290 minor improvements on limit [Forced Update!]

[mahongbin]

Repository: kylin
Updated Branches:
  refs/heads/yang22-cdh5.7 f11fecb5c -> b0c90d4af (forced update)


KYLIN-2290 minor improvements on limit


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/cdd945cb
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/cdd945cb
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/cdd945cb

Branch: refs/heads/yang22-cdh5.7
Commit: cdd945cbe3c6a3326967dda4623bb4f3df855ef1
Parents: ed497fe
Author: Hongbin Ma 
Authored: Fri Dec 16 16:28:18 2016 +0800
Committer: Hongbin Ma 
Committed: Fri Dec 16 16:28:18 2016 +0800

--
 .../apache/kylin/common/KylinConfigBase.java|  1 +
 .../kylin/cube/CubeCapabilityChecker.java   |  5 ++
 .../kylin/gridtable/GTScanRequestBuilder.java   |  2 +-
 .../kylin/metadata/realization/SQLDigest.java   |  4 +-
 .../apache/kylin/storage/StorageContext.java| 74 +++-
 .../storage/gtrecord/CubeScanRangePlanner.java  |  9 +--
 .../gtrecord/GTCubeStorageQueryBase.java|  1 -
 .../gtrecord/SequentialCubeTupleIterator.java   |  6 +-
 .../apache/kylin/query/ITKylinQueryTest.java|  1 -
 .../org/apache/kylin/query/KylinTestBase.java   |  7 +-
 .../kylin/storage/hbase/ITStorageTest.java  |  2 +-
 .../apache/kylin/query/relnode/OLAPContext.java |  6 +-
 .../kylin/query/relnode/OLAPFilterRel.java  |  2 +-
 .../kylin/query/relnode/OLAPLimitRel.java   | 10 ++-
 .../storage/hbase/cube/v1/CubeStorageQuery.java |  2 +-
 .../cube/v1/SerializedHBaseTupleIterator.java   |  2 +-
 .../coprocessor/endpoint/CubeVisitService.java  |  7 +-
 17 files changed, 62 insertions(+), 79 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/cdd945cb/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
--
diff --git 
a/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java 
b/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
index 9923bc1..a1c9050 100644
--- a/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
+++ b/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
@@ -782,6 +782,7 @@ abstract public class KylinConfigBase implements 
Serializable {
 return Boolean.valueOf(getOptional("kylin.query.skip-empty-segments", 
"true"));
 }
 
+@Deprecated//Limit is good even it's large. This config is meaning less 
since we already have scan threshold 
 public int getStoragePushDownLimitMax() {
 return Integer.parseInt(getOptional("kylin.query.max-limit-pushdown", 
"1"));
 }

http://git-wip-us.apache.org/repos/asf/kylin/blob/cdd945cb/core-cube/src/main/java/org/apache/kylin/cube/CubeCapabilityChecker.java
--
diff --git 
a/core-cube/src/main/java/org/apache/kylin/cube/CubeCapabilityChecker.java 
b/core-cube/src/main/java/org/apache/kylin/cube/CubeCapabilityChecker.java
index 38faed9..c45144b 100644
--- a/core-cube/src/main/java/org/apache/kylin/cube/CubeCapabilityChecker.java
+++ b/core-cube/src/main/java/org/apache/kylin/cube/CubeCapabilityChecker.java
@@ -120,6 +120,11 @@ public class CubeCapabilityChecker {
 return result;
 }
 
+if (digest.limitPrecedesAggr) {
+logger.info("Exclude cube " + cube.getName() + " because there's 
limit preceding aggregation");
+return result;
+}
+
 if (digest.isRawQuery && rootFactTable.equals(digest.factTable)) {
 result.influences.add(new CapabilityInfluence() {
 @Override

http://git-wip-us.apache.org/repos/asf/kylin/blob/cdd945cb/core-cube/src/main/java/org/apache/kylin/gridtable/GTScanRequestBuilder.java
--
diff --git 
a/core-cube/src/main/java/org/apache/kylin/gridtable/GTScanRequestBuilder.java 
b/core-cube/src/main/java/org/apache/kylin/gridtable/GTScanRequestBuilder.java
index f542de1..bcec1f4 100644
--- 
a/core-cube/src/main/java/org/apache/kylin/gridtable/GTScanRequestBuilder.java
+++ 
b/core-cube/src/main/java/org/apache/kylin/gridtable/GTScanRequestBuilder.java
@@ -36,7 +36,7 @@ public class GTScanRequestBuilder {
 private boolean allowStorageAggregation = true;
 private double aggCacheMemThreshold = 0;
 private int storageScanRowNumThreshold = Integer.MAX_VALUE;// storage 
should terminate itself when $storageScanRowNumThreshold cuboid rows are 
scanned, and throw exception.   
-private int storagePushDownLimit = Integer.MAX_VALUE;// storage can quit 
working when $toragePushDownLimit aggregated rows are produced. 
+private int storagePushDownLimit = Integer.MAX_VALUE;// storage can quit 

[3/8] kylin git commit: KYLIN-2290 bug fix

[mahongbin]

KYLIN-2290 bug fix


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/4232e0da
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/4232e0da
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/4232e0da

Branch: refs/heads/yang22-cdh5.7
Commit: 4232e0dac801eacf60b08b3f22e383787be54d51
Parents: 398fa0d
Author: Hongbin Ma 
Authored: Fri Dec 16 16:53:33 2016 +0800
Committer: Hongbin Ma 
Committed: Fri Dec 16 16:53:33 2016 +0800

--
 .../java/org/apache/kylin/query/relnode/OLAPAggregateRel.java| 4 
 .../main/java/org/apache/kylin/query/relnode/OLAPLimitRel.java   | 4 
 2 files changed, 4 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/4232e0da/query/src/main/java/org/apache/kylin/query/relnode/OLAPAggregateRel.java
--
diff --git 
a/query/src/main/java/org/apache/kylin/query/relnode/OLAPAggregateRel.java 
b/query/src/main/java/org/apache/kylin/query/relnode/OLAPAggregateRel.java
index 62351d3..3242371 100644
--- a/query/src/main/java/org/apache/kylin/query/relnode/OLAPAggregateRel.java
+++ b/query/src/main/java/org/apache/kylin/query/relnode/OLAPAggregateRel.java
@@ -157,6 +157,10 @@ public class OLAPAggregateRel extends Aggregate implements 
OLAPRel {
 this.context.groupByColumns.addAll(this.groups);
 this.context.aggregations.addAll(this.aggregations);
 this.context.afterAggregate = true;
+
+if (this.context.afterLimit) {
+this.context.limitPrecedesAggr = true;
+}
 } else {
 for (AggregateCall aggCall : aggCalls) {
 // check if supported by kylin

http://git-wip-us.apache.org/repos/asf/kylin/blob/4232e0da/query/src/main/java/org/apache/kylin/query/relnode/OLAPLimitRel.java
--
diff --git 
a/query/src/main/java/org/apache/kylin/query/relnode/OLAPLimitRel.java 
b/query/src/main/java/org/apache/kylin/query/relnode/OLAPLimitRel.java
index 9ebdf60..8179807 100644
--- a/query/src/main/java/org/apache/kylin/query/relnode/OLAPLimitRel.java
+++ b/query/src/main/java/org/apache/kylin/query/relnode/OLAPLimitRel.java
@@ -91,10 +91,6 @@ public class OLAPLimitRel extends SingleRel implements 
OLAPRel {
 }
 
 context.afterLimit = true;
-
-if (!this.context.afterAggregate) {
-this.context.limitPrecedesAggr = true;
-}
 }
 }
 

[4/8] kylin git commit: KYLIN-2292 workaround for CALCITE-1540

[mahongbin]

KYLIN-2292 workaround for CALCITE-1540


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/82f6c553
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/82f6c553
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/82f6c553

Branch: refs/heads/yang22-cdh5.7
Commit: 82f6c553d6b0ea171e59796754b2e42763e0ea2b
Parents: 4232e0d
Author: Hongbin Ma 
Authored: Fri Dec 16 17:21:37 2016 +0800
Committer: Hongbin Ma 
Committed: Fri Dec 16 17:21:37 2016 +0800

--
 .../adapter/enumerable/EnumerableWindow.java| 978 +++
 .../calcite/adapter/enumerable/PhysType.java| 209 
 .../adapter/enumerable/PhysTypeImpl.java| 654 +
 .../test/resources/query/sql_window/query11.sql |  23 +
 .../test/resources/query/sql_window/query12.sql |  26 +
 5 files changed, 1890 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/82f6c553/atopcalcite/src/main/java/org/apache/calcite/adapter/enumerable/EnumerableWindow.java
--
diff --git 
a/atopcalcite/src/main/java/org/apache/calcite/adapter/enumerable/EnumerableWindow.java
 
b/atopcalcite/src/main/java/org/apache/calcite/adapter/enumerable/EnumerableWindow.java
new file mode 100644
index 000..203ce02
--- /dev/null
+++ 
b/atopcalcite/src/main/java/org/apache/calcite/adapter/enumerable/EnumerableWindow.java
@@ -0,0 +1,978 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.adapter.enumerable;
+
+import org.apache.calcite.adapter.enumerable.impl.WinAggAddContextImpl;
+import org.apache.calcite.adapter.enumerable.impl.WinAggResetContextImpl;
+import org.apache.calcite.adapter.enumerable.impl.WinAggResultContextImpl;
+import org.apache.calcite.adapter.java.JavaTypeFactory;
+import org.apache.calcite.linq4j.tree.BinaryExpression;
+import org.apache.calcite.linq4j.tree.BlockBuilder;
+import org.apache.calcite.linq4j.tree.BlockStatement;
+import org.apache.calcite.linq4j.tree.DeclarationStatement;
+import org.apache.calcite.linq4j.tree.Expression;
+import org.apache.calcite.linq4j.tree.Expressions;
+import org.apache.calcite.linq4j.tree.ParameterExpression;
+import org.apache.calcite.linq4j.tree.Primitive;
+import org.apache.calcite.linq4j.tree.Statement;
+import org.apache.calcite.linq4j.tree.Types;
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelOptCost;
+import org.apache.calcite.plan.RelOptPlanner;
+import org.apache.calcite.plan.RelTraitSet;
+import org.apache.calcite.prepare.CalcitePrepareImpl;
+import org.apache.calcite.rel.RelFieldCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.Window;
+import org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexLiteral;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexWindowBound;
+import org.apache.calcite.runtime.SortedMultiMap;
+import org.apache.calcite.sql.SqlAggFunction;
+import org.apache.calcite.util.BuiltInMethod;
+import org.apache.calcite.util.Pair;
+import org.apache.calcite.util.Util;
+
+import com.google.common.base.Function;
+import com.google.common.collect.ImmutableList;
+
+import java.lang.reflect.Modifier;
+import java.lang.reflect.Type;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Iterator;
+import java.util.List;
+
+/*
+ * OVERRIDE POINT: patching CALCITE-1540 on calcite 1.8.0
+ */
+
+/** Implementation of {@link org.apache.calcite.rel.core.Window} in
+ * {@link org.apache.calcite.adapter.enumerable.EnumerableConvention 
enumerable calling convention}. */
+public class EnumerableWindow extends Window implements EnumerableRel {
+/** Creates an EnumerableWindowRel. */
+EnumerableWindow(RelOptCluster cluster, RelTraitSet 

[6/8] kylin git commit: KYLIN-1528 Create a branch for v1.5 with HBase 1.x API

[mahongbin]

http://git-wip-us.apache.org/repos/asf/kylin/blob/7f82104e/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/util/CleanHtableCLI.java
--
diff --git 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/util/CleanHtableCLI.java
 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/util/CleanHtableCLI.java
index a150607..56f867a 100644
--- 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/util/CleanHtableCLI.java
+++ 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/util/CleanHtableCLI.java
@@ -21,9 +21,11 @@ package org.apache.kylin.storage.hbase.util;
 import java.io.IOException;
 
 import org.apache.commons.cli.Options;
-import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.hbase.HTableDescriptor;
-import org.apache.hadoop.hbase.client.HBaseAdmin;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.kylin.common.KylinConfig;
 import org.apache.kylin.common.util.AbstractApplication;
 import org.apache.kylin.common.util.OptionsHelper;
 import org.apache.kylin.metadata.realization.IRealizationConstants;
@@ -38,8 +40,8 @@ public class CleanHtableCLI extends AbstractApplication {
 protected static final Logger logger = 
LoggerFactory.getLogger(CleanHtableCLI.class);
 
 private void clean() throws IOException {
-Configuration conf = HBaseConnection.getCurrentHBaseConfiguration();
-HBaseAdmin hbaseAdmin = new HBaseAdmin(conf);
+Connection conn = 
HBaseConnection.get(KylinConfig.getInstanceFromEnv().getStorageUrl());
+Admin hbaseAdmin = conn.getAdmin();
 
 for (HTableDescriptor descriptor : hbaseAdmin.listTables()) {
 String name = descriptor.getNameAsString().toLowerCase();
@@ -50,7 +52,7 @@ public class CleanHtableCLI extends AbstractApplication {
 System.out.println();
 
 descriptor.setValue(IRealizationConstants.HTableOwner, 
"dl-ebay-ky...@ebay.com");
-hbaseAdmin.modifyTable(descriptor.getNameAsString(), 
descriptor);
+
hbaseAdmin.modifyTable(TableName.valueOf(descriptor.getNameAsString()), 
descriptor);
 }
 }
 hbaseAdmin.close();

http://git-wip-us.apache.org/repos/asf/kylin/blob/7f82104e/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/util/CubeMigrationCLI.java
--
diff --git 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/util/CubeMigrationCLI.java
 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/util/CubeMigrationCLI.java
index 2e682b1..f47bf31 100644
--- 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/util/CubeMigrationCLI.java
+++ 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/util/CubeMigrationCLI.java
@@ -33,12 +33,13 @@ import org.apache.hadoop.hbase.Cell;
 import org.apache.hadoop.hbase.CellUtil;
 import org.apache.hadoop.hbase.HTableDescriptor;
 import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.Connection;
 import org.apache.hadoop.hbase.client.Delete;
 import org.apache.hadoop.hbase.client.Get;
-import org.apache.hadoop.hbase.client.HBaseAdmin;
-import org.apache.hadoop.hbase.client.HTableInterface;
 import org.apache.hadoop.hbase.client.Put;
 import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Table;
 import org.apache.kylin.common.KylinConfig;
 import org.apache.kylin.common.persistence.JsonSerializer;
 import org.apache.kylin.common.persistence.RawResource;
@@ -88,7 +89,7 @@ public class CubeMigrationCLI {
 private static ResourceStore srcStore;
 private static ResourceStore dstStore;
 private static FileSystem hdfsFS;
-private static HBaseAdmin hbaseAdmin;
+private static Admin hbaseAdmin;
 
 public static final String ACL_INFO_FAMILY = "i";
 private static final String ACL_TABLE_NAME = "_acl";
@@ -133,8 +134,8 @@ public class CubeMigrationCLI {
 
 checkAndGetHbaseUrl();
 
-Configuration conf = HBaseConnection.getCurrentHBaseConfiguration();
-hbaseAdmin = new HBaseAdmin(conf);
+Connection conn = HBaseConnection.get(srcConfig.getStorageUrl());
+hbaseAdmin = conn.getAdmin();
 
 hdfsFS = FileSystem.get(new Configuration());
 
@@ -232,6 +233,7 @@ public class CubeMigrationCLI {
 operations.add(new Opt(OptType.COPY_DICT_OR_SNAPSHOT, new Object[] 
{ item, cube.getName() }));
 }
 }
+
 private static void addCubeAndModelIntoProject(CubeInstance srcCube, 
String cubeName, String projectName) throws IOException {
 String projectResPath = 
ProjectInstance.concatResourcePath(projectName);
 if (!dstStore.exists(projectResPath))
@@ -325,8 +327,8 @@ public class 

[7/8] kylin git commit: KYLIN-1528 Create a branch for v1.5 with HBase 1.x API

[mahongbin]

KYLIN-1528 Create a branch for v1.5 with HBase 1.x API


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/7f82104e
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/7f82104e
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/7f82104e

Branch: refs/heads/yang22-cdh5.7
Commit: 7f82104e9fd4eb205bc866badb366655187e2eb2
Parents: bcf2d0e
Author: shaofengshi 
Authored: Wed Mar 23 17:07:05 2016 +0800
Committer: Hongbin Ma 
Committed: Mon Dec 19 10:11:48 2016 +0800

--
 examples/test_case_data/sandbox/hbase-site.xml  | 19 +---
 .../kylin/provision/BuildCubeWithEngine.java| 12 +--
 pom.xml | 12 +--
 .../kylin/rest/security/AclHBaseStorage.java|  4 +-
 .../rest/security/MockAclHBaseStorage.java  |  8 +-
 .../apache/kylin/rest/security/MockHTable.java  | 95 
 .../rest/security/RealAclHBaseStorage.java  |  9 +-
 .../apache/kylin/rest/service/AclService.java   | 25 +++---
 .../apache/kylin/rest/service/CubeService.java  | 35 +++-
 .../apache/kylin/rest/service/QueryService.java | 24 +++--
 .../apache/kylin/rest/service/UserService.java  | 17 ++--
 .../kylin/storage/hbase/HBaseConnection.java| 44 -
 .../kylin/storage/hbase/HBaseResourceStore.java | 31 +++
 .../kylin/storage/hbase/HBaseStorage.java   |  3 +-
 .../storage/hbase/cube/SimpleHBaseStore.java| 20 ++---
 .../hbase/cube/v1/CubeSegmentTupleIterator.java | 11 +--
 .../storage/hbase/cube/v1/CubeStorageQuery.java |  4 +-
 .../hbase/cube/v1/RegionScannerAdapter.java | 10 ++-
 .../cube/v1/SerializedHBaseTupleIterator.java   |  4 +-
 .../observer/AggregateRegionObserver.java   |  4 +-
 .../observer/AggregationScanner.java| 14 ++-
 .../observer/ObserverAggregationCache.java  | 10 ++-
 .../coprocessor/observer/ObserverEnabler.java   |  4 +-
 .../hbase/cube/v2/CubeHBaseEndpointRPC.java | 13 +--
 .../storage/hbase/cube/v2/CubeHBaseScanRPC.java |  9 +-
 .../coprocessor/endpoint/CubeVisitService.java  |  4 +-
 .../storage/hbase/steps/CubeHTableUtil.java | 16 ++--
 .../storage/hbase/steps/DeprecatedGCStep.java   | 23 ++---
 .../storage/hbase/steps/HBaseCuboidWriter.java  |  7 +-
 .../kylin/storage/hbase/steps/MergeGCStep.java  | 23 ++---
 .../storage/hbase/util/CleanHtableCLI.java  | 12 +--
 .../storage/hbase/util/CubeMigrationCLI.java| 36 
 .../hbase/util/CubeMigrationCheckCLI.java   | 17 ++--
 .../hbase/util/DeployCoprocessorCLI.java| 27 +++---
 .../hbase/util/ExtendCubeToHybridCLI.java   |  8 +-
 .../hbase/util/GridTableHBaseBenchmark.java | 34 +++
 .../kylin/storage/hbase/util/HBaseClean.java| 18 ++--
 .../hbase/util/HBaseRegionSizeCalculator.java   | 35 
 .../kylin/storage/hbase/util/HBaseUsage.java|  9 +-
 .../storage/hbase/util/HbaseStreamingInput.java | 30 +++
 .../hbase/util/HtableAlterMetadataCLI.java  |  9 +-
 .../storage/hbase/util/OrphanHBaseCleanJob.java | 19 ++--
 .../kylin/storage/hbase/util/PingHBaseCLI.java  | 15 ++--
 .../kylin/storage/hbase/util/RowCounterCLI.java | 11 +--
 .../storage/hbase/util/StorageCleanupJob.java   | 20 +++--
 .../storage/hbase/util/UpdateHTableHostCLI.java | 17 ++--
 .../observer/AggregateRegionObserverTest.java   | 26 ++
 .../v1/filter/TestFuzzyRowFilterV2EndToEnd.java |  5 +-
 .../org/apache/kylin/tool/CubeMigrationCLI.java | 19 ++--
 .../kylin/tool/ExtendCubeToHybridCLI.java   |  8 +-
 50 files changed, 417 insertions(+), 472 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/7f82104e/examples/test_case_data/sandbox/hbase-site.xml
--
diff --git a/examples/test_case_data/sandbox/hbase-site.xml 
b/examples/test_case_data/sandbox/hbase-site.xml
index 46d5345..734908e 100644
--- a/examples/test_case_data/sandbox/hbase-site.xml
+++ b/examples/test_case_data/sandbox/hbase-site.xml
@@ -190,22 +190,5 @@
 zookeeper.znode.parent
 /hbase-unsecure
 
-
-hbase.client.pause
-100
-General client pause value.  Used mostly as value to wait
-before running a retry of a failed get, region lookup, etc.
-See hbase.client.retries.number for description of how we backoff 
from
-this initial pause amount and how this pause works w/ 
retries.
-
-
-hbase.client.retries.number
-5
-Maximum retries.  Used as maximum for all retryable
-operations such as the getting of a cell's value, starting a row 
update,
-etc.  Retry interval is a rough function based on 
hbase.client.pause.  At
-first we retry at this interval but then with backoff, we pretty 
quickly reach
-   

[2/8] kylin git commit: KYLIN-2290 fix compile

[mahongbin]

KYLIN-2290 fix compile


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/398fa0de
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/398fa0de
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/398fa0de

Branch: refs/heads/yang22-cdh5.7
Commit: 398fa0ded0ea41c97aa0411fe7646fc300891887
Parents: cdd945c
Author: Hongbin Ma 
Authored: Fri Dec 16 16:37:41 2016 +0800
Committer: Hongbin Ma 
Committed: Fri Dec 16 16:37:41 2016 +0800

--
 .../main/java/org/apache/kylin/metadata/realization/SQLDigest.java | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/398fa0de/core-metadata/src/main/java/org/apache/kylin/metadata/realization/SQLDigest.java
--
diff --git 
a/core-metadata/src/main/java/org/apache/kylin/metadata/realization/SQLDigest.java
 
b/core-metadata/src/main/java/org/apache/kylin/metadata/realization/SQLDigest.java
index 4780487..36f303b 100644
--- 
a/core-metadata/src/main/java/org/apache/kylin/metadata/realization/SQLDigest.java
+++ 
b/core-metadata/src/main/java/org/apache/kylin/metadata/realization/SQLDigest.java
@@ -63,7 +63,7 @@ public class SQLDigest {
 
 public SQLDigest(String factTable, TupleFilter filter, List 
joinDescs, Set allColumns, //
 List groupbyColumns, Set 
subqueryJoinParticipants, Set filterColumns, Set 
metricColumns, //
-List aggregations, List aggrSqlCalls, 
List sortColumns, List sortOrders,boolean 
limitPrecedesAggr) {
+List aggregations, List aggrSqlCalls, 
List sortColumns, List sortOrders, boolean 
limitPrecedesAggr) {
 this.factTable = factTable;
 this.filter = filter;
 this.joinDescs = joinDescs;

[2/7] kylin git commit: KYLIN-2290 fix compile

[mahongbin]

KYLIN-2290 fix compile


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/398fa0de
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/398fa0de
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/398fa0de

Branch: refs/heads/yang22-hbase1.x
Commit: 398fa0ded0ea41c97aa0411fe7646fc300891887
Parents: cdd945c
Author: Hongbin Ma 
Authored: Fri Dec 16 16:37:41 2016 +0800
Committer: Hongbin Ma 
Committed: Fri Dec 16 16:37:41 2016 +0800

--
 .../main/java/org/apache/kylin/metadata/realization/SQLDigest.java | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/398fa0de/core-metadata/src/main/java/org/apache/kylin/metadata/realization/SQLDigest.java
--
diff --git 
a/core-metadata/src/main/java/org/apache/kylin/metadata/realization/SQLDigest.java
 
b/core-metadata/src/main/java/org/apache/kylin/metadata/realization/SQLDigest.java
index 4780487..36f303b 100644
--- 
a/core-metadata/src/main/java/org/apache/kylin/metadata/realization/SQLDigest.java
+++ 
b/core-metadata/src/main/java/org/apache/kylin/metadata/realization/SQLDigest.java
@@ -63,7 +63,7 @@ public class SQLDigest {
 
 public SQLDigest(String factTable, TupleFilter filter, List 
joinDescs, Set allColumns, //
 List groupbyColumns, Set 
subqueryJoinParticipants, Set filterColumns, Set 
metricColumns, //
-List aggregations, List aggrSqlCalls, 
List sortColumns, List sortOrders,boolean 
limitPrecedesAggr) {
+List aggregations, List aggrSqlCalls, 
List sortColumns, List sortOrders, boolean 
limitPrecedesAggr) {
 this.factTable = factTable;
 this.filter = filter;
 this.joinDescs = joinDescs;

[7/7] kylin git commit: KYLIN-1528 Create a branch for v1.5 with HBase 1.x API

[mahongbin]

KYLIN-1528 Create a branch for v1.5 with HBase 1.x API


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/7f82104e
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/7f82104e
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/7f82104e

Branch: refs/heads/yang22-hbase1.x
Commit: 7f82104e9fd4eb205bc866badb366655187e2eb2
Parents: bcf2d0e
Author: shaofengshi 
Authored: Wed Mar 23 17:07:05 2016 +0800
Committer: Hongbin Ma 
Committed: Mon Dec 19 10:11:48 2016 +0800

--
 examples/test_case_data/sandbox/hbase-site.xml  | 19 +---
 .../kylin/provision/BuildCubeWithEngine.java| 12 +--
 pom.xml | 12 +--
 .../kylin/rest/security/AclHBaseStorage.java|  4 +-
 .../rest/security/MockAclHBaseStorage.java  |  8 +-
 .../apache/kylin/rest/security/MockHTable.java  | 95 
 .../rest/security/RealAclHBaseStorage.java  |  9 +-
 .../apache/kylin/rest/service/AclService.java   | 25 +++---
 .../apache/kylin/rest/service/CubeService.java  | 35 +++-
 .../apache/kylin/rest/service/QueryService.java | 24 +++--
 .../apache/kylin/rest/service/UserService.java  | 17 ++--
 .../kylin/storage/hbase/HBaseConnection.java| 44 -
 .../kylin/storage/hbase/HBaseResourceStore.java | 31 +++
 .../kylin/storage/hbase/HBaseStorage.java   |  3 +-
 .../storage/hbase/cube/SimpleHBaseStore.java| 20 ++---
 .../hbase/cube/v1/CubeSegmentTupleIterator.java | 11 +--
 .../storage/hbase/cube/v1/CubeStorageQuery.java |  4 +-
 .../hbase/cube/v1/RegionScannerAdapter.java | 10 ++-
 .../cube/v1/SerializedHBaseTupleIterator.java   |  4 +-
 .../observer/AggregateRegionObserver.java   |  4 +-
 .../observer/AggregationScanner.java| 14 ++-
 .../observer/ObserverAggregationCache.java  | 10 ++-
 .../coprocessor/observer/ObserverEnabler.java   |  4 +-
 .../hbase/cube/v2/CubeHBaseEndpointRPC.java | 13 +--
 .../storage/hbase/cube/v2/CubeHBaseScanRPC.java |  9 +-
 .../coprocessor/endpoint/CubeVisitService.java  |  4 +-
 .../storage/hbase/steps/CubeHTableUtil.java | 16 ++--
 .../storage/hbase/steps/DeprecatedGCStep.java   | 23 ++---
 .../storage/hbase/steps/HBaseCuboidWriter.java  |  7 +-
 .../kylin/storage/hbase/steps/MergeGCStep.java  | 23 ++---
 .../storage/hbase/util/CleanHtableCLI.java  | 12 +--
 .../storage/hbase/util/CubeMigrationCLI.java| 36 
 .../hbase/util/CubeMigrationCheckCLI.java   | 17 ++--
 .../hbase/util/DeployCoprocessorCLI.java| 27 +++---
 .../hbase/util/ExtendCubeToHybridCLI.java   |  8 +-
 .../hbase/util/GridTableHBaseBenchmark.java | 34 +++
 .../kylin/storage/hbase/util/HBaseClean.java| 18 ++--
 .../hbase/util/HBaseRegionSizeCalculator.java   | 35 
 .../kylin/storage/hbase/util/HBaseUsage.java|  9 +-
 .../storage/hbase/util/HbaseStreamingInput.java | 30 +++
 .../hbase/util/HtableAlterMetadataCLI.java  |  9 +-
 .../storage/hbase/util/OrphanHBaseCleanJob.java | 19 ++--
 .../kylin/storage/hbase/util/PingHBaseCLI.java  | 15 ++--
 .../kylin/storage/hbase/util/RowCounterCLI.java | 11 +--
 .../storage/hbase/util/StorageCleanupJob.java   | 20 +++--
 .../storage/hbase/util/UpdateHTableHostCLI.java | 17 ++--
 .../observer/AggregateRegionObserverTest.java   | 26 ++
 .../v1/filter/TestFuzzyRowFilterV2EndToEnd.java |  5 +-
 .../org/apache/kylin/tool/CubeMigrationCLI.java | 19 ++--
 .../kylin/tool/ExtendCubeToHybridCLI.java   |  8 +-
 50 files changed, 417 insertions(+), 472 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/7f82104e/examples/test_case_data/sandbox/hbase-site.xml
--
diff --git a/examples/test_case_data/sandbox/hbase-site.xml 
b/examples/test_case_data/sandbox/hbase-site.xml
index 46d5345..734908e 100644
--- a/examples/test_case_data/sandbox/hbase-site.xml
+++ b/examples/test_case_data/sandbox/hbase-site.xml
@@ -190,22 +190,5 @@
 zookeeper.znode.parent
 /hbase-unsecure
 
-
-hbase.client.pause
-100
-General client pause value.  Used mostly as value to wait
-before running a retry of a failed get, region lookup, etc.
-See hbase.client.retries.number for description of how we backoff 
from
-this initial pause amount and how this pause works w/ 
retries.
-
-
-hbase.client.retries.number
-5
-Maximum retries.  Used as maximum for all retryable
-operations such as the getting of a cell's value, starting a row 
update,
-etc.  Retry interval is a rough function based on 
hbase.client.pause.  At
-first we retry at this interval but then with backoff, we pretty 
quickly reach
- 

[5/7] kylin git commit: KYLIN-2291 Collect hive table property skip-header-line-count

[mahongbin]

KYLIN-2291 Collect hive table property skip-header-line-count

Signed-off-by: Li Yang 


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/bcf2d0ef
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/bcf2d0ef
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/bcf2d0ef

Branch: refs/heads/yang22-hbase1.x
Commit: bcf2d0ef25013306798c54bb4e485bdfbe42f35c
Parents: 82f6c55
Author: Cheng Wang 
Authored: Fri Dec 16 17:23:15 2016 +0800
Committer: Li Yang 
Committed: Fri Dec 16 17:52:26 2016 +0800

--
 .../org/apache/kylin/source/hive/BeelineHiveClient.java  |  3 +++
 .../java/org/apache/kylin/source/hive/CLIHiveClient.java |  2 +-
 .../apache/kylin/source/hive/HiveSourceTableLoader.java  |  1 +
 .../java/org/apache/kylin/source/hive/HiveTableMeta.java |  4 +++-
 .../apache/kylin/source/hive/HiveTableMetaBuilder.java   | 11 ++-
 5 files changed, 18 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/bcf2d0ef/source-hive/src/main/java/org/apache/kylin/source/hive/BeelineHiveClient.java
--
diff --git 
a/source-hive/src/main/java/org/apache/kylin/source/hive/BeelineHiveClient.java 
b/source-hive/src/main/java/org/apache/kylin/source/hive/BeelineHiveClient.java
index 47b551b..468ccb1 100644
--- 
a/source-hive/src/main/java/org/apache/kylin/source/hive/BeelineHiveClient.java
+++ 
b/source-hive/src/main/java/org/apache/kylin/source/hive/BeelineHiveClient.java
@@ -193,6 +193,9 @@ public class BeelineHiveClient implements IHiveClient {
 if ("numFiles".equals(resultSet.getString(2).trim())) {
 
builder.setFileNum(Long.parseLong(resultSet.getString(3).trim()));
 }
+if 
("skip.header.line.count".equals(resultSet.getString(2).trim())) {
+
builder.setSkipHeaderLineCount(resultSet.getString(3).trim());
+}
 }
 }
 if ("InputFormat:".equals(resultSet.getString(1).trim())) {

http://git-wip-us.apache.org/repos/asf/kylin/blob/bcf2d0ef/source-hive/src/main/java/org/apache/kylin/source/hive/CLIHiveClient.java
--
diff --git 
a/source-hive/src/main/java/org/apache/kylin/source/hive/CLIHiveClient.java 
b/source-hive/src/main/java/org/apache/kylin/source/hive/CLIHiveClient.java
index 5a17f1f..e8a93bd 100644
--- a/source-hive/src/main/java/org/apache/kylin/source/hive/CLIHiveClient.java
+++ b/source-hive/src/main/java/org/apache/kylin/source/hive/CLIHiveClient.java
@@ -42,7 +42,6 @@ import com.google.common.collect.Lists;
  *
  */
 public class CLIHiveClient implements IHiveClient {
-
 protected HiveConf hiveConf = null;
 protected Driver driver = null;
 protected HiveMetaStoreClient metaStoreClient = null;
@@ -109,6 +108,7 @@ public class CLIHiveClient implements IHiveClient {
 builder.setOwner(table.getOwner());
 builder.setLastAccessTime(table.getLastAccessTime());
 builder.setTableType(table.getTableType());
+
builder.setSkipHeaderLineCount(table.getParameters().get("skip.header.line.count"));
 
 return builder.createHiveTableMeta();
 }

http://git-wip-us.apache.org/repos/asf/kylin/blob/bcf2d0ef/source-hive/src/main/java/org/apache/kylin/source/hive/HiveSourceTableLoader.java
--
diff --git 
a/source-hive/src/main/java/org/apache/kylin/source/hive/HiveSourceTableLoader.java
 
b/source-hive/src/main/java/org/apache/kylin/source/hive/HiveSourceTableLoader.java
index 57292dc..77e1084 100644
--- 
a/source-hive/src/main/java/org/apache/kylin/source/hive/HiveSourceTableLoader.java
+++ 
b/source-hive/src/main/java/org/apache/kylin/source/hive/HiveSourceTableLoader.java
@@ -140,6 +140,7 @@ public class HiveSourceTableLoader {
 tableExtDesc.addDataSourceProp("total_file_number", 
String.valueOf(hiveTableMeta.fileNum));
 tableExtDesc.addDataSourceProp("hive_inputFormat", 
hiveTableMeta.sdInputFormat);
 tableExtDesc.addDataSourceProp("hive_outputFormat", 
hiveTableMeta.sdOutputFormat);
+tableExtDesc.addDataSourceProp("skip_header_line_count", 
String.valueOf(hiveTableMeta.skipHeaderLineCount));
 
 metaMgr.saveTableExt(tableExtDesc);
 metaMgr.saveSourceTable(tableDesc);

http://git-wip-us.apache.org/repos/asf/kylin/blob/bcf2d0ef/source-hive/src/main/java/org/apache/kylin/source/hive/HiveTableMeta.java
--
diff --git 

[1/7] kylin git commit: KYLIN-2290 minor improvements on limit [Forced Update!]

[mahongbin]

Repository: kylin
Updated Branches:
  refs/heads/yang22-hbase1.x c33fa8c9d -> 7f82104e9 (forced update)


KYLIN-2290 minor improvements on limit


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/cdd945cb
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/cdd945cb
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/cdd945cb

Branch: refs/heads/yang22-hbase1.x
Commit: cdd945cbe3c6a3326967dda4623bb4f3df855ef1
Parents: ed497fe
Author: Hongbin Ma 
Authored: Fri Dec 16 16:28:18 2016 +0800
Committer: Hongbin Ma 
Committed: Fri Dec 16 16:28:18 2016 +0800

--
 .../apache/kylin/common/KylinConfigBase.java|  1 +
 .../kylin/cube/CubeCapabilityChecker.java   |  5 ++
 .../kylin/gridtable/GTScanRequestBuilder.java   |  2 +-
 .../kylin/metadata/realization/SQLDigest.java   |  4 +-
 .../apache/kylin/storage/StorageContext.java| 74 +++-
 .../storage/gtrecord/CubeScanRangePlanner.java  |  9 +--
 .../gtrecord/GTCubeStorageQueryBase.java|  1 -
 .../gtrecord/SequentialCubeTupleIterator.java   |  6 +-
 .../apache/kylin/query/ITKylinQueryTest.java|  1 -
 .../org/apache/kylin/query/KylinTestBase.java   |  7 +-
 .../kylin/storage/hbase/ITStorageTest.java  |  2 +-
 .../apache/kylin/query/relnode/OLAPContext.java |  6 +-
 .../kylin/query/relnode/OLAPFilterRel.java  |  2 +-
 .../kylin/query/relnode/OLAPLimitRel.java   | 10 ++-
 .../storage/hbase/cube/v1/CubeStorageQuery.java |  2 +-
 .../cube/v1/SerializedHBaseTupleIterator.java   |  2 +-
 .../coprocessor/endpoint/CubeVisitService.java  |  7 +-
 17 files changed, 62 insertions(+), 79 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/cdd945cb/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
--
diff --git 
a/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java 
b/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
index 9923bc1..a1c9050 100644
--- a/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
+++ b/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
@@ -782,6 +782,7 @@ abstract public class KylinConfigBase implements 
Serializable {
 return Boolean.valueOf(getOptional("kylin.query.skip-empty-segments", 
"true"));
 }
 
+@Deprecated//Limit is good even it's large. This config is meaning less 
since we already have scan threshold 
 public int getStoragePushDownLimitMax() {
 return Integer.parseInt(getOptional("kylin.query.max-limit-pushdown", 
"1"));
 }

http://git-wip-us.apache.org/repos/asf/kylin/blob/cdd945cb/core-cube/src/main/java/org/apache/kylin/cube/CubeCapabilityChecker.java
--
diff --git 
a/core-cube/src/main/java/org/apache/kylin/cube/CubeCapabilityChecker.java 
b/core-cube/src/main/java/org/apache/kylin/cube/CubeCapabilityChecker.java
index 38faed9..c45144b 100644
--- a/core-cube/src/main/java/org/apache/kylin/cube/CubeCapabilityChecker.java
+++ b/core-cube/src/main/java/org/apache/kylin/cube/CubeCapabilityChecker.java
@@ -120,6 +120,11 @@ public class CubeCapabilityChecker {
 return result;
 }
 
+if (digest.limitPrecedesAggr) {
+logger.info("Exclude cube " + cube.getName() + " because there's 
limit preceding aggregation");
+return result;
+}
+
 if (digest.isRawQuery && rootFactTable.equals(digest.factTable)) {
 result.influences.add(new CapabilityInfluence() {
 @Override

http://git-wip-us.apache.org/repos/asf/kylin/blob/cdd945cb/core-cube/src/main/java/org/apache/kylin/gridtable/GTScanRequestBuilder.java
--
diff --git 
a/core-cube/src/main/java/org/apache/kylin/gridtable/GTScanRequestBuilder.java 
b/core-cube/src/main/java/org/apache/kylin/gridtable/GTScanRequestBuilder.java
index f542de1..bcec1f4 100644
--- 
a/core-cube/src/main/java/org/apache/kylin/gridtable/GTScanRequestBuilder.java
+++ 
b/core-cube/src/main/java/org/apache/kylin/gridtable/GTScanRequestBuilder.java
@@ -36,7 +36,7 @@ public class GTScanRequestBuilder {
 private boolean allowStorageAggregation = true;
 private double aggCacheMemThreshold = 0;
 private int storageScanRowNumThreshold = Integer.MAX_VALUE;// storage 
should terminate itself when $storageScanRowNumThreshold cuboid rows are 
scanned, and throw exception.   
-private int storagePushDownLimit = Integer.MAX_VALUE;// storage can quit 
working when $toragePushDownLimit aggregated rows are produced. 
+private int storagePushDownLimit = Integer.MAX_VALUE;// storage can quit 

[3/7] kylin git commit: KYLIN-2290 bug fix

[mahongbin]

KYLIN-2290 bug fix


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/4232e0da
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/4232e0da
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/4232e0da

Branch: refs/heads/yang22-hbase1.x
Commit: 4232e0dac801eacf60b08b3f22e383787be54d51
Parents: 398fa0d
Author: Hongbin Ma 
Authored: Fri Dec 16 16:53:33 2016 +0800
Committer: Hongbin Ma 
Committed: Fri Dec 16 16:53:33 2016 +0800

--
 .../java/org/apache/kylin/query/relnode/OLAPAggregateRel.java| 4 
 .../main/java/org/apache/kylin/query/relnode/OLAPLimitRel.java   | 4 
 2 files changed, 4 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/4232e0da/query/src/main/java/org/apache/kylin/query/relnode/OLAPAggregateRel.java
--
diff --git 
a/query/src/main/java/org/apache/kylin/query/relnode/OLAPAggregateRel.java 
b/query/src/main/java/org/apache/kylin/query/relnode/OLAPAggregateRel.java
index 62351d3..3242371 100644
--- a/query/src/main/java/org/apache/kylin/query/relnode/OLAPAggregateRel.java
+++ b/query/src/main/java/org/apache/kylin/query/relnode/OLAPAggregateRel.java
@@ -157,6 +157,10 @@ public class OLAPAggregateRel extends Aggregate implements 
OLAPRel {
 this.context.groupByColumns.addAll(this.groups);
 this.context.aggregations.addAll(this.aggregations);
 this.context.afterAggregate = true;
+
+if (this.context.afterLimit) {
+this.context.limitPrecedesAggr = true;
+}
 } else {
 for (AggregateCall aggCall : aggCalls) {
 // check if supported by kylin

http://git-wip-us.apache.org/repos/asf/kylin/blob/4232e0da/query/src/main/java/org/apache/kylin/query/relnode/OLAPLimitRel.java
--
diff --git 
a/query/src/main/java/org/apache/kylin/query/relnode/OLAPLimitRel.java 
b/query/src/main/java/org/apache/kylin/query/relnode/OLAPLimitRel.java
index 9ebdf60..8179807 100644
--- a/query/src/main/java/org/apache/kylin/query/relnode/OLAPLimitRel.java
+++ b/query/src/main/java/org/apache/kylin/query/relnode/OLAPLimitRel.java
@@ -91,10 +91,6 @@ public class OLAPLimitRel extends SingleRel implements 
OLAPRel {
 }
 
 context.afterLimit = true;
-
-if (!this.context.afterAggregate) {
-this.context.limitPrecedesAggr = true;
-}
 }
 }
 

[4/7] kylin git commit: KYLIN-2292 workaround for CALCITE-1540

[mahongbin]

KYLIN-2292 workaround for CALCITE-1540


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/82f6c553
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/82f6c553
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/82f6c553

Branch: refs/heads/yang22-hbase1.x
Commit: 82f6c553d6b0ea171e59796754b2e42763e0ea2b
Parents: 4232e0d
Author: Hongbin Ma 
Authored: Fri Dec 16 17:21:37 2016 +0800
Committer: Hongbin Ma 
Committed: Fri Dec 16 17:21:37 2016 +0800

--
 .../adapter/enumerable/EnumerableWindow.java| 978 +++
 .../calcite/adapter/enumerable/PhysType.java| 209 
 .../adapter/enumerable/PhysTypeImpl.java| 654 +
 .../test/resources/query/sql_window/query11.sql |  23 +
 .../test/resources/query/sql_window/query12.sql |  26 +
 5 files changed, 1890 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/82f6c553/atopcalcite/src/main/java/org/apache/calcite/adapter/enumerable/EnumerableWindow.java
--
diff --git 
a/atopcalcite/src/main/java/org/apache/calcite/adapter/enumerable/EnumerableWindow.java
 
b/atopcalcite/src/main/java/org/apache/calcite/adapter/enumerable/EnumerableWindow.java
new file mode 100644
index 000..203ce02
--- /dev/null
+++ 
b/atopcalcite/src/main/java/org/apache/calcite/adapter/enumerable/EnumerableWindow.java
@@ -0,0 +1,978 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.adapter.enumerable;
+
+import org.apache.calcite.adapter.enumerable.impl.WinAggAddContextImpl;
+import org.apache.calcite.adapter.enumerable.impl.WinAggResetContextImpl;
+import org.apache.calcite.adapter.enumerable.impl.WinAggResultContextImpl;
+import org.apache.calcite.adapter.java.JavaTypeFactory;
+import org.apache.calcite.linq4j.tree.BinaryExpression;
+import org.apache.calcite.linq4j.tree.BlockBuilder;
+import org.apache.calcite.linq4j.tree.BlockStatement;
+import org.apache.calcite.linq4j.tree.DeclarationStatement;
+import org.apache.calcite.linq4j.tree.Expression;
+import org.apache.calcite.linq4j.tree.Expressions;
+import org.apache.calcite.linq4j.tree.ParameterExpression;
+import org.apache.calcite.linq4j.tree.Primitive;
+import org.apache.calcite.linq4j.tree.Statement;
+import org.apache.calcite.linq4j.tree.Types;
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelOptCost;
+import org.apache.calcite.plan.RelOptPlanner;
+import org.apache.calcite.plan.RelTraitSet;
+import org.apache.calcite.prepare.CalcitePrepareImpl;
+import org.apache.calcite.rel.RelFieldCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.Window;
+import org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexLiteral;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexWindowBound;
+import org.apache.calcite.runtime.SortedMultiMap;
+import org.apache.calcite.sql.SqlAggFunction;
+import org.apache.calcite.util.BuiltInMethod;
+import org.apache.calcite.util.Pair;
+import org.apache.calcite.util.Util;
+
+import com.google.common.base.Function;
+import com.google.common.collect.ImmutableList;
+
+import java.lang.reflect.Modifier;
+import java.lang.reflect.Type;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Iterator;
+import java.util.List;
+
+/*
+ * OVERRIDE POINT: patching CALCITE-1540 on calcite 1.8.0
+ */
+
+/** Implementation of {@link org.apache.calcite.rel.core.Window} in
+ * {@link org.apache.calcite.adapter.enumerable.EnumerableConvention 
enumerable calling convention}. */
+public class EnumerableWindow extends Window implements EnumerableRel {
+/** Creates an EnumerableWindowRel. */
+EnumerableWindow(RelOptCluster cluster, RelTraitSet 

kylin git commit: KYLIN-2292 workaround for CALCITE-1540

[mahongbin]

Repository: kylin
Updated Branches:
  refs/heads/yang22 4232e0dac -> 82f6c553d


KYLIN-2292 workaround for CALCITE-1540


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/82f6c553
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/82f6c553
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/82f6c553

Branch: refs/heads/yang22
Commit: 82f6c553d6b0ea171e59796754b2e42763e0ea2b
Parents: 4232e0d
Author: Hongbin Ma 
Authored: Fri Dec 16 17:21:37 2016 +0800
Committer: Hongbin Ma 
Committed: Fri Dec 16 17:21:37 2016 +0800

--
 .../adapter/enumerable/EnumerableWindow.java| 978 +++
 .../calcite/adapter/enumerable/PhysType.java| 209 
 .../adapter/enumerable/PhysTypeImpl.java| 654 +
 .../test/resources/query/sql_window/query11.sql |  23 +
 .../test/resources/query/sql_window/query12.sql |  26 +
 5 files changed, 1890 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/82f6c553/atopcalcite/src/main/java/org/apache/calcite/adapter/enumerable/EnumerableWindow.java
--
diff --git 
a/atopcalcite/src/main/java/org/apache/calcite/adapter/enumerable/EnumerableWindow.java
 
b/atopcalcite/src/main/java/org/apache/calcite/adapter/enumerable/EnumerableWindow.java
new file mode 100644
index 000..203ce02
--- /dev/null
+++ 
b/atopcalcite/src/main/java/org/apache/calcite/adapter/enumerable/EnumerableWindow.java
@@ -0,0 +1,978 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.adapter.enumerable;
+
+import org.apache.calcite.adapter.enumerable.impl.WinAggAddContextImpl;
+import org.apache.calcite.adapter.enumerable.impl.WinAggResetContextImpl;
+import org.apache.calcite.adapter.enumerable.impl.WinAggResultContextImpl;
+import org.apache.calcite.adapter.java.JavaTypeFactory;
+import org.apache.calcite.linq4j.tree.BinaryExpression;
+import org.apache.calcite.linq4j.tree.BlockBuilder;
+import org.apache.calcite.linq4j.tree.BlockStatement;
+import org.apache.calcite.linq4j.tree.DeclarationStatement;
+import org.apache.calcite.linq4j.tree.Expression;
+import org.apache.calcite.linq4j.tree.Expressions;
+import org.apache.calcite.linq4j.tree.ParameterExpression;
+import org.apache.calcite.linq4j.tree.Primitive;
+import org.apache.calcite.linq4j.tree.Statement;
+import org.apache.calcite.linq4j.tree.Types;
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelOptCost;
+import org.apache.calcite.plan.RelOptPlanner;
+import org.apache.calcite.plan.RelTraitSet;
+import org.apache.calcite.prepare.CalcitePrepareImpl;
+import org.apache.calcite.rel.RelFieldCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.Window;
+import org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexLiteral;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexWindowBound;
+import org.apache.calcite.runtime.SortedMultiMap;
+import org.apache.calcite.sql.SqlAggFunction;
+import org.apache.calcite.util.BuiltInMethod;
+import org.apache.calcite.util.Pair;
+import org.apache.calcite.util.Util;
+
+import com.google.common.base.Function;
+import com.google.common.collect.ImmutableList;
+
+import java.lang.reflect.Modifier;
+import java.lang.reflect.Type;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Iterator;
+import java.util.List;
+
+/*
+ * OVERRIDE POINT: patching CALCITE-1540 on calcite 1.8.0
+ */
+
+/** Implementation of {@link org.apache.calcite.rel.core.Window} in
+ * {@link org.apache.calcite.adapter.enumerable.EnumerableConvention 
enumerable calling convention}. */
+public class EnumerableWindow extends Window implements EnumerableRel {
+/** Creates an 

kylin git commit: KYLIN-2290 bug fix

[mahongbin]

Repository: kylin
Updated Branches:
  refs/heads/yang22 398fa0ded -> 4232e0dac


KYLIN-2290 bug fix


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/4232e0da
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/4232e0da
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/4232e0da

Branch: refs/heads/yang22
Commit: 4232e0dac801eacf60b08b3f22e383787be54d51
Parents: 398fa0d
Author: Hongbin Ma 
Authored: Fri Dec 16 16:53:33 2016 +0800
Committer: Hongbin Ma 
Committed: Fri Dec 16 16:53:33 2016 +0800

--
 .../java/org/apache/kylin/query/relnode/OLAPAggregateRel.java| 4 
 .../main/java/org/apache/kylin/query/relnode/OLAPLimitRel.java   | 4 
 2 files changed, 4 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/4232e0da/query/src/main/java/org/apache/kylin/query/relnode/OLAPAggregateRel.java
--
diff --git 
a/query/src/main/java/org/apache/kylin/query/relnode/OLAPAggregateRel.java 
b/query/src/main/java/org/apache/kylin/query/relnode/OLAPAggregateRel.java
index 62351d3..3242371 100644
--- a/query/src/main/java/org/apache/kylin/query/relnode/OLAPAggregateRel.java
+++ b/query/src/main/java/org/apache/kylin/query/relnode/OLAPAggregateRel.java
@@ -157,6 +157,10 @@ public class OLAPAggregateRel extends Aggregate implements 
OLAPRel {
 this.context.groupByColumns.addAll(this.groups);
 this.context.aggregations.addAll(this.aggregations);
 this.context.afterAggregate = true;
+
+if (this.context.afterLimit) {
+this.context.limitPrecedesAggr = true;
+}
 } else {
 for (AggregateCall aggCall : aggCalls) {
 // check if supported by kylin

http://git-wip-us.apache.org/repos/asf/kylin/blob/4232e0da/query/src/main/java/org/apache/kylin/query/relnode/OLAPLimitRel.java
--
diff --git 
a/query/src/main/java/org/apache/kylin/query/relnode/OLAPLimitRel.java 
b/query/src/main/java/org/apache/kylin/query/relnode/OLAPLimitRel.java
index 9ebdf60..8179807 100644
--- a/query/src/main/java/org/apache/kylin/query/relnode/OLAPLimitRel.java
+++ b/query/src/main/java/org/apache/kylin/query/relnode/OLAPLimitRel.java
@@ -91,10 +91,6 @@ public class OLAPLimitRel extends SingleRel implements 
OLAPRel {
 }
 
 context.afterLimit = true;
-
-if (!this.context.afterAggregate) {
-this.context.limitPrecedesAggr = true;
-}
 }
 }
 

kylin git commit: KYLIN-2290 fix compile

[mahongbin]

Repository: kylin
Updated Branches:
  refs/heads/yang22 cdd945cbe -> 398fa0ded


KYLIN-2290 fix compile


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/398fa0de
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/398fa0de
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/398fa0de

Branch: refs/heads/yang22
Commit: 398fa0ded0ea41c97aa0411fe7646fc300891887
Parents: cdd945c
Author: Hongbin Ma 
Authored: Fri Dec 16 16:37:41 2016 +0800
Committer: Hongbin Ma 
Committed: Fri Dec 16 16:37:41 2016 +0800

--
 .../main/java/org/apache/kylin/metadata/realization/SQLDigest.java | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/398fa0de/core-metadata/src/main/java/org/apache/kylin/metadata/realization/SQLDigest.java
--
diff --git 
a/core-metadata/src/main/java/org/apache/kylin/metadata/realization/SQLDigest.java
 
b/core-metadata/src/main/java/org/apache/kylin/metadata/realization/SQLDigest.java
index 4780487..36f303b 100644
--- 
a/core-metadata/src/main/java/org/apache/kylin/metadata/realization/SQLDigest.java
+++ 
b/core-metadata/src/main/java/org/apache/kylin/metadata/realization/SQLDigest.java
@@ -63,7 +63,7 @@ public class SQLDigest {
 
 public SQLDigest(String factTable, TupleFilter filter, List 
joinDescs, Set allColumns, //
 List groupbyColumns, Set 
subqueryJoinParticipants, Set filterColumns, Set 
metricColumns, //
-List aggregations, List aggrSqlCalls, 
List sortColumns, List sortOrders,boolean 
limitPrecedesAggr) {
+List aggregations, List aggrSqlCalls, 
List sortColumns, List sortOrders, boolean 
limitPrecedesAggr) {
 this.factTable = factTable;
 this.filter = filter;
 this.joinDescs = joinDescs;

kylin git commit: KYLIN-2290 minor improvements on limit

[mahongbin]

Repository: kylin
Updated Branches:
  refs/heads/yang22 ed497fe73 -> cdd945cbe


KYLIN-2290 minor improvements on limit


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/cdd945cb
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/cdd945cb
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/cdd945cb

Branch: refs/heads/yang22
Commit: cdd945cbe3c6a3326967dda4623bb4f3df855ef1
Parents: ed497fe
Author: Hongbin Ma 
Authored: Fri Dec 16 16:28:18 2016 +0800
Committer: Hongbin Ma 
Committed: Fri Dec 16 16:28:18 2016 +0800

--
 .../apache/kylin/common/KylinConfigBase.java|  1 +
 .../kylin/cube/CubeCapabilityChecker.java   |  5 ++
 .../kylin/gridtable/GTScanRequestBuilder.java   |  2 +-
 .../kylin/metadata/realization/SQLDigest.java   |  4 +-
 .../apache/kylin/storage/StorageContext.java| 74 +++-
 .../storage/gtrecord/CubeScanRangePlanner.java  |  9 +--
 .../gtrecord/GTCubeStorageQueryBase.java|  1 -
 .../gtrecord/SequentialCubeTupleIterator.java   |  6 +-
 .../apache/kylin/query/ITKylinQueryTest.java|  1 -
 .../org/apache/kylin/query/KylinTestBase.java   |  7 +-
 .../kylin/storage/hbase/ITStorageTest.java  |  2 +-
 .../apache/kylin/query/relnode/OLAPContext.java |  6 +-
 .../kylin/query/relnode/OLAPFilterRel.java  |  2 +-
 .../kylin/query/relnode/OLAPLimitRel.java   | 10 ++-
 .../storage/hbase/cube/v1/CubeStorageQuery.java |  2 +-
 .../cube/v1/SerializedHBaseTupleIterator.java   |  2 +-
 .../coprocessor/endpoint/CubeVisitService.java  |  7 +-
 17 files changed, 62 insertions(+), 79 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/cdd945cb/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
--
diff --git 
a/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java 
b/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
index 9923bc1..a1c9050 100644
--- a/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
+++ b/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
@@ -782,6 +782,7 @@ abstract public class KylinConfigBase implements 
Serializable {
 return Boolean.valueOf(getOptional("kylin.query.skip-empty-segments", 
"true"));
 }
 
+@Deprecated//Limit is good even it's large. This config is meaning less 
since we already have scan threshold 
 public int getStoragePushDownLimitMax() {
 return Integer.parseInt(getOptional("kylin.query.max-limit-pushdown", 
"1"));
 }

http://git-wip-us.apache.org/repos/asf/kylin/blob/cdd945cb/core-cube/src/main/java/org/apache/kylin/cube/CubeCapabilityChecker.java
--
diff --git 
a/core-cube/src/main/java/org/apache/kylin/cube/CubeCapabilityChecker.java 
b/core-cube/src/main/java/org/apache/kylin/cube/CubeCapabilityChecker.java
index 38faed9..c45144b 100644
--- a/core-cube/src/main/java/org/apache/kylin/cube/CubeCapabilityChecker.java
+++ b/core-cube/src/main/java/org/apache/kylin/cube/CubeCapabilityChecker.java
@@ -120,6 +120,11 @@ public class CubeCapabilityChecker {
 return result;
 }
 
+if (digest.limitPrecedesAggr) {
+logger.info("Exclude cube " + cube.getName() + " because there's 
limit preceding aggregation");
+return result;
+}
+
 if (digest.isRawQuery && rootFactTable.equals(digest.factTable)) {
 result.influences.add(new CapabilityInfluence() {
 @Override

http://git-wip-us.apache.org/repos/asf/kylin/blob/cdd945cb/core-cube/src/main/java/org/apache/kylin/gridtable/GTScanRequestBuilder.java
--
diff --git 
a/core-cube/src/main/java/org/apache/kylin/gridtable/GTScanRequestBuilder.java 
b/core-cube/src/main/java/org/apache/kylin/gridtable/GTScanRequestBuilder.java
index f542de1..bcec1f4 100644
--- 
a/core-cube/src/main/java/org/apache/kylin/gridtable/GTScanRequestBuilder.java
+++ 
b/core-cube/src/main/java/org/apache/kylin/gridtable/GTScanRequestBuilder.java
@@ -36,7 +36,7 @@ public class GTScanRequestBuilder {
 private boolean allowStorageAggregation = true;
 private double aggCacheMemThreshold = 0;
 private int storageScanRowNumThreshold = Integer.MAX_VALUE;// storage 
should terminate itself when $storageScanRowNumThreshold cuboid rows are 
scanned, and throw exception.   
-private int storagePushDownLimit = Integer.MAX_VALUE;// storage can quit 
working when $toragePushDownLimit aggregated rows are produced. 
+private int storagePushDownLimit = Integer.MAX_VALUE;// storage can quit 
scanning safely when 

kylin git commit: KYLIN-2261 revert the document change in KYLIN-2144 because it did not affect 1.6.x branch

[mahongbin]

Repository: kylin
Updated Branches:
  refs/heads/document 1dcf042af -> b570e


KYLIN-2261 revert the document change in KYLIN-2144 because it did not affect 
1.6.x branch


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/b570ebbb
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/b570ebbb
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/b570ebbb

Branch: refs/heads/document
Commit: b570e24740cb46e0601c49efdb952e51555b
Parents: 1dcf042
Author: Hongbin Ma 
Authored: Sat Dec 10 16:11:36 2016 +0800
Committer: Hongbin Ma 
Committed: Sat Dec 10 16:11:36 2016 +0800

--
 website/_docs16/howto/howto_cleanup_storage.md | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/b570ebbb/website/_docs16/howto/howto_cleanup_storage.md
--
diff --git a/website/_docs16/howto/howto_cleanup_storage.md 
b/website/_docs16/howto/howto_cleanup_storage.md
index 3db5d49..233d32d 100644
--- a/website/_docs16/howto/howto_cleanup_storage.md
+++ b/website/_docs16/howto/howto_cleanup_storage.md
@@ -12,11 +12,11 @@ Steps:
 1. Check which resources can be cleanup, this will not remove anything:
 {% highlight Groff markup %}
 export KYLIN_HOME=/path/to/kylin_home
-${KYLIN_HOME}/bin/kylin.sh org.apache.kylin.tool.StorageCleanupJob --delete 
false
+${KYLIN_HOME}/bin/kylin.sh 
org.apache.kylin.storage.hbase.util.StorageCleanupJob --delete false
 {% endhighlight %}
 Here please replace (version) with the specific Kylin jar version in your 
installation;
 2. You can pickup 1 or 2 resources to check whether they're no longer be 
referred; Then add the "--delete true" option to start the cleanup:
 {% highlight Groff markup %}
-${KYLIN_HOME}/bin/kylin.sh org.apache.kylin.tool.StorageCleanupJob --delete 
true
+${KYLIN_HOME}/bin/kylin.sh 
org.apache.kylin.storage.hbase.util.StorageCleanupJob --delete true
 {% endhighlight %}
 On finish, the intermediate HDFS location and HTables should be dropped;

kylin git commit: minor improvement on limit

[mahongbin]

Repository: kylin
Updated Branches:
  refs/heads/master af8940d5b -> c0b87039e


minor improvement on limit


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/c0b87039
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/c0b87039
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/c0b87039

Branch: refs/heads/master
Commit: c0b87039eb45ee94c6ecab71e27d14ee81bac8e2
Parents: af8940d
Author: Hongbin Ma 
Authored: Thu Dec 8 16:01:54 2016 +0800
Committer: Hongbin Ma 
Committed: Thu Dec 8 16:04:09 2016 +0800

--
 .../main/java/org/apache/kylin/storage/StorageContext.java  | 9 +++--
 .../java/org/apache/kylin/query/relnode/OLAPContext.java| 2 --
 .../java/org/apache/kylin/query/relnode/OLAPLimitRel.java   | 1 -
 3 files changed, 7 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/c0b87039/core-storage/src/main/java/org/apache/kylin/storage/StorageContext.java
--
diff --git 
a/core-storage/src/main/java/org/apache/kylin/storage/StorageContext.java 
b/core-storage/src/main/java/org/apache/kylin/storage/StorageContext.java
index 719cab6..bc43a87 100644
--- a/core-storage/src/main/java/org/apache/kylin/storage/StorageContext.java
+++ b/core-storage/src/main/java/org/apache/kylin/storage/StorageContext.java
@@ -92,7 +92,12 @@ public class StorageContext {
 }
 
 public void setLimit(int l) {
-this.limit = l;
+if (l > limit) {
+//cases like : select price from (select * from kylin_sales limit 
10) limit 5000
+logger.info("Setting limit to {} but in current olap context, the 
limit is already {}, won't apply", l, limit);
+} else {
+this.limit = l;
+}
 }
 
 public int getOffset() {
@@ -126,7 +131,7 @@ public class StorageContext {
 if (tempPushDownLimit == Integer.MAX_VALUE) {
 return;
 }
-
+
 int pushDownLimitMax = 
KylinConfig.getInstanceFromEnv().getStoragePushDownLimitMax();
 if (!realization.supportsLimitPushDown()) {
 logger.info("Not enabling limit push down because cube storage 
type not supported");

http://git-wip-us.apache.org/repos/asf/kylin/blob/c0b87039/query/src/main/java/org/apache/kylin/query/relnode/OLAPContext.java
--
diff --git 
a/query/src/main/java/org/apache/kylin/query/relnode/OLAPContext.java 
b/query/src/main/java/org/apache/kylin/query/relnode/OLAPContext.java
index 5fb2c5c..8278fb0 100644
--- a/query/src/main/java/org/apache/kylin/query/relnode/OLAPContext.java
+++ b/query/src/main/java/org/apache/kylin/query/relnode/OLAPContext.java
@@ -135,8 +135,6 @@ public class OLAPContext {
 // rewrite info
 public Map rewriteFields = new HashMap<>();
 
-public int limit;
-
 // hive query
 public String sql = "";
 

http://git-wip-us.apache.org/repos/asf/kylin/blob/c0b87039/query/src/main/java/org/apache/kylin/query/relnode/OLAPLimitRel.java
--
diff --git 
a/query/src/main/java/org/apache/kylin/query/relnode/OLAPLimitRel.java 
b/query/src/main/java/org/apache/kylin/query/relnode/OLAPLimitRel.java
index d6569f6..f0af863 100644
--- a/query/src/main/java/org/apache/kylin/query/relnode/OLAPLimitRel.java
+++ b/query/src/main/java/org/apache/kylin/query/relnode/OLAPLimitRel.java
@@ -81,7 +81,6 @@ public class OLAPLimitRel extends SingleRel implements 
OLAPRel {
 Number limitValue = (Number) (((RexLiteral) 
localFetch).getValue());
 int limit = limitValue.intValue();
 this.context.storageContext.setLimit(limit);
-this.context.limit = limit;
 
 if (localOffset != null) {
 Number offsetValue = (Number) (((RexLiteral) 
localOffset).getValue());

kylin git commit: minor improvement on limit

[mahongbin]

Repository: kylin
Updated Branches:
  refs/heads/yang22 55f87f477 -> a6abf16b4


minor improvement on limit


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/a6abf16b
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/a6abf16b
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/a6abf16b

Branch: refs/heads/yang22
Commit: a6abf16b4f7ff934330d5d6c9898f86c9b0295fd
Parents: 55f87f4
Author: Hongbin Ma 
Authored: Thu Dec 8 16:01:54 2016 +0800
Committer: Hongbin Ma 
Committed: Thu Dec 8 16:01:57 2016 +0800

--
 .../main/java/org/apache/kylin/storage/StorageContext.java  | 9 +++--
 .../java/org/apache/kylin/query/relnode/OLAPContext.java| 2 --
 .../java/org/apache/kylin/query/relnode/OLAPLimitRel.java   | 1 -
 3 files changed, 7 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/a6abf16b/core-storage/src/main/java/org/apache/kylin/storage/StorageContext.java
--
diff --git 
a/core-storage/src/main/java/org/apache/kylin/storage/StorageContext.java 
b/core-storage/src/main/java/org/apache/kylin/storage/StorageContext.java
index 719cab6..bc43a87 100644
--- a/core-storage/src/main/java/org/apache/kylin/storage/StorageContext.java
+++ b/core-storage/src/main/java/org/apache/kylin/storage/StorageContext.java
@@ -92,7 +92,12 @@ public class StorageContext {
 }
 
 public void setLimit(int l) {
-this.limit = l;
+if (l > limit) {
+//cases like : select price from (select * from kylin_sales limit 
10) limit 5000
+logger.info("Setting limit to {} but in current olap context, the 
limit is already {}, won't apply", l, limit);
+} else {
+this.limit = l;
+}
 }
 
 public int getOffset() {
@@ -126,7 +131,7 @@ public class StorageContext {
 if (tempPushDownLimit == Integer.MAX_VALUE) {
 return;
 }
-
+
 int pushDownLimitMax = 
KylinConfig.getInstanceFromEnv().getStoragePushDownLimitMax();
 if (!realization.supportsLimitPushDown()) {
 logger.info("Not enabling limit push down because cube storage 
type not supported");

http://git-wip-us.apache.org/repos/asf/kylin/blob/a6abf16b/query/src/main/java/org/apache/kylin/query/relnode/OLAPContext.java
--
diff --git 
a/query/src/main/java/org/apache/kylin/query/relnode/OLAPContext.java 
b/query/src/main/java/org/apache/kylin/query/relnode/OLAPContext.java
index 5fb2c5c..8278fb0 100644
--- a/query/src/main/java/org/apache/kylin/query/relnode/OLAPContext.java
+++ b/query/src/main/java/org/apache/kylin/query/relnode/OLAPContext.java
@@ -135,8 +135,6 @@ public class OLAPContext {
 // rewrite info
 public Map rewriteFields = new HashMap<>();
 
-public int limit;
-
 // hive query
 public String sql = "";
 

http://git-wip-us.apache.org/repos/asf/kylin/blob/a6abf16b/query/src/main/java/org/apache/kylin/query/relnode/OLAPLimitRel.java
--
diff --git 
a/query/src/main/java/org/apache/kylin/query/relnode/OLAPLimitRel.java 
b/query/src/main/java/org/apache/kylin/query/relnode/OLAPLimitRel.java
index d6569f6..f0af863 100644
--- a/query/src/main/java/org/apache/kylin/query/relnode/OLAPLimitRel.java
+++ b/query/src/main/java/org/apache/kylin/query/relnode/OLAPLimitRel.java
@@ -81,7 +81,6 @@ public class OLAPLimitRel extends SingleRel implements 
OLAPRel {
 Number limitValue = (Number) (((RexLiteral) 
localFetch).getValue());
 int limit = limitValue.intValue();
 this.context.storageContext.setLimit(limit);
-this.context.limit = limit;
 
 if (localOffset != null) {
 Number offsetValue = (Number) (((RexLiteral) 
localOffset).getValue());

[kylin] Git Push Summary

[mahongbin]

Repository: kylin
Updated Tags:  refs/tags/yinlian-1104 [deleted] f03f26af9

[kylin] Git Push Summary

[mahongbin]

Repository: kylin
Updated Tags:  refs/tags/SAIC-1104 [deleted] f03f26af9

[kylin] Git Push Summary

[mahongbin]

Repository: kylin
Updated Branches:
  refs/heads/yang22 [created] 76999735f

[kylin] Git Push Summary

[mahongbin]

Repository: kylin
Updated Tags:  refs/tags/yang22 [deleted] 76999735f

[2/3] kylin git commit: minor, generate item_count value for sample data

[mahongbin]

http://git-wip-us.apache.org/repos/asf/kylin/blob/d2aaf270/examples/sample_cube/data/DEFAULT.KYLIN_SALES.csv
--
diff --git a/examples/sample_cube/data/DEFAULT.KYLIN_SALES.csv 
b/examples/sample_cube/data/DEFAULT.KYLIN_SALES.csv
index 7349b37..e13b7ad 100644
--- a/examples/sample_cube/data/DEFAULT.KYLIN_SALES.csv
+++ b/examples/sample_cube/data/DEFAULT.KYLIN_SALES.csv
@@ -1,1 +1,1 @@
-0,2012-12-14,Others,88750,0,11,36.2828,0,1349,10002313,ANALYST,Beijing
-1,2012-08-28,Others,175750,0,13,23.8563,0,1927,10004376,ANALYST,Beijing
-2,2012-02-16,ABIN,148324,15,13,88.3418,0,1005,10006710,ADMIN,Shanghai
-3,2013-10-19,FP-non GTC,37831,0,13,47.3015,0,1209,10003717,ANALYST,Beijing
-4,2012-10-22,Others,140746,100,11,83.454,0,1154,10006076,ADMIN,Shanghai
-5,2013-01-25,FP-GTC,16509,0,-99,22.9896,0,1372,10007436,ADMIN,Shanghai
-6,2013-04-04,Others,963,0,13,88.5907,0,1648,10009869,MODELER,Hongkong
-7,2012-04-11,Others,15687,0,15,88.194,0,1866,1400,ADMIN,Shanghai
-8,2013-01-19,ABIN,60606,3,13,77.9727,0,1936,1687,MODELER,Hongkong
-9,2012-04-30,FP-non GTC,106246,0,14,52.295,0,1047,10009223,MODELER,Hongkong
-10,2013-02-03,Auction,45333,0,16,56.3584,0,1470,10006759,MODELER,Hongkong
-11,2012-09-02,FP-GTC,25147,0,-99,15.3553,0,1203,1062,ANALYST,Beijing
-12,2013-07-30,Others,24760,0,16,25.077,0,1382,10003539,ADMIN,Shanghai
-13,2012-04-17,Auction,31519,0,16,72.7384,0,1069,10001245,ADMIN,Shanghai
-14,2013-04-30,FP-GTC,51582,0,15,75.82,0,1085,10008148,ANALYST,Beijing
-15,2013-03-23,FP-GTC,16509,0,15,8.6653,0,1559,10008079,MODELER,Hongkong
-16,2013-04-01,ABIN,175750,3,16,64.2802,0,1507,10007797,MODELER,Hongkong
-17,2013-07-12,ABIN,50677,0,-99,24.5987,0,1938,10006224,ADMIN,Shanghai
-18,2012-04-23,FP-GTC,45333,0,5,72.6553,0,1863,10003824,MODELER,Hongkong
-19,2012-12-19,ABIN,3838,0,12,87.4142,0,1612,10007157,ADMIN,Shanghai
-20,2012-08-15,FP-GTC,62179,0,16,67.4238,0,1839,10003452,MODELER,Hongkong
-21,2013-08-22,Auction,1357,0,5,18.36,0,1303,1114,ANALYST,Beijing
-22,2013-08-23,ABIN,31519,0,5,67.6344,0,1877,10007417,ANALYST,Beijing
-23,2013-09-12,FP-GTC,223,0,14,71.7898,0,1848,1627,MODELER,Hongkong
-24,2012-03-14,FP-GTC,132939,0,15,26.1496,0,1815,10005292,MODELER,Hongkong
-25,2013-07-13,Others,67703,3,16,83.9661,0,1858,10008123,MODELER,Hongkong
-26,2013-12-15,FP-non GTC,100847,0,5,34.3451,0,1387,10007806,ADMIN,Shanghai
-27,2012-08-30,Auction,62179,0,11,21.0496,0,1393,10007360,ANALYST,Beijing
-28,2013-01-10,FP-GTC,48027,0,13,23.4142,0,1757,10002135,MODELER,Hongkong
-29,2013-01-25,ABIN,145970,0,5,26.7842,0,1402,10009682,ANALYST,Beijing
-30,2012-07-05,Auction,48027,0,-99,46.732,0,1972,10009522,MODELER,Hongkong
-31,2012-11-07,FP-GTC,20213,0,-99,19.727,0,1873,10009944,ANALYST,Beijing
-32,2013-06-20,ABIN,100847,0,5,76.2783,0,1409,10005395,ADMIN,Shanghai
-33,2012-10-08,ABIN,26262,0,5,48.7447,0,1029,10001836,ADMIN,Shanghai
-34,2013-04-30,ABIN,87118,0,15,63.2187,0,1529,1928,ANALYST,Beijing
-35,2013-04-24,FP-non GTC,20213,0,16,39.5057,0,1329,1768,ANALYST,Beijing
-36,2012-05-13,Others,164261,0,13,84.1246,0,1688,1727,ADMIN,Shanghai
-37,2013-08-02,ABIN,31519,0,5,37.1504,0,1905,1719,MODELER,Hongkong
-38,2012-04-03,Auction,164261,0,11,92.0974,0,1341,10001337,ADMIN,Shanghai
-39,2012-09-02,FP-GTC,31519,0,14,87.7829,0,1460,10008484,ANALYST,Beijing
-40,2013-10-05,ABIN,13987,0,16,74.0719,0,1381,10002975,MODELER,Hongkong
-41,2012-01-13,FP-GTC,48027,0,15,71.2637,0,1915,10005778,ADMIN,Shanghai
-42,2013-03-01,Others,13836,0,14,16.7288,0,1953,10009385,ADMIN,Shanghai
-43,2012-09-12,FP-non GTC,16509,0,5,12.2933,0,1277,10002286,MODELER,Hongkong
-44,2012-02-07,Auction,45333,0,5,64.977,0,1040,10005583,MODELER,Hongkong
-45,2013-12-14,FP-non 
GTC,158798,0,16,72.4413,0,1500,10007635,MODELER,Hongkong
-46,2013-10-13,Auction,31519,0,5,79.3053,0,1816,10006423,MODELER,Hongkong
-47,2012-06-11,ABIN,4943,0,5,11.6942,0,1696,10001237,ANALYST,Beijing
-48,2012-10-18,ABIN,80053,0,-99,54.0933,0,1604,10006851,MODELER,Hongkong
-49,2012-11-03,Auction,20886,0,5,9.8258,0,1764,10001220,MODELER,Hongkong
-50,2012-01-15,Auction,44079,0,14,13.0371,0,1343,10001259,MODELER,Hongkong
-51,2012-07-03,FP-GTC,159184,0,-99,92.5314,0,1724,10009008,ADMIN,Shanghai
-52,2012-02-16,FP-GTC,314,0,13,49.1825,0,1334,10004005,ADMIN,Shanghai
-53,2012-06-17,FP-non GTC,31387,3,12,0.6677,0,1603,10002595,ANALYST,Beijing
-54,2012-07-15,FP-non 
GTC,32876,0,-99,50.9634,0,1248,10006890,ANALYST,Beijing
-55,2012-07-24,FP-GTC,150265,15,16,57.5645,0,1653,10007269,ADMIN,Shanghai
-56,2012-10-06,Auction,145970,0,5,44.3091,0,1082,10001640,ADMIN,Shanghai
-57,2013-06-14,ABIN,80053,0,5,20.2388,0,1632,10002068,MODELER,Hongkong
-58,2013-10-10,FP-non 

[3/3] kylin git commit: minor, generate item_count value for sample data

[mahongbin]

minor, generate item_count value for sample data

Signed-off-by: Hongbin Ma 


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/d2aaf270
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/d2aaf270
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/d2aaf270

Branch: refs/heads/master
Commit: d2aaf2705a9035be9dbef8205fc9a1358b4316c2
Parents: 9968c22
Author: Yiming Liu 
Authored: Fri Dec 2 16:39:00 2016 +0800
Committer: Hongbin Ma 
Committed: Tue Dec 6 13:50:28 2016 +0800

--
 .../sample_cube/data/DEFAULT.KYLIN_SALES.csv| 2 -
 1 file changed, 1 insertions(+), 1 deletions(-)
--

kylin git commit: KYLIN-2246 redesign the way to decide layer cubing reducer count

[mahongbin]

Repository: kylin
Updated Branches:
  refs/heads/KYLIN-2246 [created] 815887e73


KYLIN-2246 redesign the way to decide layer cubing reducer count


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/815887e7
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/815887e7
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/815887e7

Branch: refs/heads/KYLIN-2246
Commit: 815887e73a5c3b0852b6cf5650400235797d7ce9
Parents: 59a30f6
Author: Hongbin Ma 
Authored: Mon Dec 5 21:02:36 2016 +0800
Committer: Hongbin Ma 
Committed: Mon Dec 5 21:02:44 2016 +0800

--
 .../kylin/cube/cuboid/CuboidScheduler.java  | 31 +++-
 .../kylin/engine/mr/common/CubeStatsReader.java | 26 ++-
 .../apache/kylin/engine/mr/steps/CuboidJob.java | 52 +
 .../engine/mr/steps/LayerReduerNumSizing.java   | 82 
 .../kylin/engine/mr/steps/MergeCuboidJob.java   |  2 +-
 5 files changed, 138 insertions(+), 55 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/815887e7/core-cube/src/main/java/org/apache/kylin/cube/cuboid/CuboidScheduler.java
--
diff --git 
a/core-cube/src/main/java/org/apache/kylin/cube/cuboid/CuboidScheduler.java 
b/core-cube/src/main/java/org/apache/kylin/cube/cuboid/CuboidScheduler.java
index bd6a37a..733aded 100644
--- a/core-cube/src/main/java/org/apache/kylin/cube/cuboid/CuboidScheduler.java
+++ b/core-cube/src/main/java/org/apache/kylin/cube/cuboid/CuboidScheduler.java
@@ -18,7 +18,7 @@
 
 package org.apache.kylin.cube.cuboid;
 
-/** 
+/**
  */
 
 import java.util.Collections;
@@ -31,6 +31,7 @@ import java.util.concurrent.ConcurrentHashMap;
 import org.apache.kylin.cube.model.AggregationGroup;
 import org.apache.kylin.cube.model.CubeDesc;
 
+import com.google.common.base.Preconditions;
 import com.google.common.collect.Lists;
 import com.google.common.collect.Sets;
 
@@ -39,6 +40,7 @@ public class CuboidScheduler {
 private final CubeDesc cubeDesc;
 private final long max;
 private final Map cache;
+private List cuboidsByLayer;
 
 public CuboidScheduler(CubeDesc cubeDesc) {
 this.cubeDesc = cubeDesc;
@@ -232,4 +234,31 @@ public class CuboidScheduler {
 getSubCuboidIds(cuboidId, result);
 }
 }
+
+public List getCuboidsByLayer() {
+if (cuboidsByLayer != null) {
+return cuboidsByLayer;
+}
+
+int totalNum = 0;
+int layerNum = cubeDesc.getBuildLevel();
+cuboidsByLayer = Lists.newArrayList();
+
+
cuboidsByLayer.add(Collections.singletonList(Cuboid.getBaseCuboidId(cubeDesc)));
+totalNum++;
+
+for (int i = 1; i <= layerNum; i++) {
+List lastLayer = cuboidsByLayer.get(i - 1);
+List newLayer = Lists.newArrayList();
+for (Long parent : lastLayer) {
+newLayer.addAll(getSpanningCuboid(parent));
+}
+cuboidsByLayer.add(newLayer);
+totalNum += newLayer.size();
+}
+
+int size = getAllCuboidIds().size();
+Preconditions.checkState(totalNum == size, "total Num: " + totalNum + 
" actual size: " + size);
+return cuboidsByLayer;
+}
 }

http://git-wip-us.apache.org/repos/asf/kylin/blob/815887e7/engine-mr/src/main/java/org/apache/kylin/engine/mr/common/CubeStatsReader.java
--
diff --git 
a/engine-mr/src/main/java/org/apache/kylin/engine/mr/common/CubeStatsReader.java
 
b/engine-mr/src/main/java/org/apache/kylin/engine/mr/common/CubeStatsReader.java
index c917cfb..1cf5da6 100644
--- 
a/engine-mr/src/main/java/org/apache/kylin/engine/mr/common/CubeStatsReader.java
+++ 
b/engine-mr/src/main/java/org/apache/kylin/engine/mr/common/CubeStatsReader.java
@@ -29,6 +29,7 @@ import java.util.Collections;
 import java.util.List;
 import java.util.Map;
 
+import org.apache.commons.lang.StringUtils;
 import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.fs.Path;
 import org.apache.hadoop.io.BytesWritable;
@@ -75,9 +76,11 @@ public class CubeStatsReader {
 final int mapperNumberOfFirstBuild; // becomes meaningless after merge
 final double mapperOverlapRatioOfFirstBuild; // becomes meaningless after 
merge
 final Map cuboidRowEstimatesHLL;
+final CuboidScheduler cuboidScheduler;
 
 public CubeStatsReader(CubeSegment cubeSegment, KylinConfig kylinConfig) 
throws IOException {
 ResourceStore store = ResourceStore.getStore(kylinConfig);
+cuboidScheduler = new CuboidScheduler(cubeSegment.getCubeDesc());
 String statsKey = cubeSegment.getStatisticsResourcePath();
 

kylin git commit: KYLIN-2212 add more test queries

[mahongbin]

Repository: kylin
Updated Branches:
  refs/heads/master 8f3239bf9 -> af429e5cb


KYLIN-2212 add more test queries


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/af429e5c
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/af429e5c
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/af429e5c

Branch: refs/heads/master
Commit: af429e5cbbda2a64159b25bc5ce4ad70f2f1a1f2
Parents: 8f3239b
Author: Hongbin Ma 
Authored: Mon Dec 5 15:05:35 2016 +0800
Committer: Hongbin Ma 
Committed: Mon Dec 5 15:05:35 2016 +0800

--
 .../metadata/filter/LogicalTupleFilter.java |  3 +-
 .../src/test/resources/query/sql/query101.sql   | 13 
 .../src/test/resources/query/sql/query102.sql   | 13 
 .../src/test/resources/query/sql/query103.sql   | 13 
 .../test/resources/query/sql_like/query21.sql   | 31 
 5 files changed, 71 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/af429e5c/core-metadata/src/main/java/org/apache/kylin/metadata/filter/LogicalTupleFilter.java
--
diff --git 
a/core-metadata/src/main/java/org/apache/kylin/metadata/filter/LogicalTupleFilter.java
 
b/core-metadata/src/main/java/org/apache/kylin/metadata/filter/LogicalTupleFilter.java
index 61657fb..1744309 100644
--- 
a/core-metadata/src/main/java/org/apache/kylin/metadata/filter/LogicalTupleFilter.java
+++ 
b/core-metadata/src/main/java/org/apache/kylin/metadata/filter/LogicalTupleFilter.java
@@ -64,8 +64,7 @@ public class LogicalTupleFilter extends TupleFilter {
 public TupleFilter reverse() {
 switch (operator) {
 case NOT:
-throw new IllegalStateException("not( not in ()) is invalid 
syntax");
-//return reverseNestedNots(this, 0);
+throw new IllegalStateException("NOT will be replaced in 
org.apache.kylin.query.relnode.OLAPFilterRel.TupleFilterVisitor");
 case AND:
 case OR:
 LogicalTupleFilter reverse = new 
LogicalTupleFilter(REVERSE_OP_MAP.get(operator));

http://git-wip-us.apache.org/repos/asf/kylin/blob/af429e5c/kylin-it/src/test/resources/query/sql/query101.sql
--
diff --git a/kylin-it/src/test/resources/query/sql/query101.sql 
b/kylin-it/src/test/resources/query/sql/query101.sql
new file mode 100644
index 000..fb42bca
--- /dev/null
+++ b/kylin-it/src/test/resources/query/sql/query101.sql
@@ -0,0 +1,13 @@
+
+select meta_categ_name, count(1) as cnt, sum(price) as GMV 
+
+ from test_kylin_fact 
+ left JOIN edw.test_cal_dt as test_cal_dt
+ ON test_kylin_fact.cal_dt = test_cal_dt.cal_dt
+ left JOIN test_category_groupings
+ ON test_kylin_fact.leaf_categ_id = test_category_groupings.leaf_categ_id AND 
test_kylin_fact.lstg_site_id = test_category_groupings.site_id
+ left JOIN edw.test_sites as test_sites
+ ON test_kylin_fact.lstg_site_id = test_sites.site_id
+
+ where not ( meta_categ_name not in ('', 'a','Computers') )
+ group by meta_categ_name 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/kylin/blob/af429e5c/kylin-it/src/test/resources/query/sql/query102.sql
--
diff --git a/kylin-it/src/test/resources/query/sql/query102.sql 
b/kylin-it/src/test/resources/query/sql/query102.sql
new file mode 100644
index 000..bd1e15e
--- /dev/null
+++ b/kylin-it/src/test/resources/query/sql/query102.sql
@@ -0,0 +1,13 @@
+
+select meta_categ_name, count(1) as cnt, sum(price) as GMV 
+
+ from test_kylin_fact 
+ left JOIN edw.test_cal_dt as test_cal_dt
+ ON test_kylin_fact.cal_dt = test_cal_dt.cal_dt
+ left JOIN test_category_groupings
+ ON test_kylin_fact.leaf_categ_id = test_category_groupings.leaf_categ_id AND 
test_kylin_fact.lstg_site_id = test_category_groupings.site_id
+ left JOIN edw.test_sites as test_sites
+ ON test_kylin_fact.lstg_site_id = test_sites.site_id
+
+ where not ( meta_categ_name not in ('', 'a','Computers') and meta_categ_name 
not in ('Crafts','Computers'))
+ group by meta_categ_name 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/kylin/blob/af429e5c/kylin-it/src/test/resources/query/sql/query103.sql
--
diff --git a/kylin-it/src/test/resources/query/sql/query103.sql 
b/kylin-it/src/test/resources/query/sql/query103.sql
new file mode 100644
index 000..c5f9bf9
--- /dev/null
+++ b/kylin-it/src/test/resources/query/sql/query103.sql
@@ -0,0 +1,13 @@
+
+select meta_categ_name, count(1) as cnt, sum(price) as GMV 
+
+ from test_kylin_fact 
+ left JOIN edw.test_cal_dt as test_cal_dt
+ ON test_kylin_fact.cal_dt = 

kylin git commit: KYLIN-2236 let query cache honor backdoortoggles in query request

[mahongbin]

Repository: kylin
Updated Branches:
  refs/heads/master 955615262 -> 8f3239bf9


KYLIN-2236 let query cache honor backdoortoggles in query request


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/8f3239bf
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/8f3239bf
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/8f3239bf

Branch: refs/heads/master
Commit: 8f3239bf9ea4f1abd8d2c814967d425d4da68a45
Parents: 9556152
Author: Hongbin Ma 
Authored: Mon Dec 5 13:43:03 2016 +0800
Committer: Hongbin Ma 
Committed: Mon Dec 5 13:43:03 2016 +0800

--
 .../org/apache/kylin/jdbc/ITJDBCDriverTest.java | 30 ++
 .../apache/kylin/rest/request/SQLRequest.java   | 63 +++-
 .../apache/kylin/rest/service/QueryService.java |  1 +
 3 files changed, 53 insertions(+), 41 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/8f3239bf/kylin-it/src/test/java/org/apache/kylin/jdbc/ITJDBCDriverTest.java
--
diff --git a/kylin-it/src/test/java/org/apache/kylin/jdbc/ITJDBCDriverTest.java 
b/kylin-it/src/test/java/org/apache/kylin/jdbc/ITJDBCDriverTest.java
index 4c847bf..2f8991b 100644
--- a/kylin-it/src/test/java/org/apache/kylin/jdbc/ITJDBCDriverTest.java
+++ b/kylin-it/src/test/java/org/apache/kylin/jdbc/ITJDBCDriverTest.java
@@ -261,6 +261,36 @@ public class ITJDBCDriverTest extends 
HBaseMetadataTestCase {
 
 }
 
+
+@Test
+public void testResultSetWithMaxRows() throws Exception {
+String sql = "select LSTG_FORMAT_NAME, sum(price) as GMV, count(1) as 
TRANS_CNT from test_kylin_fact \n" + " group by LSTG_FORMAT_NAME ";
+
+Connection conn = getConnection();
+Statement statement = conn.createStatement();
+statement.setMaxRows(2);
+
+statement.execute(sql);
+
+ResultSet rs = statement.getResultSet();
+
+int count = 0;
+while (rs.next()) {
+count++;
+String lstg = rs.getString(1);
+double gmv = rs.getDouble(2);
+int trans_count = rs.getInt(3);
+
+System.out.println("Get a line: LSTG_FORMAT_NAME=" + lstg + ", 
GMV=" + gmv + ", TRANS_CNT=" + trans_count);
+}
+
+Assert.assertTrue(count == 2);
+statement.close();
+rs.close();
+conn.close();
+
+}
+
 private static class SystemPropertiesOverride {
 HashMap backup = new HashMap();
 

http://git-wip-us.apache.org/repos/asf/kylin/blob/8f3239bf/server-base/src/main/java/org/apache/kylin/rest/request/SQLRequest.java
--
diff --git 
a/server-base/src/main/java/org/apache/kylin/rest/request/SQLRequest.java 
b/server-base/src/main/java/org/apache/kylin/rest/request/SQLRequest.java
index 96f5faa..bd8b7e2 100644
--- a/server-base/src/main/java/org/apache/kylin/rest/request/SQLRequest.java
+++ b/server-base/src/main/java/org/apache/kylin/rest/request/SQLRequest.java
@@ -83,50 +83,31 @@ public class SQLRequest implements Serializable {
 this.acceptPartial = acceptPartial;
 }
 
+
 @Override
-public int hashCode() {
-final int prime = 31;
-int result = 1;
-result = prime * result + (acceptPartial ? 1231 : 1237);
-result = prime * result + ((offset == null) ? 0 : offset.hashCode());
-result = prime * result + ((limit == null) ? 0 : limit.hashCode());
-result = prime * result + ((project == null) ? 0 : project.hashCode());
-result = prime * result + ((sql == null) ? 0 : sql.hashCode());
-return result;
+public boolean equals(Object o) {
+if (this == o) return true;
+if (o == null || getClass() != o.getClass()) return false;
+
+SQLRequest that = (SQLRequest) o;
+
+if (acceptPartial != that.acceptPartial) return false;
+if (sql != null ? !sql.equals(that.sql) : that.sql != null) return 
false;
+if (project != null ? !project.equals(that.project) : that.project != 
null) return false;
+if (offset != null ? !offset.equals(that.offset) : that.offset != 
null) return false;
+if (limit != null ? !limit.equals(that.limit) : that.limit != null) 
return false;
+return backdoorToggles != null ? 
backdoorToggles.equals(that.backdoorToggles) : that.backdoorToggles == null;
+
 }
 
 @Override
-public boolean equals(Object obj) {
-if (this == obj)
-return true;
-if (obj == null)
-return false;
-if (getClass() != obj.getClass())
-return false;
-SQLRequest other = (SQLRequest) obj;
-if (acceptPartial != other.acceptPartial)
-

kylin git commit: minor, add setter for BuiltInFunctionTupleFilter's reversed field

[mahongbin]

Repository: kylin
Updated Branches:
  refs/heads/master 00a4c5064 -> 955615262


minor, add setter for BuiltInFunctionTupleFilter's reversed field


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/95561526
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/95561526
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/95561526

Branch: refs/heads/master
Commit: 9556152622599a87610f16ae8f06185e24ec02d6
Parents: 00a4c50
Author: Hongbin Ma 
Authored: Mon Dec 5 10:58:31 2016 +0800
Committer: Hongbin Ma 
Committed: Mon Dec 5 10:58:31 2016 +0800

--
 .../apache/kylin/metadata/filter/BuiltInFunctionTupleFilter.java | 4 
 1 file changed, 4 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/95561526/core-metadata/src/main/java/org/apache/kylin/metadata/filter/BuiltInFunctionTupleFilter.java
--
diff --git 
a/core-metadata/src/main/java/org/apache/kylin/metadata/filter/BuiltInFunctionTupleFilter.java
 
b/core-metadata/src/main/java/org/apache/kylin/metadata/filter/BuiltInFunctionTupleFilter.java
index b678394..90123c4 100755
--- 
a/core-metadata/src/main/java/org/apache/kylin/metadata/filter/BuiltInFunctionTupleFilter.java
+++ 
b/core-metadata/src/main/java/org/apache/kylin/metadata/filter/BuiltInFunctionTupleFilter.java
@@ -109,6 +109,10 @@ public class BuiltInFunctionTupleFilter extends 
FunctionTupleFilter {
 return isReversed;
 }
 
+public void setReversed(boolean reversed) {
+this.isReversed = reversed;
+}
+
 @Override
 public void addChild(TupleFilter child) {
 if (child instanceof ColumnTupleFilter || child instanceof 
BuiltInFunctionTupleFilter) {

[1/3] kylin git commit: minor: normalize scientific decimal values

[mahongbin]

Repository: kylin
Updated Branches:
  refs/heads/master eb686a942 -> 00a4c5064


minor: normalize scientific decimal values


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/fe9efa86
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/fe9efa86
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/fe9efa86

Branch: refs/heads/master
Commit: fe9efa86cd34da31ec41dc176140ee4ee2f49fed
Parents: eb686a9
Author: Hongbin Ma 
Authored: Sun Dec 4 21:26:18 2016 +0800
Committer: Hongbin Ma 
Committed: Sun Dec 4 21:26:18 2016 +0800

--
 .../java/org/apache/kylin/metadata/tuple/Tuple.java | 12 +++-
 1 file changed, 11 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/fe9efa86/core-metadata/src/main/java/org/apache/kylin/metadata/tuple/Tuple.java
--
diff --git 
a/core-metadata/src/main/java/org/apache/kylin/metadata/tuple/Tuple.java 
b/core-metadata/src/main/java/org/apache/kylin/metadata/tuple/Tuple.java
index 721a719..d51ae9e 100644
--- a/core-metadata/src/main/java/org/apache/kylin/metadata/tuple/Tuple.java
+++ b/core-metadata/src/main/java/org/apache/kylin/metadata/tuple/Tuple.java
@@ -115,6 +115,8 @@ public class Tuple implements ITuple {
 // BigDecimal during cube build for best precision
 if ("double".equals(dataType) && fieldValue instanceof BigDecimal) {
 fieldValue = ((BigDecimal) fieldValue).doubleValue();
+} else if ("decimal".equals(dataType) && fieldValue instanceof 
BigDecimal) {
+fieldValue = normalizeDecimal((BigDecimal) fieldValue);
 } else if ("integer".equals(dataType) && fieldValue instanceof Number) 
{
 fieldValue = ((Number) fieldValue).intValue();
 } else if ("smallint".equals(dataType) && fieldValue instanceof 
Number) {
@@ -145,6 +147,14 @@ public class Tuple implements ITuple {
 return o;
 }
 
+private static BigDecimal normalizeDecimal(BigDecimal input) {
+if (input.scale() < 0) {
+return input.setScale(0);
+} else {
+return input;
+}
+}
+
 public boolean hasColumn(TblColRef column) {
 return info.hasColumn(column);
 }
@@ -199,7 +209,7 @@ public class Tuple implements ITuple {
 case "double":
 return Double.valueOf(strValue);
 case "decimal":
-return new BigDecimal(strValue);
+return normalizeDecimal(new BigDecimal(strValue));
 case "float":
 return Float.valueOf(strValue);
 case "boolean":

[2/3] kylin git commit: minor, refine MassInTupleFilter

[mahongbin]

minor, refine MassInTupleFilter

Signed-off-by: Hongbin Ma 


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/a03eec88
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/a03eec88
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/a03eec88

Branch: refs/heads/master
Commit: a03eec88f3b4e60d1f1d6bd2fd5922a97aec23f5
Parents: fe9efa8
Author: Roger Shi 
Authored: Sun Dec 4 16:28:47 2016 +0800
Committer: Hongbin Ma 
Committed: Sun Dec 4 21:28:17 2016 +0800

--
 .../metadata/filter/UDF/MassInTupleFilter.java  | 34 +---
 1 file changed, 30 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/a03eec88/core-metadata/src/main/java/org/apache/kylin/metadata/filter/UDF/MassInTupleFilter.java
--
diff --git 
a/core-metadata/src/main/java/org/apache/kylin/metadata/filter/UDF/MassInTupleFilter.java
 
b/core-metadata/src/main/java/org/apache/kylin/metadata/filter/UDF/MassInTupleFilter.java
index 29c5550..e4e311e 100644
--- 
a/core-metadata/src/main/java/org/apache/kylin/metadata/filter/UDF/MassInTupleFilter.java
+++ 
b/core-metadata/src/main/java/org/apache/kylin/metadata/filter/UDF/MassInTupleFilter.java
@@ -19,6 +19,7 @@
 package org.apache.kylin.metadata.filter.UDF;
 
 import java.nio.ByteBuffer;
+import java.util.ArrayList;
 import java.util.Collection;
 
 import org.apache.kylin.common.KylinConfig;
@@ -40,7 +41,6 @@ import com.google.common.base.Preconditions;
 import com.google.common.collect.Lists;
 
 public class MassInTupleFilter extends FunctionTupleFilter {
-
 public static final Logger logger = 
LoggerFactory.getLogger(MassInTupleFilter.class);
 public static MassInValueProviderFactory VALUE_PROVIDER_FACTORY = null;
 
@@ -56,6 +56,16 @@ public class MassInTupleFilter extends FunctionTupleFilter {
 super(Lists. newArrayList(), 
TupleFilter.FilterOperatorEnum.MASSIN);
 }
 
+public MassInTupleFilter(MassInTupleFilter filter) {
+super(new ArrayList(filter.children), filter.operator);
+this.valueProvider = filter.getValueProvider();
+this.column = filter.getColumn();
+this.filterTableName = filter.getFilterTableName();
+this.filterTableResourceIdentifier = 
filter.getFilterTableResourceIdentifier();
+this.filterTableType = filter.getFilterTableType();
+this.reverse = filter.isReverse();
+}
+
 @Override
 public boolean evaluate(IEvaluatableTuple tuple, IFilterCodeSystem cs) {
 Preconditions.checkNotNull(tuple);
@@ -132,6 +142,7 @@ public class MassInTupleFilter extends FunctionTupleFilter {
 BytesUtil.writeUTFString(filterTableName, buffer);
 BytesUtil.writeUTFString(filterTableResourceIdentifier, buffer);
 BytesUtil.writeUTFString(filterTableType.toString(), buffer);
+BytesUtil.writeUTFString(String.valueOf(reverse), buffer);
 }
 
 @Override
@@ -139,6 +150,7 @@ public class MassInTupleFilter extends FunctionTupleFilter {
 filterTableName = BytesUtil.readUTFString(buffer);
 filterTableResourceIdentifier = BytesUtil.readUTFString(buffer);
 filterTableType = 
Functions.FilterTableType.valueOf(BytesUtil.readUTFString(buffer));
+reverse = Boolean.valueOf(BytesUtil.readUTFString(buffer));
 }
 
 public static boolean containsMassInTupleFilter(TupleFilter filter) {
@@ -166,8 +178,22 @@ public class MassInTupleFilter extends FunctionTupleFilter 
{
 
 @Override
 protected Object clone() throws CloneNotSupportedException {
-MassInTupleFilter result = new MassInTupleFilter();
-result.setReverse(this.isReverse());
-return result;
+return new MassInTupleFilter(this);
+}
+
+public MassInValueProvider getValueProvider() {
+return valueProvider;
+}
+
+public String getFilterTableName() {
+return filterTableName;
+}
+
+public String getFilterTableResourceIdentifier() {
+return filterTableResourceIdentifier;
+}
+
+public Functions.FilterTableType getFilterTableType() {
+return filterTableType;
 }
 }

[3/3] kylin git commit: minor, change isReverse visibility in BuiltInFunctionTupleFilter.java

[mahongbin]

minor, change isReverse visibility in BuiltInFunctionTupleFilter.java


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/00a4c506
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/00a4c506
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/00a4c506

Branch: refs/heads/master
Commit: 00a4c5064bfe894bb8c106a214277d4d1d0f3c70
Parents: a03eec8
Author: Hongbin Ma 
Authored: Sun Dec 4 21:41:07 2016 +0800
Committer: Hongbin Ma 
Committed: Sun Dec 4 21:41:07 2016 +0800

--
 .../apache/kylin/metadata/filter/BuiltInFunctionTupleFilter.java   | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/00a4c506/core-metadata/src/main/java/org/apache/kylin/metadata/filter/BuiltInFunctionTupleFilter.java
--
diff --git 
a/core-metadata/src/main/java/org/apache/kylin/metadata/filter/BuiltInFunctionTupleFilter.java
 
b/core-metadata/src/main/java/org/apache/kylin/metadata/filter/BuiltInFunctionTupleFilter.java
index f6e687b..b678394 100755
--- 
a/core-metadata/src/main/java/org/apache/kylin/metadata/filter/BuiltInFunctionTupleFilter.java
+++ 
b/core-metadata/src/main/java/org/apache/kylin/metadata/filter/BuiltInFunctionTupleFilter.java
@@ -47,7 +47,7 @@ public class BuiltInFunctionTupleFilter extends 
FunctionTupleFilter {
 protected Method method;
 protected List methodParams;
 protected boolean isValidFunc = false;
-private boolean isReversed = false;
+protected boolean isReversed = false;
 
 public BuiltInFunctionTupleFilter(String name) {
 this(name, null);

kylin git commit: KYLIN-2240 Add a toggle to ignore all cube signature inconsistency temporally

[mahongbin]

Repository: kylin
Updated Branches:
  refs/heads/master 3091f06a6 -> e562aafee


KYLIN-2240 Add a toggle to ignore all cube signature inconsistency temporally


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/e562aafe
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/e562aafe
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/e562aafe

Branch: refs/heads/master
Commit: e562aafee10c8bbd125937503d42b958c1843aad
Parents: 3091f06
Author: Hongbin Ma 
Authored: Fri Dec 2 13:33:02 2016 +0800
Committer: Hongbin Ma 
Committed: Fri Dec 2 13:33:18 2016 +0800

--
 .../kylin/common/debug/BackdoorToggles.java   | 18 ++
 .../java/org/apache/kylin/jdbc/IRemoteClient.java |  3 ++-
 .../java/org/apache/kylin/jdbc/KylinClient.java   |  7 ---
 .../org/apache/kylin/jdbc/KylinResultSet.java |  9 -
 .../org/apache/kylin/jdbc/json/QueryRequest.java  | 12 
 .../java/org/apache/kylin/jdbc/DummyClient.java   |  3 ++-
 .../apache/kylin/rest/service/QueryService.java   | 11 ++-
 7 files changed, 56 insertions(+), 7 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/e562aafe/core-common/src/main/java/org/apache/kylin/common/debug/BackdoorToggles.java
--
diff --git 
a/core-common/src/main/java/org/apache/kylin/common/debug/BackdoorToggles.java 
b/core-common/src/main/java/org/apache/kylin/common/debug/BackdoorToggles.java
index f7c90aa..28f7697 100644
--- 
a/core-common/src/main/java/org/apache/kylin/common/debug/BackdoorToggles.java
+++ 
b/core-common/src/main/java/org/apache/kylin/common/debug/BackdoorToggles.java
@@ -79,6 +79,14 @@ public class BackdoorToggles {
 }
 }
 
+public static Integer getStatementMaxRows() {
+String v = getString(ATTR_STATEMENT_MAX_ROWS);
+if (v == null)
+return null;
+else
+return Integer.valueOf(v);
+}
+
 private static String getString(String key) {
 Map toggles = _backdoorToggles.get();
 if (toggles == null) {
@@ -183,4 +191,14 @@ public class BackdoorToggles {
  */
 public final static String DEBUG_TOGGLE_SHARD_ASSIGNMENT = 
"DEBUG_TOGGLE_SHARD_ASSIGNMENT";
 
+// properties on statement may go with this "channel" too
+/**
+ * set ATTR_STATEMENT_MAX_ROWS="maxRows" to statement's max rows property
+ *
+ example:(put it into request body)
+ "backdoorToggles": {
+ "ATTR_STATEMENT_MAX_ROWS": "10"
+ }
+ */
+public final static String ATTR_STATEMENT_MAX_ROWS = 
"ATTR_STATEMENT_MAX_ROWS";
 }

http://git-wip-us.apache.org/repos/asf/kylin/blob/e562aafe/jdbc/src/main/java/org/apache/kylin/jdbc/IRemoteClient.java
--
diff --git a/jdbc/src/main/java/org/apache/kylin/jdbc/IRemoteClient.java 
b/jdbc/src/main/java/org/apache/kylin/jdbc/IRemoteClient.java
index b6a13e5..dfd8d76 100644
--- a/jdbc/src/main/java/org/apache/kylin/jdbc/IRemoteClient.java
+++ b/jdbc/src/main/java/org/apache/kylin/jdbc/IRemoteClient.java
@@ -21,6 +21,7 @@ package org.apache.kylin.jdbc;
 import java.io.Closeable;
 import java.io.IOException;
 import java.util.List;
+import java.util.Map;
 
 import org.apache.calcite.avatica.AvaticaParameter;
 import org.apache.calcite.avatica.ColumnMetaData;
@@ -51,6 +52,6 @@ public interface IRemoteClient extends Closeable {
 /**
  * Execute query remotely and get back result.
  */
-public QueryResult executeQuery(String sql, List params, 
List paramValues) throws IOException;
+public QueryResult executeQuery(String sql, List params, 
List paramValues, Map queryToggles) throws IOException;
 
 }

http://git-wip-us.apache.org/repos/asf/kylin/blob/e562aafe/jdbc/src/main/java/org/apache/kylin/jdbc/KylinClient.java
--
diff --git a/jdbc/src/main/java/org/apache/kylin/jdbc/KylinClient.java 
b/jdbc/src/main/java/org/apache/kylin/jdbc/KylinClient.java
index 2d06a92..86c3a5b 100644
--- a/jdbc/src/main/java/org/apache/kylin/jdbc/KylinClient.java
+++ b/jdbc/src/main/java/org/apache/kylin/jdbc/KylinClient.java
@@ -321,9 +321,9 @@ public class KylinClient implements IRemoteClient {
 }
 
 @Override
-public QueryResult executeQuery(String sql, List params, 
List paramValues) throws IOException {
+public QueryResult executeQuery(String sql, List params, 
List paramValues, Map queryToggles) throws IOException {
 
-SQLResponseStub queryResp = executeKylinQuery(sql, 
convertParameters(params, paramValues));
+SQLResponseStub queryResp = executeKylinQuery(sql, 

kylin git commit: measure encoding length bug

[mahongbin]

Repository: kylin
Updated Branches:
  refs/heads/master 58224921d -> 1a5295bad


measure encoding length bug

Signed-off-by: Hongbin Ma 


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/1a5295ba
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/1a5295ba
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/1a5295ba

Branch: refs/heads/master
Commit: 1a5295bad21fc18766bb3c7ba6eee59678d42185
Parents: 5822492
Author: luguosheng <550175...@qq.com>
Authored: Thu Dec 1 17:54:35 2016 +0800
Committer: Hongbin Ma 
Committed: Thu Dec 1 18:17:35 2016 +0800

--
 webapp/app/js/controllers/cubeAdvanceSetting.js | 4 ++--
 webapp/app/js/controllers/cubeEdit.js   | 2 +-
 webapp/app/js/controllers/cubeMeasures.js   | 8 
 3 files changed, 7 insertions(+), 7 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/1a5295ba/webapp/app/js/controllers/cubeAdvanceSetting.js
--
diff --git a/webapp/app/js/controllers/cubeAdvanceSetting.js 
b/webapp/app/js/controllers/cubeAdvanceSetting.js
index f2b36fe..8192419 100644
--- a/webapp/app/js/controllers/cubeAdvanceSetting.js
+++ b/webapp/app/js/controllers/cubeAdvanceSetting.js
@@ -47,7 +47,7 @@ KylinApp.controller('CubeAdvanceSettingCtrl', function 
($scope, $modal,cubeConfi
 var _encoding = item.encoding;
 var _valueLength ;
 var baseKey=item.encoding.replace(/:\d+/,'');
-if(needLengthKeyList.indexOf(baseKey)>=-1){
+if(needLengthKeyList.indexOf(baseKey)!=-1){
   var result=/:(\d+)/.exec(item.encoding);
   _valueLength=result?result[1]:0;
 }
@@ -78,7 +78,7 @@ KylinApp.controller('CubeAdvanceSettingCtrl', function 
($scope, $modal,cubeConfi
 var version=$scope.getTypeVersion(item.encoding);
 var encodingType=$scope.removeVersion(item.encoding);
 
-if(needLengthKeyList.indexOf(encodingType)>=-1){
+if(needLengthKeyList.indexOf(encodingType)!=-1){
   encoding = encodingType+":"+item.valueLength;
 }else{
   encoding = encodingType;

http://git-wip-us.apache.org/repos/asf/kylin/blob/1a5295ba/webapp/app/js/controllers/cubeEdit.js
--
diff --git a/webapp/app/js/controllers/cubeEdit.js 
b/webapp/app/js/controllers/cubeEdit.js
index 0704a62..85bd4b1 100755
--- a/webapp/app/js/controllers/cubeEdit.js
+++ b/webapp/app/js/controllers/cubeEdit.js
@@ -78,7 +78,7 @@ KylinApp.controller('CubeEditCtrl', function ($scope, $q, 
$routeParams, $locatio
 $scope.store.supportedEncoding = $scope.cubeConfig.encodings;
   })
   $scope.createFilter=function(type){
- if(type.indexOf("varchar")<=0){
+ if(type.indexOf("varchar")==-1){
return ['fixed_length_hex'];
  }else if(type!="date"){
return ['date'];

http://git-wip-us.apache.org/repos/asf/kylin/blob/1a5295ba/webapp/app/js/controllers/cubeMeasures.js
--
diff --git a/webapp/app/js/controllers/cubeMeasures.js 
b/webapp/app/js/controllers/cubeMeasures.js
index 085338f..18f53c2 100644
--- a/webapp/app/js/controllers/cubeMeasures.js
+++ b/webapp/app/js/controllers/cubeMeasures.js
@@ -45,7 +45,7 @@ KylinApp.controller('CubeMeasuresCtrl', function ($scope, 
$modal,MetaModel,cubes
 }
   }
   $scope.createFilter=function(type){
-if(type.indexOf("varchar")<=0){
+if(type.indexOf("varchar")==-1){
   return ['fixed_length_hex'];
 }else if(type!="date"){
   return ['date'];
@@ -60,7 +60,7 @@ KylinApp.controller('CubeMeasuresCtrl', function ($scope, 
$modal,MetaModel,cubes
 var encodings =$scope.store.supportedEncoding,filterEncoding=[];
 var filerList=$scope.createFilter(type);
 if($scope.isEdit) {
-  if (name && $scope.newMeasure.function.configuration) {
+  if (name && 
$scope.newMeasure.function.configuration&&$scope.newMeasure.function.configuration['topn.encoding.'
 + name]) {
 var version = 
$scope.newMeasure.function.configuration['topn.encoding_version.' + name] || 1;
 filterEncoding = VdmUtil.getFilterObjectListByOrFilterVal(encodings, 
'value', $scope.newMeasure.function.configuration['topn.encoding.' + 
name].replace(/:\d+/, "") + (version ? "[v" + version + "]" : "[v1]"), 
'suggest', true);
   }else{
@@ -119,7 +119,7 @@ KylinApp.controller('CubeMeasuresCtrl', function ($scope, 
$modal,MetaModel,cubes
 var 
version=$scope.newMeasure.function.configuration['topn.encoding_version.'+_name]||1;
 item=$scope.removeVersion(item);
 var baseKey=item.replace(/:\d+/,'');
-if(needLengthKeyList.indexOf(baseKey)>=-1){
+if(needLengthKeyList.indexOf(baseKey)!=-1){
   

kylin git commit: refine mapper and reducer log

[mahongbin]

Repository: kylin
Updated Branches:
  refs/heads/master 28ba1eaea -> 58224921d


refine mapper and reducer log


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/58224921
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/58224921
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/58224921

Branch: refs/heads/master
Commit: 58224921d896e4479f5d034d43c044aacaf14200
Parents: 28ba1ea
Author: Hongbin Ma 
Authored: Thu Dec 1 18:15:46 2016 +0800
Committer: Hongbin Ma 
Committed: Thu Dec 1 18:15:46 2016 +0800

--
 .../java/org/apache/kylin/engine/mr/KylinMapper.java   | 12 +---
 .../java/org/apache/kylin/engine/mr/KylinReducer.java  | 13 ++---
 .../apache/kylin/engine/mr/steps/CuboidReducer.java| 11 ---
 .../kylin/engine/mr/steps/HiveToBaseCuboidMapper.java  |  6 --
 .../kylin/engine/mr/steps/InMemCuboidMapper.java   |  7 +--
 .../kylin/engine/mr/steps/InMemCuboidReducer.java  | 10 +-
 .../apache/kylin/engine/mr/steps/NDCuboidMapper.java   | 10 --
 7 files changed, 33 insertions(+), 36 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/58224921/engine-mr/src/main/java/org/apache/kylin/engine/mr/KylinMapper.java
--
diff --git 
a/engine-mr/src/main/java/org/apache/kylin/engine/mr/KylinMapper.java 
b/engine-mr/src/main/java/org/apache/kylin/engine/mr/KylinMapper.java
index a01f7a2..2b564e9 100644
--- a/engine-mr/src/main/java/org/apache/kylin/engine/mr/KylinMapper.java
+++ b/engine-mr/src/main/java/org/apache/kylin/engine/mr/KylinMapper.java
@@ -22,6 +22,7 @@ import java.io.IOException;
 
 import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.mapreduce.Mapper;
+import org.apache.kylin.engine.mr.common.BatchConstants;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -30,6 +31,8 @@ import org.slf4j.LoggerFactory;
 public class KylinMapper extends 
Mapper {
 private static final Logger logger = 
LoggerFactory.getLogger(KylinMapper.class);
 
+protected int mapCounter = 0;
+
 protected void bindCurrentConfiguration(Configuration conf) {
 logger.info("The conf for current mapper will be " + 
System.identityHashCode(conf));
 HadoopUtil.setCurrentConfiguration(conf);
@@ -38,6 +41,9 @@ public class KylinMapper 
extends Mapper.Context context) throws IOException, InterruptedException {
 try {
+if (mapCounter++ % BatchConstants.NORMAL_RECORD_LOG_THRESHOLD == 
0) {
+logger.info("Accepting Mapper Key with ordinal: " + 
mapCounter);
+}
 doMap(key, value, context);
 } catch (IOException ex) { // KYLIN-2170
 logger.error("", ex);
@@ -53,11 +59,11 @@ public class KylinMapper 
extends Mapper.Context context) throws IOException, InterruptedException {
 super.map(key, value, context);
 }
-
+
 @Override
 final protected void cleanup(Mapper.Context context) throws IOException, InterruptedException {
 try {
@@ -76,7 +82,7 @@ public class KylinMapper 
extends Mapper.Context 
context) throws IOException, InterruptedException {
 }
 }

http://git-wip-us.apache.org/repos/asf/kylin/blob/58224921/engine-mr/src/main/java/org/apache/kylin/engine/mr/KylinReducer.java
--
diff --git 
a/engine-mr/src/main/java/org/apache/kylin/engine/mr/KylinReducer.java 
b/engine-mr/src/main/java/org/apache/kylin/engine/mr/KylinReducer.java
index 2b63ce0..cb2d7a7 100644
--- a/engine-mr/src/main/java/org/apache/kylin/engine/mr/KylinReducer.java
+++ b/engine-mr/src/main/java/org/apache/kylin/engine/mr/KylinReducer.java
@@ -22,6 +22,7 @@ import java.io.IOException;
 
 import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.mapreduce.Reducer;
+import org.apache.kylin.engine.mr.common.BatchConstants;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -29,7 +30,9 @@ import org.slf4j.LoggerFactory;
  */
 public class KylinReducer extends 
Reducer {
 

kylin git commit: refine mapper and reducer log

[mahongbin]

Repository: kylin
Updated Branches:
  refs/heads/yang21 80018874c -> d3ecb0d9c


refine mapper and reducer log


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/d3ecb0d9
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/d3ecb0d9
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/d3ecb0d9

Branch: refs/heads/yang21
Commit: d3ecb0d9c381dbb035c7cada7d3c798e24fef1d1
Parents: 8001887
Author: Hongbin Ma 
Authored: Thu Dec 1 18:01:55 2016 +0800
Committer: Hongbin Ma 
Committed: Thu Dec 1 18:01:55 2016 +0800

--
 .../org/apache/kylin/engine/mr/KylinMapper.java | 17 +++---
 .../apache/kylin/engine/mr/KylinReducer.java| 17 +++---
 .../engine/mr/steps/BaseCuboidMapperBase.java   |  1 -
 .../kylin/engine/mr/steps/CuboidReducer.java| 21 ++--
 .../engine/mr/steps/HiveToBaseCuboidMapper.java | 10 ++
 .../engine/mr/steps/InMemCuboidMapper.java  | 34 
 .../engine/mr/steps/InMemCuboidReducer.java | 20 ++--
 .../kylin/engine/mr/steps/NDCuboidMapper.java   | 22 ++---
 8 files changed, 70 insertions(+), 72 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/d3ecb0d9/engine-mr/src/main/java/org/apache/kylin/engine/mr/KylinMapper.java
--
diff --git 
a/engine-mr/src/main/java/org/apache/kylin/engine/mr/KylinMapper.java 
b/engine-mr/src/main/java/org/apache/kylin/engine/mr/KylinMapper.java
index a01f7a2..c5af2fe 100644
--- a/engine-mr/src/main/java/org/apache/kylin/engine/mr/KylinMapper.java
+++ b/engine-mr/src/main/java/org/apache/kylin/engine/mr/KylinMapper.java
@@ -18,18 +18,21 @@
 
 package org.apache.kylin.engine.mr;
 
-import java.io.IOException;
-
 import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.mapreduce.Mapper;
+import org.apache.kylin.engine.mr.common.BatchConstants;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
+import java.io.IOException;
+
 /**
  */
 public class KylinMapper extends 
Mapper {
 private static final Logger logger = 
LoggerFactory.getLogger(KylinMapper.class);
 
+protected int mapCounter = 0;
+
 protected void bindCurrentConfiguration(Configuration conf) {
 logger.info("The conf for current mapper will be " + 
System.identityHashCode(conf));
 HadoopUtil.setCurrentConfiguration(conf);
@@ -38,6 +41,10 @@ public class KylinMapper 
extends Mapper.Context context) throws IOException, InterruptedException {
 try {
+if (mapCounter++ % BatchConstants.NORMAL_RECORD_LOG_THRESHOLD == 
0) {
+logger.info("Accepting Mapper Key with ordinal: " + 
mapCounter);
+}
+
 doMap(key, value, context);
 } catch (IOException ex) { // KYLIN-2170
 logger.error("", ex);
@@ -53,11 +60,11 @@ public class KylinMapper 
extends Mapper.Context context) throws IOException, InterruptedException {
 super.map(key, value, context);
 }
-
+
 @Override
 final protected void cleanup(Mapper.Context context) throws IOException, InterruptedException {
 try {
@@ -76,7 +83,7 @@ public class KylinMapper 
extends Mapper.Context 
context) throws IOException, InterruptedException {
 }
 }

http://git-wip-us.apache.org/repos/asf/kylin/blob/d3ecb0d9/engine-mr/src/main/java/org/apache/kylin/engine/mr/KylinReducer.java
--
diff --git 
a/engine-mr/src/main/java/org/apache/kylin/engine/mr/KylinReducer.java 
b/engine-mr/src/main/java/org/apache/kylin/engine/mr/KylinReducer.java
index 2b63ce0..83266ea 100644
--- a/engine-mr/src/main/java/org/apache/kylin/engine/mr/KylinReducer.java
+++ b/engine-mr/src/main/java/org/apache/kylin/engine/mr/KylinReducer.java
@@ -18,18 +18,22 @@
 
 package org.apache.kylin.engine.mr;
 
-import java.io.IOException;
-
 import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.mapreduce.Reducer;
+import org.apache.kylin.engine.mr.common.BatchConstants;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
+import java.io.IOException;
+
 /**
  */
 public class 

[3/3] kylin git commit: KYLIN-2233 Support HBase 1.0.2

[mahongbin]

KYLIN-2233 Support HBase 1.0.2

Signed-off-by: Yang Li 


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/11b7fff7
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/11b7fff7
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/11b7fff7

Branch: refs/heads/yang21-hbase102
Commit: 11b7fff75f3b8c4091741ee68f468d207a07fa4a
Parents: 215329e
Author: Cheng Wang 
Authored: Wed Oct 12 14:39:24 2016 +0800
Committer: Hongbin Ma 
Committed: Thu Dec 1 16:18:42 2016 +0800

--
 pom.xml   |  2 +-
 .../storage/hbase/cube/v1/RegionScannerAdapter.java   | 11 ++-
 .../coprocessor/observer/AggregateRegionObserver.java |  4 ++--
 .../v1/coprocessor/observer/AggregationScanner.java   | 14 --
 .../observer/ObserverAggregationCache.java| 10 ++
 .../observer/AggregateRegionObserverTest.java | 12 +++-
 .../cube/v1/filter/TestFuzzyRowFilterV2EndToEnd.java  |  3 +--
 7 files changed, 15 insertions(+), 41 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/11b7fff7/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 1df14f4..ad70ee3 100644
--- a/pom.xml
+++ b/pom.xml
@@ -54,7 +54,7 @@
 1.2.1
 
 
-1.1.1
+1.0.2
 0.8.1
 
 

http://git-wip-us.apache.org/repos/asf/kylin/blob/11b7fff7/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/cube/v1/RegionScannerAdapter.java
--
diff --git 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/cube/v1/RegionScannerAdapter.java
 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/cube/v1/RegionScannerAdapter.java
index 0ade920..40da772 100644
--- 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/cube/v1/RegionScannerAdapter.java
+++ 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/cube/v1/RegionScannerAdapter.java
@@ -26,7 +26,6 @@ import org.apache.hadoop.hbase.HRegionInfo;
 import org.apache.hadoop.hbase.client.Result;
 import org.apache.hadoop.hbase.client.ResultScanner;
 import org.apache.hadoop.hbase.regionserver.RegionScanner;
-import org.apache.hadoop.hbase.regionserver.ScannerContext;
 
 /**
  * @author yangli9
@@ -51,7 +50,7 @@ public class RegionScannerAdapter implements RegionScanner {
 }
 
 @Override
-public boolean next(List result, ScannerContext scannerContext) 
throws IOException {
+public boolean next(List result, int limit) throws IOException {
 return next(result);
 }
 
@@ -61,7 +60,7 @@ public class RegionScannerAdapter implements RegionScanner {
 }
 
 @Override
-public boolean nextRaw(List result, ScannerContext scannerContext) 
throws IOException {
+public boolean nextRaw(List result, int limit) throws IOException {
 return next(result);
 }
 
@@ -94,10 +93,4 @@ public class RegionScannerAdapter implements RegionScanner {
 public long getMvccReadPoint() {
 return Long.MAX_VALUE;
 }
-
-@Override
-public int getBatch() {
-return -1;
-}
-
 }

http://git-wip-us.apache.org/repos/asf/kylin/blob/11b7fff7/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/cube/v1/coprocessor/observer/AggregateRegionObserver.java
--
diff --git 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/cube/v1/coprocessor/observer/AggregateRegionObserver.java
 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/cube/v1/coprocessor/observer/AggregateRegionObserver.java
index 7e25e4c..7139ca7 100644
--- 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/cube/v1/coprocessor/observer/AggregateRegionObserver.java
+++ 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/cube/v1/coprocessor/observer/AggregateRegionObserver.java
@@ -26,7 +26,7 @@ import org.apache.hadoop.hbase.client.Scan;
 import org.apache.hadoop.hbase.coprocessor.BaseRegionObserver;
 import org.apache.hadoop.hbase.coprocessor.ObserverContext;
 import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
-import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.regionserver.HRegion;
 import org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost;
 import org.apache.hadoop.hbase.regionserver.RegionScanner;
 import org.apache.kylin.gridtable.StorageSideBehavior;
@@ -99,7 +99,7 @@ public class AggregateRegionObserver extends 
BaseRegionObserver {
 // start/end region operation & sync on scanner is suggested by the
 // javadoc of RegionScanner.nextRaw()
 // FIXME: will 

[2/3] kylin git commit: KYLIN-1528 Create a branch for v1.5 with HBase 1.x API

[mahongbin]

KYLIN-1528 Create a branch for v1.5 with HBase 1.x API


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/215329ea
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/215329ea
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/215329ea

Branch: refs/heads/yang21-hbase102
Commit: 215329ea084465e654bfc6e2e32e91f68a64254a
Parents: 8001887
Author: shaofengshi 
Authored: Wed Mar 23 17:07:05 2016 +0800
Committer: Hongbin Ma 
Committed: Thu Dec 1 16:15:27 2016 +0800

--
 examples/test_case_data/sandbox/hbase-site.xml  | 19 +---
 .../kylin/provision/BuildCubeWithEngine.java| 13 ++-
 pom.xml | 12 +--
 .../kylin/rest/security/AclHBaseStorage.java|  4 +-
 .../rest/security/MockAclHBaseStorage.java  |  8 +-
 .../apache/kylin/rest/security/MockHTable.java  | 95 
 .../rest/security/RealAclHBaseStorage.java  |  9 +-
 .../apache/kylin/rest/service/AclService.java   | 25 +++---
 .../apache/kylin/rest/service/CubeService.java  | 36 +++-
 .../apache/kylin/rest/service/QueryService.java | 24 +++--
 .../apache/kylin/rest/service/UserService.java  | 17 ++--
 .../kylin/storage/hbase/HBaseConnection.java| 44 -
 .../kylin/storage/hbase/HBaseResourceStore.java | 31 +++
 .../kylin/storage/hbase/HBaseStorage.java   |  3 +-
 .../storage/hbase/cube/SimpleHBaseStore.java| 20 ++---
 .../hbase/cube/v1/CubeSegmentTupleIterator.java | 11 +--
 .../storage/hbase/cube/v1/CubeStorageQuery.java |  6 +-
 .../hbase/cube/v1/RegionScannerAdapter.java | 10 ++-
 .../cube/v1/SerializedHBaseTupleIterator.java   |  4 +-
 .../observer/AggregateRegionObserver.java   |  4 +-
 .../observer/AggregationScanner.java| 14 ++-
 .../observer/ObserverAggregationCache.java  | 10 ++-
 .../coprocessor/observer/ObserverEnabler.java   |  4 +-
 .../hbase/cube/v2/CubeHBaseEndpointRPC.java | 13 +--
 .../storage/hbase/cube/v2/CubeHBaseScanRPC.java |  9 +-
 .../coprocessor/endpoint/CubeVisitService.java  |  4 +-
 .../storage/hbase/steps/CubeHTableUtil.java | 16 ++--
 .../storage/hbase/steps/DeprecatedGCStep.java   | 23 ++---
 .../storage/hbase/steps/HBaseCuboidWriter.java  |  7 +-
 .../hbase/steps/HBaseStreamingOutput.java   |  9 +-
 .../kylin/storage/hbase/steps/MergeGCStep.java  | 23 ++---
 .../storage/hbase/util/CleanHtableCLI.java  | 12 +--
 .../storage/hbase/util/CubeMigrationCLI.java| 36 
 .../hbase/util/CubeMigrationCheckCLI.java   | 17 ++--
 .../hbase/util/DeployCoprocessorCLI.java| 22 ++---
 .../hbase/util/ExtendCubeToHybridCLI.java   |  8 +-
 .../hbase/util/GridTableHBaseBenchmark.java | 34 +++
 .../kylin/storage/hbase/util/HBaseClean.java| 18 ++--
 .../hbase/util/HBaseRegionSizeCalculator.java   | 35 
 .../kylin/storage/hbase/util/HBaseUsage.java|  9 +-
 .../storage/hbase/util/HbaseStreamingInput.java | 30 +++
 .../hbase/util/HtableAlterMetadataCLI.java  |  9 +-
 .../storage/hbase/util/OrphanHBaseCleanJob.java | 19 ++--
 .../kylin/storage/hbase/util/PingHBaseCLI.java  | 15 ++--
 .../kylin/storage/hbase/util/RowCounterCLI.java | 11 +--
 .../storage/hbase/util/StorageCleanupJob.java   | 20 +++--
 .../storage/hbase/util/UpdateHTableHostCLI.java | 17 ++--
 .../observer/AggregateRegionObserverTest.java   | 26 ++
 .../v1/filter/TestFuzzyRowFilterV2EndToEnd.java |  5 +-
 .../org/apache/kylin/tool/CubeMigrationCLI.java | 36 
 .../kylin/tool/CubeMigrationCheckCLI.java   | 16 ++--
 .../kylin/tool/ExtendCubeToHybridCLI.java   |  8 +-
 .../apache/kylin/tool/StorageCleanupJob.java| 20 +++--
 53 files changed, 450 insertions(+), 500 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/215329ea/examples/test_case_data/sandbox/hbase-site.xml
--
diff --git a/examples/test_case_data/sandbox/hbase-site.xml 
b/examples/test_case_data/sandbox/hbase-site.xml
index 46d5345..734908e 100644
--- a/examples/test_case_data/sandbox/hbase-site.xml
+++ b/examples/test_case_data/sandbox/hbase-site.xml
@@ -190,22 +190,5 @@
 zookeeper.znode.parent
 /hbase-unsecure
 
-
-hbase.client.pause
-100
-General client pause value.  Used mostly as value to wait
-before running a retry of a failed get, region lookup, etc.
-See hbase.client.retries.number for description of how we backoff 
from
-this initial pause amount and how this pause works w/ 
retries.
-
-
-hbase.client.retries.number
-5
-Maximum retries.  Used as maximum for all retryable
-operations such as the getting of a cell's value, starting a row 
update,
-

[1/3] kylin git commit: KYLIN-1528 Create a branch for v1.5 with HBase 1.x API [Forced Update!]

[mahongbin]

Repository: kylin
Updated Branches:
  refs/heads/yang21-hbase102 5d7e7f239 -> 11b7fff75 (forced update)


http://git-wip-us.apache.org/repos/asf/kylin/blob/215329ea/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseStreamingOutput.java
--
diff --git 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseStreamingOutput.java
 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseStreamingOutput.java
index 0e95102..c59fb33 100644
--- 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseStreamingOutput.java
+++ 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseStreamingOutput.java
@@ -27,7 +27,8 @@ import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.fs.FSDataInputStream;
 import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.fs.Path;
-import org.apache.hadoop.hbase.client.HTableInterface;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Table;
 import org.apache.kylin.common.KylinConfig;
 import org.apache.kylin.common.persistence.ResourceStore;
 import org.apache.kylin.cube.CubeSegment;
@@ -56,7 +57,7 @@ public class HBaseStreamingOutput implements IStreamingOutput 
{
 try {
 CubeSegment cubeSegment = (CubeSegment) buildable;
 
-final HTableInterface hTable;
+final Table hTable;
 hTable = createHTable(cubeSegment);
 List cuboidWriters = Lists.newArrayList();
 cuboidWriters.add(new HBaseCuboidWriter(cubeSegment, hTable));
@@ -88,10 +89,10 @@ public class HBaseStreamingOutput implements 
IStreamingOutput {
 }
 }
 
-private HTableInterface createHTable(final CubeSegment cubeSegment) throws 
IOException {
+private Table createHTable(final CubeSegment cubeSegment) throws 
IOException {
 final String hTableName = cubeSegment.getStorageLocationIdentifier();
 CubeHTableUtil.createHTable(cubeSegment, null);
-final HTableInterface hTable = 
HBaseConnection.get(KylinConfig.getInstanceFromEnv().getStorageUrl()).getTable(hTableName);
+final Table hTable = 
HBaseConnection.get(KylinConfig.getInstanceFromEnv().getStorageUrl()).getTable(TableName.valueOf(hTableName));
 logger.info("hTable:" + hTableName + " for segment:" + 
cubeSegment.getName() + " created!");
 return hTable;
 }

http://git-wip-us.apache.org/repos/asf/kylin/blob/215329ea/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/MergeGCStep.java
--
diff --git 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/MergeGCStep.java
 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/MergeGCStep.java
index 5b2441c..2f7e164 100644
--- 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/MergeGCStep.java
+++ 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/MergeGCStep.java
@@ -24,11 +24,11 @@ import java.util.Collections;
 import java.util.List;
 
 import org.apache.commons.lang.StringUtils;
-import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.hbase.HTableDescriptor;
-import org.apache.hadoop.hbase.client.HBaseAdmin;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.Connection;
 import org.apache.kylin.common.KylinConfig;
-import org.apache.kylin.common.util.Bytes;
 import org.apache.kylin.job.exception.ExecuteException;
 import org.apache.kylin.job.execution.AbstractExecutable;
 import org.apache.kylin.job.execution.ExecutableContext;
@@ -69,19 +69,20 @@ public class MergeGCStep extends AbstractExecutable {
 List oldTables = getOldHTables();
 if (oldTables != null && oldTables.size() > 0) {
 String metadataUrlPrefix = 
KylinConfig.getInstanceFromEnv().getMetadataUrlPrefix();
-Configuration conf = 
HBaseConnection.getCurrentHBaseConfiguration();
-HBaseAdmin admin = null;
+Admin admin = null;
 try {
-admin = new HBaseAdmin(conf);
+Connection conn = 
HBaseConnection.get(KylinConfig.getInstanceFromEnv().getStorageUrl());
+admin = conn.getAdmin();
+
 for (String table : oldTables) {
-if (admin.tableExists(table)) {
-HTableDescriptor tableDescriptor = 
admin.getTableDescriptor(Bytes.toBytes(table));
+if (admin.tableExists(TableName.valueOf(table))) {
+HTableDescriptor tableDescriptor = 
admin.getTableDescriptor(TableName.valueOf((table)));
 String host = 
tableDescriptor.getValue(IRealizationConstants.HTableTag);
 if (metadataUrlPrefix.equalsIgnoreCase(host)) {
-  

[2/3] kylin git commit: KYLIN-1528 Create a branch for v1.5 with HBase 1.x API

[mahongbin]

KYLIN-1528 Create a branch for v1.5 with HBase 1.x API


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/215329ea
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/215329ea
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/215329ea

Branch: refs/heads/yang21-cdh5.7
Commit: 215329ea084465e654bfc6e2e32e91f68a64254a
Parents: 8001887
Author: shaofengshi 
Authored: Wed Mar 23 17:07:05 2016 +0800
Committer: Hongbin Ma 
Committed: Thu Dec 1 16:15:27 2016 +0800

--
 examples/test_case_data/sandbox/hbase-site.xml  | 19 +---
 .../kylin/provision/BuildCubeWithEngine.java| 13 ++-
 pom.xml | 12 +--
 .../kylin/rest/security/AclHBaseStorage.java|  4 +-
 .../rest/security/MockAclHBaseStorage.java  |  8 +-
 .../apache/kylin/rest/security/MockHTable.java  | 95 
 .../rest/security/RealAclHBaseStorage.java  |  9 +-
 .../apache/kylin/rest/service/AclService.java   | 25 +++---
 .../apache/kylin/rest/service/CubeService.java  | 36 +++-
 .../apache/kylin/rest/service/QueryService.java | 24 +++--
 .../apache/kylin/rest/service/UserService.java  | 17 ++--
 .../kylin/storage/hbase/HBaseConnection.java| 44 -
 .../kylin/storage/hbase/HBaseResourceStore.java | 31 +++
 .../kylin/storage/hbase/HBaseStorage.java   |  3 +-
 .../storage/hbase/cube/SimpleHBaseStore.java| 20 ++---
 .../hbase/cube/v1/CubeSegmentTupleIterator.java | 11 +--
 .../storage/hbase/cube/v1/CubeStorageQuery.java |  6 +-
 .../hbase/cube/v1/RegionScannerAdapter.java | 10 ++-
 .../cube/v1/SerializedHBaseTupleIterator.java   |  4 +-
 .../observer/AggregateRegionObserver.java   |  4 +-
 .../observer/AggregationScanner.java| 14 ++-
 .../observer/ObserverAggregationCache.java  | 10 ++-
 .../coprocessor/observer/ObserverEnabler.java   |  4 +-
 .../hbase/cube/v2/CubeHBaseEndpointRPC.java | 13 +--
 .../storage/hbase/cube/v2/CubeHBaseScanRPC.java |  9 +-
 .../coprocessor/endpoint/CubeVisitService.java  |  4 +-
 .../storage/hbase/steps/CubeHTableUtil.java | 16 ++--
 .../storage/hbase/steps/DeprecatedGCStep.java   | 23 ++---
 .../storage/hbase/steps/HBaseCuboidWriter.java  |  7 +-
 .../hbase/steps/HBaseStreamingOutput.java   |  9 +-
 .../kylin/storage/hbase/steps/MergeGCStep.java  | 23 ++---
 .../storage/hbase/util/CleanHtableCLI.java  | 12 +--
 .../storage/hbase/util/CubeMigrationCLI.java| 36 
 .../hbase/util/CubeMigrationCheckCLI.java   | 17 ++--
 .../hbase/util/DeployCoprocessorCLI.java| 22 ++---
 .../hbase/util/ExtendCubeToHybridCLI.java   |  8 +-
 .../hbase/util/GridTableHBaseBenchmark.java | 34 +++
 .../kylin/storage/hbase/util/HBaseClean.java| 18 ++--
 .../hbase/util/HBaseRegionSizeCalculator.java   | 35 
 .../kylin/storage/hbase/util/HBaseUsage.java|  9 +-
 .../storage/hbase/util/HbaseStreamingInput.java | 30 +++
 .../hbase/util/HtableAlterMetadataCLI.java  |  9 +-
 .../storage/hbase/util/OrphanHBaseCleanJob.java | 19 ++--
 .../kylin/storage/hbase/util/PingHBaseCLI.java  | 15 ++--
 .../kylin/storage/hbase/util/RowCounterCLI.java | 11 +--
 .../storage/hbase/util/StorageCleanupJob.java   | 20 +++--
 .../storage/hbase/util/UpdateHTableHostCLI.java | 17 ++--
 .../observer/AggregateRegionObserverTest.java   | 26 ++
 .../v1/filter/TestFuzzyRowFilterV2EndToEnd.java |  5 +-
 .../org/apache/kylin/tool/CubeMigrationCLI.java | 36 
 .../kylin/tool/CubeMigrationCheckCLI.java   | 16 ++--
 .../kylin/tool/ExtendCubeToHybridCLI.java   |  8 +-
 .../apache/kylin/tool/StorageCleanupJob.java| 20 +++--
 53 files changed, 450 insertions(+), 500 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/215329ea/examples/test_case_data/sandbox/hbase-site.xml
--
diff --git a/examples/test_case_data/sandbox/hbase-site.xml 
b/examples/test_case_data/sandbox/hbase-site.xml
index 46d5345..734908e 100644
--- a/examples/test_case_data/sandbox/hbase-site.xml
+++ b/examples/test_case_data/sandbox/hbase-site.xml
@@ -190,22 +190,5 @@
 zookeeper.znode.parent
 /hbase-unsecure
 
-
-hbase.client.pause
-100
-General client pause value.  Used mostly as value to wait
-before running a retry of a failed get, region lookup, etc.
-See hbase.client.retries.number for description of how we backoff 
from
-this initial pause amount and how this pause works w/ 
retries.
-
-
-hbase.client.retries.number
-5
-Maximum retries.  Used as maximum for all retryable
-operations such as the getting of a cell's value, starting a row 
update,
-  

[3/3] kylin git commit: KYLIN-1672 support kylin on cdh 5.7

[mahongbin]

KYLIN-1672 support kylin on cdh 5.7

Signed-off-by: Li Yang 


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/c0291138
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/c0291138
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/c0291138

Branch: refs/heads/yang21-cdh5.7
Commit: c029113811fe3cd1e2d1539afb7ad5903987b3d2
Parents: 215329e
Author: Lynne Jiang 
Authored: Mon May 16 03:33:27 2016 -0700
Committer: Hongbin Ma 
Committed: Thu Dec 1 16:17:04 2016 +0800

--
 build/conf/kylin.properties |   3 +
 dev-support/test_all_against_hdp_2_2_4_2_2.sh   |   0
 .../kylin/engine/mr/steps/MockupMapContext.java |  15 +-
 examples/test_case_data/sandbox/core-site.xml   | 146 +++---
 examples/test_case_data/sandbox/hbase-site.xml  | 162 ++
 examples/test_case_data/sandbox/hdfs-site.xml   | 259 ++
 examples/test_case_data/sandbox/mapred-site.xml | 398 ++-
 examples/test_case_data/sandbox/yarn-site.xml   | 496 ++-
 pom.xml |  16 +-
 server/pom.xml  |  36 ++
 .../storage/hbase/steps/MockupMapContext.java   |  19 +-
 tool/pom.xml|  12 +
 12 files changed, 431 insertions(+), 1131 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/c0291138/build/conf/kylin.properties
--
diff --git a/build/conf/kylin.properties b/build/conf/kylin.properties
index ed86bdb..ebd8cde 100644
--- a/build/conf/kylin.properties
+++ b/build/conf/kylin.properties
@@ -118,6 +118,9 @@ kylin.job.mapreduce.mapper.input.rows=100
 
 kylin.job.step.timeout=7200
 
+# for secure cdh, filtering hive dependency is risky, so filter nothing
+kylin.job.dependency.filterlist=[^,]+
+
 ### CUBE ###
 
 # 'auto', 'inmem', 'layer' or 'random' for testing

http://git-wip-us.apache.org/repos/asf/kylin/blob/c0291138/dev-support/test_all_against_hdp_2_2_4_2_2.sh
--
diff --git a/dev-support/test_all_against_hdp_2_2_4_2_2.sh 
b/dev-support/test_all_against_hdp_2_2_4_2_2.sh
old mode 100644
new mode 100755

http://git-wip-us.apache.org/repos/asf/kylin/blob/c0291138/engine-mr/src/test/java/org/apache/kylin/engine/mr/steps/MockupMapContext.java
--
diff --git 
a/engine-mr/src/test/java/org/apache/kylin/engine/mr/steps/MockupMapContext.java
 
b/engine-mr/src/test/java/org/apache/kylin/engine/mr/steps/MockupMapContext.java
index 847071d..9900465 100644
--- 
a/engine-mr/src/test/java/org/apache/kylin/engine/mr/steps/MockupMapContext.java
+++ 
b/engine-mr/src/test/java/org/apache/kylin/engine/mr/steps/MockupMapContext.java
@@ -77,6 +77,7 @@ public class MockupMapContext {
 outKV[0] = key;
 outKV[1] = value;
 }
+
 }
 
 @Override
@@ -99,6 +100,7 @@ public class MockupMapContext {
 throw new NotImplementedException();
 }
 
+
 @Override
 public float getProgress() {
 throw new NotImplementedException();
@@ -195,17 +197,17 @@ public class MockupMapContext {
 }
 
 @Override
-public RawComparator getSortComparator() {
+public boolean userClassesTakesPrecedence() {
 throw new NotImplementedException();
 }
 
 @Override
-public String getJar() {
+public RawComparator getSortComparator() {
 throw new NotImplementedException();
 }
 
 @Override
-public RawComparator getGroupingComparator() {
+public String getJar() {
 throw new NotImplementedException();
 }
 
@@ -221,7 +223,7 @@ public class MockupMapContext {
 
 @Override
 public boolean getProfileEnabled() {
-throw new NotImplementedException();
+return false;
 }
 
 @Override
@@ -308,6 +310,11 @@ public class MockupMapContext {
 public RawComparator getCombinerKeyGroupingComparator() {
 throw new NotImplementedException();
 }
+
+@Override
+public RawComparator getGroupingComparator() {
+return null;
+}
 });
 }
 }

http://git-wip-us.apache.org/repos/asf/kylin/blob/c0291138/examples/test_case_data/sandbox/core-site.xml
--
diff --git a/examples/test_case_data/sandbox/core-site.xml 

[1/3] kylin git commit: KYLIN-1528 Create a branch for v1.5 with HBase 1.x API [Forced Update!]

[mahongbin]

Repository: kylin
Updated Branches:
  refs/heads/yang21-cdh5.7 0b6d6662e -> c02911381 (forced update)


http://git-wip-us.apache.org/repos/asf/kylin/blob/215329ea/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseStreamingOutput.java
--
diff --git 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseStreamingOutput.java
 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseStreamingOutput.java
index 0e95102..c59fb33 100644
--- 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseStreamingOutput.java
+++ 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseStreamingOutput.java
@@ -27,7 +27,8 @@ import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.fs.FSDataInputStream;
 import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.fs.Path;
-import org.apache.hadoop.hbase.client.HTableInterface;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Table;
 import org.apache.kylin.common.KylinConfig;
 import org.apache.kylin.common.persistence.ResourceStore;
 import org.apache.kylin.cube.CubeSegment;
@@ -56,7 +57,7 @@ public class HBaseStreamingOutput implements IStreamingOutput 
{
 try {
 CubeSegment cubeSegment = (CubeSegment) buildable;
 
-final HTableInterface hTable;
+final Table hTable;
 hTable = createHTable(cubeSegment);
 List cuboidWriters = Lists.newArrayList();
 cuboidWriters.add(new HBaseCuboidWriter(cubeSegment, hTable));
@@ -88,10 +89,10 @@ public class HBaseStreamingOutput implements 
IStreamingOutput {
 }
 }
 
-private HTableInterface createHTable(final CubeSegment cubeSegment) throws 
IOException {
+private Table createHTable(final CubeSegment cubeSegment) throws 
IOException {
 final String hTableName = cubeSegment.getStorageLocationIdentifier();
 CubeHTableUtil.createHTable(cubeSegment, null);
-final HTableInterface hTable = 
HBaseConnection.get(KylinConfig.getInstanceFromEnv().getStorageUrl()).getTable(hTableName);
+final Table hTable = 
HBaseConnection.get(KylinConfig.getInstanceFromEnv().getStorageUrl()).getTable(TableName.valueOf(hTableName));
 logger.info("hTable:" + hTableName + " for segment:" + 
cubeSegment.getName() + " created!");
 return hTable;
 }

http://git-wip-us.apache.org/repos/asf/kylin/blob/215329ea/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/MergeGCStep.java
--
diff --git 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/MergeGCStep.java
 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/MergeGCStep.java
index 5b2441c..2f7e164 100644
--- 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/MergeGCStep.java
+++ 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/MergeGCStep.java
@@ -24,11 +24,11 @@ import java.util.Collections;
 import java.util.List;
 
 import org.apache.commons.lang.StringUtils;
-import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.hbase.HTableDescriptor;
-import org.apache.hadoop.hbase.client.HBaseAdmin;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.Connection;
 import org.apache.kylin.common.KylinConfig;
-import org.apache.kylin.common.util.Bytes;
 import org.apache.kylin.job.exception.ExecuteException;
 import org.apache.kylin.job.execution.AbstractExecutable;
 import org.apache.kylin.job.execution.ExecutableContext;
@@ -69,19 +69,20 @@ public class MergeGCStep extends AbstractExecutable {
 List oldTables = getOldHTables();
 if (oldTables != null && oldTables.size() > 0) {
 String metadataUrlPrefix = 
KylinConfig.getInstanceFromEnv().getMetadataUrlPrefix();
-Configuration conf = 
HBaseConnection.getCurrentHBaseConfiguration();
-HBaseAdmin admin = null;
+Admin admin = null;
 try {
-admin = new HBaseAdmin(conf);
+Connection conn = 
HBaseConnection.get(KylinConfig.getInstanceFromEnv().getStorageUrl());
+admin = conn.getAdmin();
+
 for (String table : oldTables) {
-if (admin.tableExists(table)) {
-HTableDescriptor tableDescriptor = 
admin.getTableDescriptor(Bytes.toBytes(table));
+if (admin.tableExists(TableName.valueOf(table))) {
+HTableDescriptor tableDescriptor = 
admin.getTableDescriptor(TableName.valueOf((table)));
 String host = 
tableDescriptor.getValue(IRealizationConstants.HTableTag);
 if (metadataUrlPrefix.equalsIgnoreCase(host)) {
-

[2/2] kylin git commit: KYLIN-1528 Create a branch for v1.5 with HBase 1.x API

[mahongbin]

KYLIN-1528 Create a branch for v1.5 with HBase 1.x API


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/215329ea
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/215329ea
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/215329ea

Branch: refs/heads/yang21-hbase1.x
Commit: 215329ea084465e654bfc6e2e32e91f68a64254a
Parents: 8001887
Author: shaofengshi 
Authored: Wed Mar 23 17:07:05 2016 +0800
Committer: Hongbin Ma 
Committed: Thu Dec 1 16:15:27 2016 +0800

--
 examples/test_case_data/sandbox/hbase-site.xml  | 19 +---
 .../kylin/provision/BuildCubeWithEngine.java| 13 ++-
 pom.xml | 12 +--
 .../kylin/rest/security/AclHBaseStorage.java|  4 +-
 .../rest/security/MockAclHBaseStorage.java  |  8 +-
 .../apache/kylin/rest/security/MockHTable.java  | 95 
 .../rest/security/RealAclHBaseStorage.java  |  9 +-
 .../apache/kylin/rest/service/AclService.java   | 25 +++---
 .../apache/kylin/rest/service/CubeService.java  | 36 +++-
 .../apache/kylin/rest/service/QueryService.java | 24 +++--
 .../apache/kylin/rest/service/UserService.java  | 17 ++--
 .../kylin/storage/hbase/HBaseConnection.java| 44 -
 .../kylin/storage/hbase/HBaseResourceStore.java | 31 +++
 .../kylin/storage/hbase/HBaseStorage.java   |  3 +-
 .../storage/hbase/cube/SimpleHBaseStore.java| 20 ++---
 .../hbase/cube/v1/CubeSegmentTupleIterator.java | 11 +--
 .../storage/hbase/cube/v1/CubeStorageQuery.java |  6 +-
 .../hbase/cube/v1/RegionScannerAdapter.java | 10 ++-
 .../cube/v1/SerializedHBaseTupleIterator.java   |  4 +-
 .../observer/AggregateRegionObserver.java   |  4 +-
 .../observer/AggregationScanner.java| 14 ++-
 .../observer/ObserverAggregationCache.java  | 10 ++-
 .../coprocessor/observer/ObserverEnabler.java   |  4 +-
 .../hbase/cube/v2/CubeHBaseEndpointRPC.java | 13 +--
 .../storage/hbase/cube/v2/CubeHBaseScanRPC.java |  9 +-
 .../coprocessor/endpoint/CubeVisitService.java  |  4 +-
 .../storage/hbase/steps/CubeHTableUtil.java | 16 ++--
 .../storage/hbase/steps/DeprecatedGCStep.java   | 23 ++---
 .../storage/hbase/steps/HBaseCuboidWriter.java  |  7 +-
 .../hbase/steps/HBaseStreamingOutput.java   |  9 +-
 .../kylin/storage/hbase/steps/MergeGCStep.java  | 23 ++---
 .../storage/hbase/util/CleanHtableCLI.java  | 12 +--
 .../storage/hbase/util/CubeMigrationCLI.java| 36 
 .../hbase/util/CubeMigrationCheckCLI.java   | 17 ++--
 .../hbase/util/DeployCoprocessorCLI.java| 22 ++---
 .../hbase/util/ExtendCubeToHybridCLI.java   |  8 +-
 .../hbase/util/GridTableHBaseBenchmark.java | 34 +++
 .../kylin/storage/hbase/util/HBaseClean.java| 18 ++--
 .../hbase/util/HBaseRegionSizeCalculator.java   | 35 
 .../kylin/storage/hbase/util/HBaseUsage.java|  9 +-
 .../storage/hbase/util/HbaseStreamingInput.java | 30 +++
 .../hbase/util/HtableAlterMetadataCLI.java  |  9 +-
 .../storage/hbase/util/OrphanHBaseCleanJob.java | 19 ++--
 .../kylin/storage/hbase/util/PingHBaseCLI.java  | 15 ++--
 .../kylin/storage/hbase/util/RowCounterCLI.java | 11 +--
 .../storage/hbase/util/StorageCleanupJob.java   | 20 +++--
 .../storage/hbase/util/UpdateHTableHostCLI.java | 17 ++--
 .../observer/AggregateRegionObserverTest.java   | 26 ++
 .../v1/filter/TestFuzzyRowFilterV2EndToEnd.java |  5 +-
 .../org/apache/kylin/tool/CubeMigrationCLI.java | 36 
 .../kylin/tool/CubeMigrationCheckCLI.java   | 16 ++--
 .../kylin/tool/ExtendCubeToHybridCLI.java   |  8 +-
 .../apache/kylin/tool/StorageCleanupJob.java| 20 +++--
 53 files changed, 450 insertions(+), 500 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/215329ea/examples/test_case_data/sandbox/hbase-site.xml
--
diff --git a/examples/test_case_data/sandbox/hbase-site.xml 
b/examples/test_case_data/sandbox/hbase-site.xml
index 46d5345..734908e 100644
--- a/examples/test_case_data/sandbox/hbase-site.xml
+++ b/examples/test_case_data/sandbox/hbase-site.xml
@@ -190,22 +190,5 @@
 zookeeper.znode.parent
 /hbase-unsecure
 
-
-hbase.client.pause
-100
-General client pause value.  Used mostly as value to wait
-before running a retry of a failed get, region lookup, etc.
-See hbase.client.retries.number for description of how we backoff 
from
-this initial pause amount and how this pause works w/ 
retries.
-
-
-hbase.client.retries.number
-5
-Maximum retries.  Used as maximum for all retryable
-operations such as the getting of a cell's value, starting a row 
update,
-

[1/2] kylin git commit: KYLIN-1528 Create a branch for v1.5 with HBase 1.x API [Forced Update!]

[mahongbin]

Repository: kylin
Updated Branches:
  refs/heads/yang21-hbase1.x 02f37bd82 -> 215329ea0 (forced update)


http://git-wip-us.apache.org/repos/asf/kylin/blob/215329ea/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseStreamingOutput.java
--
diff --git 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseStreamingOutput.java
 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseStreamingOutput.java
index 0e95102..c59fb33 100644
--- 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseStreamingOutput.java
+++ 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseStreamingOutput.java
@@ -27,7 +27,8 @@ import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.fs.FSDataInputStream;
 import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.fs.Path;
-import org.apache.hadoop.hbase.client.HTableInterface;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Table;
 import org.apache.kylin.common.KylinConfig;
 import org.apache.kylin.common.persistence.ResourceStore;
 import org.apache.kylin.cube.CubeSegment;
@@ -56,7 +57,7 @@ public class HBaseStreamingOutput implements IStreamingOutput 
{
 try {
 CubeSegment cubeSegment = (CubeSegment) buildable;
 
-final HTableInterface hTable;
+final Table hTable;
 hTable = createHTable(cubeSegment);
 List cuboidWriters = Lists.newArrayList();
 cuboidWriters.add(new HBaseCuboidWriter(cubeSegment, hTable));
@@ -88,10 +89,10 @@ public class HBaseStreamingOutput implements 
IStreamingOutput {
 }
 }
 
-private HTableInterface createHTable(final CubeSegment cubeSegment) throws 
IOException {
+private Table createHTable(final CubeSegment cubeSegment) throws 
IOException {
 final String hTableName = cubeSegment.getStorageLocationIdentifier();
 CubeHTableUtil.createHTable(cubeSegment, null);
-final HTableInterface hTable = 
HBaseConnection.get(KylinConfig.getInstanceFromEnv().getStorageUrl()).getTable(hTableName);
+final Table hTable = 
HBaseConnection.get(KylinConfig.getInstanceFromEnv().getStorageUrl()).getTable(TableName.valueOf(hTableName));
 logger.info("hTable:" + hTableName + " for segment:" + 
cubeSegment.getName() + " created!");
 return hTable;
 }

http://git-wip-us.apache.org/repos/asf/kylin/blob/215329ea/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/MergeGCStep.java
--
diff --git 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/MergeGCStep.java
 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/MergeGCStep.java
index 5b2441c..2f7e164 100644
--- 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/MergeGCStep.java
+++ 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/MergeGCStep.java
@@ -24,11 +24,11 @@ import java.util.Collections;
 import java.util.List;
 
 import org.apache.commons.lang.StringUtils;
-import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.hbase.HTableDescriptor;
-import org.apache.hadoop.hbase.client.HBaseAdmin;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.Connection;
 import org.apache.kylin.common.KylinConfig;
-import org.apache.kylin.common.util.Bytes;
 import org.apache.kylin.job.exception.ExecuteException;
 import org.apache.kylin.job.execution.AbstractExecutable;
 import org.apache.kylin.job.execution.ExecutableContext;
@@ -69,19 +69,20 @@ public class MergeGCStep extends AbstractExecutable {
 List oldTables = getOldHTables();
 if (oldTables != null && oldTables.size() > 0) {
 String metadataUrlPrefix = 
KylinConfig.getInstanceFromEnv().getMetadataUrlPrefix();
-Configuration conf = 
HBaseConnection.getCurrentHBaseConfiguration();
-HBaseAdmin admin = null;
+Admin admin = null;
 try {
-admin = new HBaseAdmin(conf);
+Connection conn = 
HBaseConnection.get(KylinConfig.getInstanceFromEnv().getStorageUrl());
+admin = conn.getAdmin();
+
 for (String table : oldTables) {
-if (admin.tableExists(table)) {
-HTableDescriptor tableDescriptor = 
admin.getTableDescriptor(Bytes.toBytes(table));
+if (admin.tableExists(TableName.valueOf(table))) {
+HTableDescriptor tableDescriptor = 
admin.getTableDescriptor(TableName.valueOf((table)));
 String host = 
tableDescriptor.getValue(IRealizationConstants.HTableTag);
 if (metadataUrlPrefix.equalsIgnoreCase(host)) {
-  

[4/6] kylin git commit: KYLIN-1528 Create a branch for v1.5 with HBase 1.x API

[mahongbin]

http://git-wip-us.apache.org/repos/asf/kylin/blob/02f37bd8/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseStreamingOutput.java
--
diff --git 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseStreamingOutput.java
 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseStreamingOutput.java
index 0e95102..c59fb33 100644
--- 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseStreamingOutput.java
+++ 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseStreamingOutput.java
@@ -27,7 +27,8 @@ import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.fs.FSDataInputStream;
 import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.fs.Path;
-import org.apache.hadoop.hbase.client.HTableInterface;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Table;
 import org.apache.kylin.common.KylinConfig;
 import org.apache.kylin.common.persistence.ResourceStore;
 import org.apache.kylin.cube.CubeSegment;
@@ -56,7 +57,7 @@ public class HBaseStreamingOutput implements IStreamingOutput 
{
 try {
 CubeSegment cubeSegment = (CubeSegment) buildable;
 
-final HTableInterface hTable;
+final Table hTable;
 hTable = createHTable(cubeSegment);
 List cuboidWriters = Lists.newArrayList();
 cuboidWriters.add(new HBaseCuboidWriter(cubeSegment, hTable));
@@ -88,10 +89,10 @@ public class HBaseStreamingOutput implements 
IStreamingOutput {
 }
 }
 
-private HTableInterface createHTable(final CubeSegment cubeSegment) throws 
IOException {
+private Table createHTable(final CubeSegment cubeSegment) throws 
IOException {
 final String hTableName = cubeSegment.getStorageLocationIdentifier();
 CubeHTableUtil.createHTable(cubeSegment, null);
-final HTableInterface hTable = 
HBaseConnection.get(KylinConfig.getInstanceFromEnv().getStorageUrl()).getTable(hTableName);
+final Table hTable = 
HBaseConnection.get(KylinConfig.getInstanceFromEnv().getStorageUrl()).getTable(TableName.valueOf(hTableName));
 logger.info("hTable:" + hTableName + " for segment:" + 
cubeSegment.getName() + " created!");
 return hTable;
 }

http://git-wip-us.apache.org/repos/asf/kylin/blob/02f37bd8/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/MergeGCStep.java
--
diff --git 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/MergeGCStep.java
 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/MergeGCStep.java
index 5b2441c..2f7e164 100644
--- 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/MergeGCStep.java
+++ 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/MergeGCStep.java
@@ -24,11 +24,11 @@ import java.util.Collections;
 import java.util.List;
 
 import org.apache.commons.lang.StringUtils;
-import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.hbase.HTableDescriptor;
-import org.apache.hadoop.hbase.client.HBaseAdmin;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.Connection;
 import org.apache.kylin.common.KylinConfig;
-import org.apache.kylin.common.util.Bytes;
 import org.apache.kylin.job.exception.ExecuteException;
 import org.apache.kylin.job.execution.AbstractExecutable;
 import org.apache.kylin.job.execution.ExecutableContext;
@@ -69,19 +69,20 @@ public class MergeGCStep extends AbstractExecutable {
 List oldTables = getOldHTables();
 if (oldTables != null && oldTables.size() > 0) {
 String metadataUrlPrefix = 
KylinConfig.getInstanceFromEnv().getMetadataUrlPrefix();
-Configuration conf = 
HBaseConnection.getCurrentHBaseConfiguration();
-HBaseAdmin admin = null;
+Admin admin = null;
 try {
-admin = new HBaseAdmin(conf);
+Connection conn = 
HBaseConnection.get(KylinConfig.getInstanceFromEnv().getStorageUrl());
+admin = conn.getAdmin();
+
 for (String table : oldTables) {
-if (admin.tableExists(table)) {
-HTableDescriptor tableDescriptor = 
admin.getTableDescriptor(Bytes.toBytes(table));
+if (admin.tableExists(TableName.valueOf(table))) {
+HTableDescriptor tableDescriptor = 
admin.getTableDescriptor(TableName.valueOf((table)));
 String host = 
tableDescriptor.getValue(IRealizationConstants.HTableTag);
 if (metadataUrlPrefix.equalsIgnoreCase(host)) {
-if (admin.isTableEnabled(table)) {
-admin.disableTable(table);
+

[6/6] kylin git commit: KYLIN-2233 Support HBase 1.0.2

[mahongbin]

KYLIN-2233 Support HBase 1.0.2

Signed-off-by: Yang Li 


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/5d7e7f23
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/5d7e7f23
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/5d7e7f23

Branch: refs/heads/yang21-hbase102
Commit: 5d7e7f239b6cd5be85aff3a84ef34a5f7a75b890
Parents: 02f37bd
Author: Cheng Wang 
Authored: Wed Oct 12 14:39:24 2016 +0800
Committer: Hongbin Ma 
Committed: Thu Dec 1 13:24:55 2016 +0800

--
 pom.xml   |  2 +-
 .../storage/hbase/cube/v1/RegionScannerAdapter.java   | 11 ++-
 .../coprocessor/observer/AggregateRegionObserver.java |  4 ++--
 .../v1/coprocessor/observer/AggregationScanner.java   | 14 --
 .../observer/ObserverAggregationCache.java| 10 ++
 .../observer/AggregateRegionObserverTest.java | 12 +++-
 .../cube/v1/filter/TestFuzzyRowFilterV2EndToEnd.java  |  3 +--
 7 files changed, 15 insertions(+), 41 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/5d7e7f23/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 1df14f4..ad70ee3 100644
--- a/pom.xml
+++ b/pom.xml
@@ -54,7 +54,7 @@
 1.2.1
 
 
-1.1.1
+1.0.2
 0.8.1
 
 

http://git-wip-us.apache.org/repos/asf/kylin/blob/5d7e7f23/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/cube/v1/RegionScannerAdapter.java
--
diff --git 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/cube/v1/RegionScannerAdapter.java
 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/cube/v1/RegionScannerAdapter.java
index 0ade920..40da772 100644
--- 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/cube/v1/RegionScannerAdapter.java
+++ 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/cube/v1/RegionScannerAdapter.java
@@ -26,7 +26,6 @@ import org.apache.hadoop.hbase.HRegionInfo;
 import org.apache.hadoop.hbase.client.Result;
 import org.apache.hadoop.hbase.client.ResultScanner;
 import org.apache.hadoop.hbase.regionserver.RegionScanner;
-import org.apache.hadoop.hbase.regionserver.ScannerContext;
 
 /**
  * @author yangli9
@@ -51,7 +50,7 @@ public class RegionScannerAdapter implements RegionScanner {
 }
 
 @Override
-public boolean next(List result, ScannerContext scannerContext) 
throws IOException {
+public boolean next(List result, int limit) throws IOException {
 return next(result);
 }
 
@@ -61,7 +60,7 @@ public class RegionScannerAdapter implements RegionScanner {
 }
 
 @Override
-public boolean nextRaw(List result, ScannerContext scannerContext) 
throws IOException {
+public boolean nextRaw(List result, int limit) throws IOException {
 return next(result);
 }
 
@@ -94,10 +93,4 @@ public class RegionScannerAdapter implements RegionScanner {
 public long getMvccReadPoint() {
 return Long.MAX_VALUE;
 }
-
-@Override
-public int getBatch() {
-return -1;
-}
-
 }

http://git-wip-us.apache.org/repos/asf/kylin/blob/5d7e7f23/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/cube/v1/coprocessor/observer/AggregateRegionObserver.java
--
diff --git 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/cube/v1/coprocessor/observer/AggregateRegionObserver.java
 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/cube/v1/coprocessor/observer/AggregateRegionObserver.java
index 7e25e4c..7139ca7 100644
--- 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/cube/v1/coprocessor/observer/AggregateRegionObserver.java
+++ 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/cube/v1/coprocessor/observer/AggregateRegionObserver.java
@@ -26,7 +26,7 @@ import org.apache.hadoop.hbase.client.Scan;
 import org.apache.hadoop.hbase.coprocessor.BaseRegionObserver;
 import org.apache.hadoop.hbase.coprocessor.ObserverContext;
 import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
-import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.regionserver.HRegion;
 import org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost;
 import org.apache.hadoop.hbase.regionserver.RegionScanner;
 import org.apache.kylin.gridtable.StorageSideBehavior;
@@ -99,7 +99,7 @@ public class AggregateRegionObserver extends 
BaseRegionObserver {
 // start/end region operation & sync on scanner is suggested by the
 // javadoc of RegionScanner.nextRaw()
 // FIXME: will 

[2/6] kylin git commit: org.apache.kylin.metadata.model.FunctionDesc#configuration should use default linkedhashmap to keep order

[mahongbin]

org.apache.kylin.metadata.model.FunctionDesc#configuration should use default 
linkedhashmap to keep order


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/076c77ae
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/076c77ae
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/076c77ae

Branch: refs/heads/yang21-hbase102
Commit: 076c77ae217ceb320e06e534b7149f1fc60075ee
Parents: 0206860
Author: Hongbin Ma 
Authored: Thu Dec 1 11:20:53 2016 +0800
Committer: Hongbin Ma 
Committed: Thu Dec 1 11:20:53 2016 +0800

--
 .../org/apache/kylin/common/util/BasicTest.java | 24 ++--
 .../apache/kylin/common/util/JacksonBean.java   | 16 +++
 .../apache/kylin/common/util/JacksonTest.java   | 19 ++---
 .../kylin/metadata/model/FunctionDesc.java  | 29 ++--
 4 files changed, 61 insertions(+), 27 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/076c77ae/core-common/src/test/java/org/apache/kylin/common/util/BasicTest.java
--
diff --git 
a/core-common/src/test/java/org/apache/kylin/common/util/BasicTest.java 
b/core-common/src/test/java/org/apache/kylin/common/util/BasicTest.java
index ee15832..ffdbd25 100644
--- a/core-common/src/test/java/org/apache/kylin/common/util/BasicTest.java
+++ b/core-common/src/test/java/org/apache/kylin/common/util/BasicTest.java
@@ -18,6 +18,12 @@
 
 package org.apache.kylin.common.util;
 
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import org.junit.Ignore;
+import org.junit.Test;
+import org.slf4j.LoggerFactory;
+
 import java.io.IOException;
 import java.nio.ByteBuffer;
 import java.text.DateFormat;
@@ -26,6 +32,7 @@ import java.util.ArrayList;
 import java.util.Calendar;
 import java.util.HashMap;
 import java.util.List;
+import java.util.Map;
 import java.util.concurrent.ExecutionException;
 import java.util.concurrent.ExecutorService;
 import java.util.concurrent.Executors;
@@ -33,13 +40,6 @@ import java.util.concurrent.Future;
 import java.util.concurrent.TimeUnit;
 import java.util.concurrent.TimeoutException;
 
-import org.junit.Ignore;
-import org.junit.Test;
-import org.slf4j.LoggerFactory;
-
-import com.google.common.collect.Lists;
-import com.google.common.collect.Maps;
-
 /**
  * 
  * Keep this test case to test basic java functionality
@@ -79,7 +79,15 @@ public class BasicTest {
 
 @Test
 public void testxx() throws InterruptedException {
-System.out.println( 0x8fL);
+Map a = Maps.newLinkedHashMap();
+a.put("1", "1");
+a.put("3", "3");
+a.put("2", "2");
+for (Map.Entry e : a.entrySet()) {
+System.out.println(e.getKey());
+}
+
+System.out.println(0x8fL);
 byte[] space = new byte[100];
 ByteBuffer buffer = ByteBuffer.wrap(space, 10, 20);
 buffer.put((byte) 1);

http://git-wip-us.apache.org/repos/asf/kylin/blob/076c77ae/core-common/src/test/java/org/apache/kylin/common/util/JacksonBean.java
--
diff --git 
a/core-common/src/test/java/org/apache/kylin/common/util/JacksonBean.java 
b/core-common/src/test/java/org/apache/kylin/common/util/JacksonBean.java
index 42357f2..6d1 100644
--- a/core-common/src/test/java/org/apache/kylin/common/util/JacksonBean.java
+++ b/core-common/src/test/java/org/apache/kylin/common/util/JacksonBean.java
@@ -23,6 +23,10 @@ import 
com.fasterxml.jackson.annotation.JsonAutoDetect.Visibility;
 import com.fasterxml.jackson.annotation.JsonInclude;
 import com.fasterxml.jackson.annotation.JsonProperty;
 
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.Map;
+
 @JsonAutoDetect(fieldVisibility = Visibility.NONE, getterVisibility = 
Visibility.NONE, isGetterVisibility = Visibility.NONE, setterVisibility = 
Visibility.NONE)
 public class JacksonBean {
 
@@ -32,6 +36,10 @@ public class JacksonBean {
 @JsonInclude(JsonInclude.Include.NON_NULL)
 private int b;
 
+@JsonProperty("configuration")
+@JsonInclude(JsonInclude.Include.NON_EMPTY)
+private Map configuration = new HashMap();
+
 public String getA() {
 return a;
 }
@@ -48,6 +56,14 @@ public class JacksonBean {
 this.b = b;
 }
 
+public Map getConfiguration() {
+return configuration;
+}
+
+public void setConfiguration(Map configuration) {
+this.configuration = configuration;
+}
+
 @Override
 public String toString() {
 return "JacksonBean{" + "a='" + a + '\'' + ", b=" + b + '}';

[3/6] kylin git commit: KYLIN-2240 Add a toggle to ignore all cube signature inconsistency temporally

[mahongbin]

 KYLIN-2240 Add a toggle to ignore all cube signature inconsistency temporally


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/80018874
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/80018874
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/80018874

Branch: refs/heads/yang21-hbase102
Commit: 80018874c26d17f57bd288654996d3b6508d8294
Parents: 076c77a
Author: Hongbin Ma 
Authored: Thu Dec 1 11:35:34 2016 +0800
Committer: Hongbin Ma 
Committed: Thu Dec 1 11:35:34 2016 +0800

--
 .../apache/kylin/common/KylinConfigBase.java| 39 ++---
 .../org/apache/kylin/cube/model/CubeDesc.java   | 90 +++-
 2 files changed, 76 insertions(+), 53 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/80018874/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
--
diff --git 
a/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java 
b/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
index a91e42b..f35f969 100644
--- a/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
+++ b/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
@@ -18,6 +18,13 @@
 
 package org.apache.kylin.common;
 
+import com.google.common.collect.Maps;
+import com.google.common.collect.Sets;
+import org.apache.commons.lang.StringUtils;
+import org.apache.kylin.common.util.CliCommandExecutor;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
 import java.io.File;
 import java.io.IOException;
 import java.io.Serializable;
@@ -28,14 +35,6 @@ import java.util.SortedSet;
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
 
-import org.apache.commons.lang.StringUtils;
-import org.apache.kylin.common.util.CliCommandExecutor;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import com.google.common.collect.Maps;
-import com.google.common.collect.Sets;
-
 /**
  * An abstract class to encapsulate access to a set of 'properties'.
  * Subclass can override methods in this class to extend the content of the 
'properties',
@@ -174,19 +173,25 @@ abstract public class KylinConfigBase implements 
Serializable {
 setProperty("kylin.storage.url", storageUrl);
 }
 
-/** was for route to hive, not used any more */
+/**
+ * was for route to hive, not used any more
+ */
 @Deprecated
 public String getHiveUrl() {
 return getOptional("hive.url", "");
 }
 
-/** was for route to hive, not used any more */
+/**
+ * was for route to hive, not used any more
+ */
 @Deprecated
 public String getHiveUser() {
 return getOptional("hive.user", "");
 }
 
-/** was for route to hive, not used any more */
+/**
+ * was for route to hive, not used any more
+ */
 @Deprecated
 public String getHivePassword() {
 return getOptional("hive.password", "");
@@ -202,7 +207,7 @@ abstract public class KylinConfigBase implements 
Serializable {
 
 public String[] getRealizationProviders() {
 return getOptionalStringArray("kylin.realization.providers", //
-new String[] { "org.apache.kylin.cube.CubeManager", 
"org.apache.kylin.storage.hybrid.HybridManager" });
+new String[]{"org.apache.kylin.cube.CubeManager", 
"org.apache.kylin.storage.hybrid.HybridManager"});
 }
 
 public CliCommandExecutor getCliCommandExecutor() throws IOException {
@@ -427,6 +432,10 @@ abstract public class KylinConfigBase implements 
Serializable {
 return 
Integer.parseInt(getOptional("kylin.cube.algorithm.auto.mapper.limit", "500"));
 }
 
+public boolean isIgnoreCubeSignatureInconsistency() {
+return 
Boolean.parseBoolean(getOptional("kylin.cube.ignore-signature-inconsistency", 
"false"));
+}
+
 @Deprecated
 public int getCubeAggrGroupMaxSize() {
 return Integer.parseInt(getOptional("kylin.cube.aggrgroup.max.size", 
"12"));
@@ -480,10 +489,11 @@ abstract public class KylinConfigBase implements 
Serializable {
 public float getCubeVisitTimeoutTimes() {
 return 
Float.parseFloat(getOptional("kylin.query.cube.visit.timeout.times", "1"));
 }
-
+
 public int getDerivedInThreshold() {
 return 
Integer.parseInt(getOptional("kylin.query.filter.derived_in.max", "20"));
 }
+
 public int getBadQueryStackTraceDepth() {
 return 
Integer.parseInt(getOptional("kylin.query.badquery.stacktrace.depth", "10"));
 }
@@ -569,7 +579,7 @@ abstract public class KylinConfigBase implements 
Serializable {
 }
 
 public int[] getQueryMetricsPercentilesIntervals() {
-String[] dft = { "60", 

[1/6] kylin git commit: KYLIN-1978: fix compatible issue on Ubuntu [Forced Update!]

[mahongbin]

Repository: kylin
Updated Branches:
  refs/heads/yang21-hbase102 6deff5829 -> 5d7e7f239 (forced update)


KYLIN-1978: fix compatible issue on Ubuntu

Signed-off-by: shaofengshi 


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/02068603
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/02068603
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/02068603

Branch: refs/heads/yang21-hbase102
Commit: 020686034f985a215476259caecfdc81a891e111
Parents: 150bab6
Author: Yiming Liu 
Authored: Sun Sep 18 10:35:17 2016 +0800
Committer: shaofengshi 
Committed: Tue Nov 29 16:19:11 2016 +0800

--
 build/bin/get-properties.sh | 9 ++---
 1 file changed, 2 insertions(+), 7 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/02068603/build/bin/get-properties.sh
--
diff --git a/build/bin/get-properties.sh b/build/bin/get-properties.sh
index 45bc49e..7ce4bd2 100755
--- a/build/bin/get-properties.sh
+++ b/build/bin/get-properties.sh
@@ -26,10 +26,5 @@ then
 fi
 
 IFS=$'\n'
-result=
-for i in `cat ${KYLIN_HOME}/conf/kylin.properties | grep -w "^$1" | grep -v 
'^#' | awk -F= '{ n = index($0,"="); print substr($0,n+1)}' | cut -c 1-`
-do
-   :
-   result=$i
-done
-echo "$result"
\ No newline at end of file
+result=`cat ${KYLIN_HOME}/conf/kylin.properties | grep -w "^$1" | grep -v '^#' 
| awk -F= '{ n = index($0,"="); print substr($0,n+1)}' | cut -c 1- |tail -1`
+echo $result

[5/6] kylin git commit: KYLIN-1528 Create a branch for v1.5 with HBase 1.x API

[mahongbin]

KYLIN-1528 Create a branch for v1.5 with HBase 1.x API


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/02f37bd8
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/02f37bd8
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/02f37bd8

Branch: refs/heads/yang21-hbase102
Commit: 02f37bd8247e6e2067ad6c46f1c3230f2f22e4d9
Parents: 8001887
Author: shaofengshi 
Authored: Wed Mar 23 17:07:05 2016 +0800
Committer: Hongbin Ma 
Committed: Thu Dec 1 13:21:03 2016 +0800

--
 examples/test_case_data/sandbox/hbase-site.xml  | 19 +---
 .../kylin/provision/BuildCubeWithEngine.java| 13 ++-
 pom.xml | 12 +--
 .../kylin/rest/security/AclHBaseStorage.java|  4 +-
 .../rest/security/MockAclHBaseStorage.java  |  8 +-
 .../apache/kylin/rest/security/MockHTable.java  | 95 
 .../rest/security/RealAclHBaseStorage.java  |  9 +-
 .../apache/kylin/rest/service/AclService.java   | 25 +++---
 .../apache/kylin/rest/service/CubeService.java  | 36 +++-
 .../apache/kylin/rest/service/QueryService.java | 24 +++--
 .../apache/kylin/rest/service/UserService.java  | 17 ++--
 .../kylin/storage/hbase/HBaseConnection.java| 44 -
 .../kylin/storage/hbase/HBaseResourceStore.java | 31 +++
 .../kylin/storage/hbase/HBaseStorage.java   |  3 +-
 .../storage/hbase/cube/SimpleHBaseStore.java| 20 ++---
 .../hbase/cube/v1/CubeSegmentTupleIterator.java | 11 +--
 .../storage/hbase/cube/v1/CubeStorageQuery.java |  6 +-
 .../hbase/cube/v1/RegionScannerAdapter.java | 10 ++-
 .../cube/v1/SerializedHBaseTupleIterator.java   |  4 +-
 .../observer/AggregateRegionObserver.java   |  4 +-
 .../observer/AggregationScanner.java| 14 ++-
 .../observer/ObserverAggregationCache.java  | 10 ++-
 .../coprocessor/observer/ObserverEnabler.java   |  4 +-
 .../hbase/cube/v2/CubeHBaseEndpointRPC.java | 13 +--
 .../storage/hbase/cube/v2/CubeHBaseScanRPC.java |  9 +-
 .../coprocessor/endpoint/CubeVisitService.java  |  4 +-
 .../storage/hbase/steps/CubeHTableUtil.java | 16 ++--
 .../storage/hbase/steps/DeprecatedGCStep.java   | 23 ++---
 .../storage/hbase/steps/HBaseCuboidWriter.java  |  7 +-
 .../hbase/steps/HBaseStreamingOutput.java   |  9 +-
 .../kylin/storage/hbase/steps/MergeGCStep.java  | 23 ++---
 .../storage/hbase/util/CleanHtableCLI.java  | 12 +--
 .../storage/hbase/util/CubeMigrationCLI.java| 36 
 .../hbase/util/CubeMigrationCheckCLI.java   | 17 ++--
 .../hbase/util/DeployCoprocessorCLI.java| 22 ++---
 .../hbase/util/ExtendCubeToHybridCLI.java   |  8 +-
 .../hbase/util/GridTableHBaseBenchmark.java | 34 +++
 .../kylin/storage/hbase/util/HBaseClean.java| 18 ++--
 .../hbase/util/HBaseRegionSizeCalculator.java   | 35 
 .../kylin/storage/hbase/util/HBaseUsage.java|  9 +-
 .../storage/hbase/util/HbaseStreamingInput.java | 30 +++
 .../hbase/util/HtableAlterMetadataCLI.java  |  9 +-
 .../storage/hbase/util/OrphanHBaseCleanJob.java | 19 ++--
 .../kylin/storage/hbase/util/PingHBaseCLI.java  | 15 ++--
 .../kylin/storage/hbase/util/RowCounterCLI.java | 11 +--
 .../storage/hbase/util/StorageCleanupJob.java   | 20 +++--
 .../storage/hbase/util/UpdateHTableHostCLI.java | 17 ++--
 .../observer/AggregateRegionObserverTest.java   | 26 ++
 .../v1/filter/TestFuzzyRowFilterV2EndToEnd.java |  5 +-
 .../org/apache/kylin/tool/CubeMigrationCLI.java | 36 
 .../kylin/tool/CubeMigrationCheckCLI.java   | 16 ++--
 .../kylin/tool/ExtendCubeToHybridCLI.java   |  8 +-
 .../apache/kylin/tool/StorageCleanupJob.java| 20 +++--
 53 files changed, 450 insertions(+), 500 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/02f37bd8/examples/test_case_data/sandbox/hbase-site.xml
--
diff --git a/examples/test_case_data/sandbox/hbase-site.xml 
b/examples/test_case_data/sandbox/hbase-site.xml
index 46d5345..734908e 100644
--- a/examples/test_case_data/sandbox/hbase-site.xml
+++ b/examples/test_case_data/sandbox/hbase-site.xml
@@ -190,22 +190,5 @@
 zookeeper.znode.parent
 /hbase-unsecure
 
-
-hbase.client.pause
-100
-General client pause value.  Used mostly as value to wait
-before running a retry of a failed get, region lookup, etc.
-See hbase.client.retries.number for description of how we backoff 
from
-this initial pause amount and how this pause works w/ 
retries.
-
-
-hbase.client.retries.number
-5
-Maximum retries.  Used as maximum for all retryable
-operations such as the getting of a cell's value, starting a row 
update,
-

[5/6] kylin git commit: KYLIN-1528 Create a branch for v1.5 with HBase 1.x API

[mahongbin]

KYLIN-1528 Create a branch for v1.5 with HBase 1.x API


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/02f37bd8
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/02f37bd8
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/02f37bd8

Branch: refs/heads/yang21-cdh5.7
Commit: 02f37bd8247e6e2067ad6c46f1c3230f2f22e4d9
Parents: 8001887
Author: shaofengshi 
Authored: Wed Mar 23 17:07:05 2016 +0800
Committer: Hongbin Ma 
Committed: Thu Dec 1 13:21:03 2016 +0800

--
 examples/test_case_data/sandbox/hbase-site.xml  | 19 +---
 .../kylin/provision/BuildCubeWithEngine.java| 13 ++-
 pom.xml | 12 +--
 .../kylin/rest/security/AclHBaseStorage.java|  4 +-
 .../rest/security/MockAclHBaseStorage.java  |  8 +-
 .../apache/kylin/rest/security/MockHTable.java  | 95 
 .../rest/security/RealAclHBaseStorage.java  |  9 +-
 .../apache/kylin/rest/service/AclService.java   | 25 +++---
 .../apache/kylin/rest/service/CubeService.java  | 36 +++-
 .../apache/kylin/rest/service/QueryService.java | 24 +++--
 .../apache/kylin/rest/service/UserService.java  | 17 ++--
 .../kylin/storage/hbase/HBaseConnection.java| 44 -
 .../kylin/storage/hbase/HBaseResourceStore.java | 31 +++
 .../kylin/storage/hbase/HBaseStorage.java   |  3 +-
 .../storage/hbase/cube/SimpleHBaseStore.java| 20 ++---
 .../hbase/cube/v1/CubeSegmentTupleIterator.java | 11 +--
 .../storage/hbase/cube/v1/CubeStorageQuery.java |  6 +-
 .../hbase/cube/v1/RegionScannerAdapter.java | 10 ++-
 .../cube/v1/SerializedHBaseTupleIterator.java   |  4 +-
 .../observer/AggregateRegionObserver.java   |  4 +-
 .../observer/AggregationScanner.java| 14 ++-
 .../observer/ObserverAggregationCache.java  | 10 ++-
 .../coprocessor/observer/ObserverEnabler.java   |  4 +-
 .../hbase/cube/v2/CubeHBaseEndpointRPC.java | 13 +--
 .../storage/hbase/cube/v2/CubeHBaseScanRPC.java |  9 +-
 .../coprocessor/endpoint/CubeVisitService.java  |  4 +-
 .../storage/hbase/steps/CubeHTableUtil.java | 16 ++--
 .../storage/hbase/steps/DeprecatedGCStep.java   | 23 ++---
 .../storage/hbase/steps/HBaseCuboidWriter.java  |  7 +-
 .../hbase/steps/HBaseStreamingOutput.java   |  9 +-
 .../kylin/storage/hbase/steps/MergeGCStep.java  | 23 ++---
 .../storage/hbase/util/CleanHtableCLI.java  | 12 +--
 .../storage/hbase/util/CubeMigrationCLI.java| 36 
 .../hbase/util/CubeMigrationCheckCLI.java   | 17 ++--
 .../hbase/util/DeployCoprocessorCLI.java| 22 ++---
 .../hbase/util/ExtendCubeToHybridCLI.java   |  8 +-
 .../hbase/util/GridTableHBaseBenchmark.java | 34 +++
 .../kylin/storage/hbase/util/HBaseClean.java| 18 ++--
 .../hbase/util/HBaseRegionSizeCalculator.java   | 35 
 .../kylin/storage/hbase/util/HBaseUsage.java|  9 +-
 .../storage/hbase/util/HbaseStreamingInput.java | 30 +++
 .../hbase/util/HtableAlterMetadataCLI.java  |  9 +-
 .../storage/hbase/util/OrphanHBaseCleanJob.java | 19 ++--
 .../kylin/storage/hbase/util/PingHBaseCLI.java  | 15 ++--
 .../kylin/storage/hbase/util/RowCounterCLI.java | 11 +--
 .../storage/hbase/util/StorageCleanupJob.java   | 20 +++--
 .../storage/hbase/util/UpdateHTableHostCLI.java | 17 ++--
 .../observer/AggregateRegionObserverTest.java   | 26 ++
 .../v1/filter/TestFuzzyRowFilterV2EndToEnd.java |  5 +-
 .../org/apache/kylin/tool/CubeMigrationCLI.java | 36 
 .../kylin/tool/CubeMigrationCheckCLI.java   | 16 ++--
 .../kylin/tool/ExtendCubeToHybridCLI.java   |  8 +-
 .../apache/kylin/tool/StorageCleanupJob.java| 20 +++--
 53 files changed, 450 insertions(+), 500 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/02f37bd8/examples/test_case_data/sandbox/hbase-site.xml
--
diff --git a/examples/test_case_data/sandbox/hbase-site.xml 
b/examples/test_case_data/sandbox/hbase-site.xml
index 46d5345..734908e 100644
--- a/examples/test_case_data/sandbox/hbase-site.xml
+++ b/examples/test_case_data/sandbox/hbase-site.xml
@@ -190,22 +190,5 @@
 zookeeper.znode.parent
 /hbase-unsecure
 
-
-hbase.client.pause
-100
-General client pause value.  Used mostly as value to wait
-before running a retry of a failed get, region lookup, etc.
-See hbase.client.retries.number for description of how we backoff 
from
-this initial pause amount and how this pause works w/ 
retries.
-
-
-hbase.client.retries.number
-5
-Maximum retries.  Used as maximum for all retryable
-operations such as the getting of a cell's value, starting a row 
update,
-  

[3/6] kylin git commit: KYLIN-2240 Add a toggle to ignore all cube signature inconsistency temporally

[mahongbin]

 KYLIN-2240 Add a toggle to ignore all cube signature inconsistency temporally


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/80018874
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/80018874
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/80018874

Branch: refs/heads/yang21-cdh5.7
Commit: 80018874c26d17f57bd288654996d3b6508d8294
Parents: 076c77a
Author: Hongbin Ma 
Authored: Thu Dec 1 11:35:34 2016 +0800
Committer: Hongbin Ma 
Committed: Thu Dec 1 11:35:34 2016 +0800

--
 .../apache/kylin/common/KylinConfigBase.java| 39 ++---
 .../org/apache/kylin/cube/model/CubeDesc.java   | 90 +++-
 2 files changed, 76 insertions(+), 53 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/80018874/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
--
diff --git 
a/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java 
b/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
index a91e42b..f35f969 100644
--- a/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
+++ b/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
@@ -18,6 +18,13 @@
 
 package org.apache.kylin.common;
 
+import com.google.common.collect.Maps;
+import com.google.common.collect.Sets;
+import org.apache.commons.lang.StringUtils;
+import org.apache.kylin.common.util.CliCommandExecutor;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
 import java.io.File;
 import java.io.IOException;
 import java.io.Serializable;
@@ -28,14 +35,6 @@ import java.util.SortedSet;
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
 
-import org.apache.commons.lang.StringUtils;
-import org.apache.kylin.common.util.CliCommandExecutor;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import com.google.common.collect.Maps;
-import com.google.common.collect.Sets;
-
 /**
  * An abstract class to encapsulate access to a set of 'properties'.
  * Subclass can override methods in this class to extend the content of the 
'properties',
@@ -174,19 +173,25 @@ abstract public class KylinConfigBase implements 
Serializable {
 setProperty("kylin.storage.url", storageUrl);
 }
 
-/** was for route to hive, not used any more */
+/**
+ * was for route to hive, not used any more
+ */
 @Deprecated
 public String getHiveUrl() {
 return getOptional("hive.url", "");
 }
 
-/** was for route to hive, not used any more */
+/**
+ * was for route to hive, not used any more
+ */
 @Deprecated
 public String getHiveUser() {
 return getOptional("hive.user", "");
 }
 
-/** was for route to hive, not used any more */
+/**
+ * was for route to hive, not used any more
+ */
 @Deprecated
 public String getHivePassword() {
 return getOptional("hive.password", "");
@@ -202,7 +207,7 @@ abstract public class KylinConfigBase implements 
Serializable {
 
 public String[] getRealizationProviders() {
 return getOptionalStringArray("kylin.realization.providers", //
-new String[] { "org.apache.kylin.cube.CubeManager", 
"org.apache.kylin.storage.hybrid.HybridManager" });
+new String[]{"org.apache.kylin.cube.CubeManager", 
"org.apache.kylin.storage.hybrid.HybridManager"});
 }
 
 public CliCommandExecutor getCliCommandExecutor() throws IOException {
@@ -427,6 +432,10 @@ abstract public class KylinConfigBase implements 
Serializable {
 return 
Integer.parseInt(getOptional("kylin.cube.algorithm.auto.mapper.limit", "500"));
 }
 
+public boolean isIgnoreCubeSignatureInconsistency() {
+return 
Boolean.parseBoolean(getOptional("kylin.cube.ignore-signature-inconsistency", 
"false"));
+}
+
 @Deprecated
 public int getCubeAggrGroupMaxSize() {
 return Integer.parseInt(getOptional("kylin.cube.aggrgroup.max.size", 
"12"));
@@ -480,10 +489,11 @@ abstract public class KylinConfigBase implements 
Serializable {
 public float getCubeVisitTimeoutTimes() {
 return 
Float.parseFloat(getOptional("kylin.query.cube.visit.timeout.times", "1"));
 }
-
+
 public int getDerivedInThreshold() {
 return 
Integer.parseInt(getOptional("kylin.query.filter.derived_in.max", "20"));
 }
+
 public int getBadQueryStackTraceDepth() {
 return 
Integer.parseInt(getOptional("kylin.query.badquery.stacktrace.depth", "10"));
 }
@@ -569,7 +579,7 @@ abstract public class KylinConfigBase implements 
Serializable {
 }
 
 public int[] getQueryMetricsPercentilesIntervals() {
-String[] dft = { "60", "300", 

[1/6] kylin git commit: KYLIN-1978: fix compatible issue on Ubuntu [Forced Update!]

[mahongbin]

Repository: kylin
Updated Branches:
  refs/heads/yang21-cdh5.7 743b76fbf -> 0b6d6662e (forced update)


KYLIN-1978: fix compatible issue on Ubuntu

Signed-off-by: shaofengshi 


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/02068603
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/02068603
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/02068603

Branch: refs/heads/yang21-cdh5.7
Commit: 020686034f985a215476259caecfdc81a891e111
Parents: 150bab6
Author: Yiming Liu 
Authored: Sun Sep 18 10:35:17 2016 +0800
Committer: shaofengshi 
Committed: Tue Nov 29 16:19:11 2016 +0800

--
 build/bin/get-properties.sh | 9 ++---
 1 file changed, 2 insertions(+), 7 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/02068603/build/bin/get-properties.sh
--
diff --git a/build/bin/get-properties.sh b/build/bin/get-properties.sh
index 45bc49e..7ce4bd2 100755
--- a/build/bin/get-properties.sh
+++ b/build/bin/get-properties.sh
@@ -26,10 +26,5 @@ then
 fi
 
 IFS=$'\n'
-result=
-for i in `cat ${KYLIN_HOME}/conf/kylin.properties | grep -w "^$1" | grep -v 
'^#' | awk -F= '{ n = index($0,"="); print substr($0,n+1)}' | cut -c 1-`
-do
-   :
-   result=$i
-done
-echo "$result"
\ No newline at end of file
+result=`cat ${KYLIN_HOME}/conf/kylin.properties | grep -w "^$1" | grep -v '^#' 
| awk -F= '{ n = index($0,"="); print substr($0,n+1)}' | cut -c 1- |tail -1`
+echo $result

[2/6] kylin git commit: org.apache.kylin.metadata.model.FunctionDesc#configuration should use default linkedhashmap to keep order

[mahongbin]

org.apache.kylin.metadata.model.FunctionDesc#configuration should use default 
linkedhashmap to keep order


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/076c77ae
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/076c77ae
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/076c77ae

Branch: refs/heads/yang21-cdh5.7
Commit: 076c77ae217ceb320e06e534b7149f1fc60075ee
Parents: 0206860
Author: Hongbin Ma 
Authored: Thu Dec 1 11:20:53 2016 +0800
Committer: Hongbin Ma 
Committed: Thu Dec 1 11:20:53 2016 +0800

--
 .../org/apache/kylin/common/util/BasicTest.java | 24 ++--
 .../apache/kylin/common/util/JacksonBean.java   | 16 +++
 .../apache/kylin/common/util/JacksonTest.java   | 19 ++---
 .../kylin/metadata/model/FunctionDesc.java  | 29 ++--
 4 files changed, 61 insertions(+), 27 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/076c77ae/core-common/src/test/java/org/apache/kylin/common/util/BasicTest.java
--
diff --git 
a/core-common/src/test/java/org/apache/kylin/common/util/BasicTest.java 
b/core-common/src/test/java/org/apache/kylin/common/util/BasicTest.java
index ee15832..ffdbd25 100644
--- a/core-common/src/test/java/org/apache/kylin/common/util/BasicTest.java
+++ b/core-common/src/test/java/org/apache/kylin/common/util/BasicTest.java
@@ -18,6 +18,12 @@
 
 package org.apache.kylin.common.util;
 
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import org.junit.Ignore;
+import org.junit.Test;
+import org.slf4j.LoggerFactory;
+
 import java.io.IOException;
 import java.nio.ByteBuffer;
 import java.text.DateFormat;
@@ -26,6 +32,7 @@ import java.util.ArrayList;
 import java.util.Calendar;
 import java.util.HashMap;
 import java.util.List;
+import java.util.Map;
 import java.util.concurrent.ExecutionException;
 import java.util.concurrent.ExecutorService;
 import java.util.concurrent.Executors;
@@ -33,13 +40,6 @@ import java.util.concurrent.Future;
 import java.util.concurrent.TimeUnit;
 import java.util.concurrent.TimeoutException;
 
-import org.junit.Ignore;
-import org.junit.Test;
-import org.slf4j.LoggerFactory;
-
-import com.google.common.collect.Lists;
-import com.google.common.collect.Maps;
-
 /**
  * 
  * Keep this test case to test basic java functionality
@@ -79,7 +79,15 @@ public class BasicTest {
 
 @Test
 public void testxx() throws InterruptedException {
-System.out.println( 0x8fL);
+Map a = Maps.newLinkedHashMap();
+a.put("1", "1");
+a.put("3", "3");
+a.put("2", "2");
+for (Map.Entry e : a.entrySet()) {
+System.out.println(e.getKey());
+}
+
+System.out.println(0x8fL);
 byte[] space = new byte[100];
 ByteBuffer buffer = ByteBuffer.wrap(space, 10, 20);
 buffer.put((byte) 1);

http://git-wip-us.apache.org/repos/asf/kylin/blob/076c77ae/core-common/src/test/java/org/apache/kylin/common/util/JacksonBean.java
--
diff --git 
a/core-common/src/test/java/org/apache/kylin/common/util/JacksonBean.java 
b/core-common/src/test/java/org/apache/kylin/common/util/JacksonBean.java
index 42357f2..6d1 100644
--- a/core-common/src/test/java/org/apache/kylin/common/util/JacksonBean.java
+++ b/core-common/src/test/java/org/apache/kylin/common/util/JacksonBean.java
@@ -23,6 +23,10 @@ import 
com.fasterxml.jackson.annotation.JsonAutoDetect.Visibility;
 import com.fasterxml.jackson.annotation.JsonInclude;
 import com.fasterxml.jackson.annotation.JsonProperty;
 
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.Map;
+
 @JsonAutoDetect(fieldVisibility = Visibility.NONE, getterVisibility = 
Visibility.NONE, isGetterVisibility = Visibility.NONE, setterVisibility = 
Visibility.NONE)
 public class JacksonBean {
 
@@ -32,6 +36,10 @@ public class JacksonBean {
 @JsonInclude(JsonInclude.Include.NON_NULL)
 private int b;
 
+@JsonProperty("configuration")
+@JsonInclude(JsonInclude.Include.NON_EMPTY)
+private Map configuration = new HashMap();
+
 public String getA() {
 return a;
 }
@@ -48,6 +56,14 @@ public class JacksonBean {
 this.b = b;
 }
 
+public Map getConfiguration() {
+return configuration;
+}
+
+public void setConfiguration(Map configuration) {
+this.configuration = configuration;
+}
+
 @Override
 public String toString() {
 return "JacksonBean{" + "a='" + a + '\'' + ", b=" + b + '}';

[4/6] kylin git commit: KYLIN-1528 Create a branch for v1.5 with HBase 1.x API

[mahongbin]

http://git-wip-us.apache.org/repos/asf/kylin/blob/02f37bd8/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseStreamingOutput.java
--
diff --git 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseStreamingOutput.java
 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseStreamingOutput.java
index 0e95102..c59fb33 100644
--- 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseStreamingOutput.java
+++ 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseStreamingOutput.java
@@ -27,7 +27,8 @@ import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.fs.FSDataInputStream;
 import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.fs.Path;
-import org.apache.hadoop.hbase.client.HTableInterface;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Table;
 import org.apache.kylin.common.KylinConfig;
 import org.apache.kylin.common.persistence.ResourceStore;
 import org.apache.kylin.cube.CubeSegment;
@@ -56,7 +57,7 @@ public class HBaseStreamingOutput implements IStreamingOutput 
{
 try {
 CubeSegment cubeSegment = (CubeSegment) buildable;
 
-final HTableInterface hTable;
+final Table hTable;
 hTable = createHTable(cubeSegment);
 List cuboidWriters = Lists.newArrayList();
 cuboidWriters.add(new HBaseCuboidWriter(cubeSegment, hTable));
@@ -88,10 +89,10 @@ public class HBaseStreamingOutput implements 
IStreamingOutput {
 }
 }
 
-private HTableInterface createHTable(final CubeSegment cubeSegment) throws 
IOException {
+private Table createHTable(final CubeSegment cubeSegment) throws 
IOException {
 final String hTableName = cubeSegment.getStorageLocationIdentifier();
 CubeHTableUtil.createHTable(cubeSegment, null);
-final HTableInterface hTable = 
HBaseConnection.get(KylinConfig.getInstanceFromEnv().getStorageUrl()).getTable(hTableName);
+final Table hTable = 
HBaseConnection.get(KylinConfig.getInstanceFromEnv().getStorageUrl()).getTable(TableName.valueOf(hTableName));
 logger.info("hTable:" + hTableName + " for segment:" + 
cubeSegment.getName() + " created!");
 return hTable;
 }

http://git-wip-us.apache.org/repos/asf/kylin/blob/02f37bd8/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/MergeGCStep.java
--
diff --git 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/MergeGCStep.java
 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/MergeGCStep.java
index 5b2441c..2f7e164 100644
--- 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/MergeGCStep.java
+++ 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/MergeGCStep.java
@@ -24,11 +24,11 @@ import java.util.Collections;
 import java.util.List;
 
 import org.apache.commons.lang.StringUtils;
-import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.hbase.HTableDescriptor;
-import org.apache.hadoop.hbase.client.HBaseAdmin;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.Connection;
 import org.apache.kylin.common.KylinConfig;
-import org.apache.kylin.common.util.Bytes;
 import org.apache.kylin.job.exception.ExecuteException;
 import org.apache.kylin.job.execution.AbstractExecutable;
 import org.apache.kylin.job.execution.ExecutableContext;
@@ -69,19 +69,20 @@ public class MergeGCStep extends AbstractExecutable {
 List oldTables = getOldHTables();
 if (oldTables != null && oldTables.size() > 0) {
 String metadataUrlPrefix = 
KylinConfig.getInstanceFromEnv().getMetadataUrlPrefix();
-Configuration conf = 
HBaseConnection.getCurrentHBaseConfiguration();
-HBaseAdmin admin = null;
+Admin admin = null;
 try {
-admin = new HBaseAdmin(conf);
+Connection conn = 
HBaseConnection.get(KylinConfig.getInstanceFromEnv().getStorageUrl());
+admin = conn.getAdmin();
+
 for (String table : oldTables) {
-if (admin.tableExists(table)) {
-HTableDescriptor tableDescriptor = 
admin.getTableDescriptor(Bytes.toBytes(table));
+if (admin.tableExists(TableName.valueOf(table))) {
+HTableDescriptor tableDescriptor = 
admin.getTableDescriptor(TableName.valueOf((table)));
 String host = 
tableDescriptor.getValue(IRealizationConstants.HTableTag);
 if (metadataUrlPrefix.equalsIgnoreCase(host)) {
-if (admin.isTableEnabled(table)) {
-admin.disableTable(table);
+

[6/6] kylin git commit: KYLIN-1672 support kylin on cdh 5.7

[mahongbin]

KYLIN-1672 support kylin on cdh 5.7

Signed-off-by: Li Yang 


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/0b6d6662
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/0b6d6662
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/0b6d6662

Branch: refs/heads/yang21-cdh5.7
Commit: 0b6d6662edb660a2672d4dba90ff5cd459343a70
Parents: 02f37bd
Author: Lynne Jiang 
Authored: Mon May 16 03:33:27 2016 -0700
Committer: Hongbin Ma 
Committed: Thu Dec 1 13:22:49 2016 +0800

--
 build/conf/kylin.properties |   3 +
 dev-support/test_all_against_hdp_2_2_4_2_2.sh   |   0
 .../kylin/engine/mr/steps/MockupMapContext.java |  15 +-
 examples/test_case_data/sandbox/core-site.xml   | 146 +++---
 examples/test_case_data/sandbox/hbase-site.xml  | 162 ++
 examples/test_case_data/sandbox/hdfs-site.xml   | 259 ++
 examples/test_case_data/sandbox/mapred-site.xml | 398 ++-
 examples/test_case_data/sandbox/yarn-site.xml   | 496 ++-
 pom.xml |  16 +-
 server/pom.xml  |  36 ++
 .../storage/hbase/steps/MockupMapContext.java   |  19 +-
 tool/pom.xml|  12 +
 12 files changed, 431 insertions(+), 1131 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/0b6d6662/build/conf/kylin.properties
--
diff --git a/build/conf/kylin.properties b/build/conf/kylin.properties
index ed86bdb..ebd8cde 100644
--- a/build/conf/kylin.properties
+++ b/build/conf/kylin.properties
@@ -118,6 +118,9 @@ kylin.job.mapreduce.mapper.input.rows=100
 
 kylin.job.step.timeout=7200
 
+# for secure cdh, filtering hive dependency is risky, so filter nothing
+kylin.job.dependency.filterlist=[^,]+
+
 ### CUBE ###
 
 # 'auto', 'inmem', 'layer' or 'random' for testing

http://git-wip-us.apache.org/repos/asf/kylin/blob/0b6d6662/dev-support/test_all_against_hdp_2_2_4_2_2.sh
--
diff --git a/dev-support/test_all_against_hdp_2_2_4_2_2.sh 
b/dev-support/test_all_against_hdp_2_2_4_2_2.sh
old mode 100644
new mode 100755

http://git-wip-us.apache.org/repos/asf/kylin/blob/0b6d6662/engine-mr/src/test/java/org/apache/kylin/engine/mr/steps/MockupMapContext.java
--
diff --git 
a/engine-mr/src/test/java/org/apache/kylin/engine/mr/steps/MockupMapContext.java
 
b/engine-mr/src/test/java/org/apache/kylin/engine/mr/steps/MockupMapContext.java
index 847071d..9900465 100644
--- 
a/engine-mr/src/test/java/org/apache/kylin/engine/mr/steps/MockupMapContext.java
+++ 
b/engine-mr/src/test/java/org/apache/kylin/engine/mr/steps/MockupMapContext.java
@@ -77,6 +77,7 @@ public class MockupMapContext {
 outKV[0] = key;
 outKV[1] = value;
 }
+
 }
 
 @Override
@@ -99,6 +100,7 @@ public class MockupMapContext {
 throw new NotImplementedException();
 }
 
+
 @Override
 public float getProgress() {
 throw new NotImplementedException();
@@ -195,17 +197,17 @@ public class MockupMapContext {
 }
 
 @Override
-public RawComparator getSortComparator() {
+public boolean userClassesTakesPrecedence() {
 throw new NotImplementedException();
 }
 
 @Override
-public String getJar() {
+public RawComparator getSortComparator() {
 throw new NotImplementedException();
 }
 
 @Override
-public RawComparator getGroupingComparator() {
+public String getJar() {
 throw new NotImplementedException();
 }
 
@@ -221,7 +223,7 @@ public class MockupMapContext {
 
 @Override
 public boolean getProfileEnabled() {
-throw new NotImplementedException();
+return false;
 }
 
 @Override
@@ -308,6 +310,11 @@ public class MockupMapContext {
 public RawComparator getCombinerKeyGroupingComparator() {
 throw new NotImplementedException();
 }
+
+@Override
+public RawComparator getGroupingComparator() {
+return null;
+}
 });
 }
 }

http://git-wip-us.apache.org/repos/asf/kylin/blob/0b6d6662/examples/test_case_data/sandbox/core-site.xml
--
diff --git a/examples/test_case_data/sandbox/core-site.xml 

[5/5] kylin git commit: KYLIN-1528 Create a branch for v1.5 with HBase 1.x API

[mahongbin]

KYLIN-1528 Create a branch for v1.5 with HBase 1.x API


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/02f37bd8
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/02f37bd8
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/02f37bd8

Branch: refs/heads/yang21-hbase1.x
Commit: 02f37bd8247e6e2067ad6c46f1c3230f2f22e4d9
Parents: 8001887
Author: shaofengshi 
Authored: Wed Mar 23 17:07:05 2016 +0800
Committer: Hongbin Ma 
Committed: Thu Dec 1 13:21:03 2016 +0800

--
 examples/test_case_data/sandbox/hbase-site.xml  | 19 +---
 .../kylin/provision/BuildCubeWithEngine.java| 13 ++-
 pom.xml | 12 +--
 .../kylin/rest/security/AclHBaseStorage.java|  4 +-
 .../rest/security/MockAclHBaseStorage.java  |  8 +-
 .../apache/kylin/rest/security/MockHTable.java  | 95 
 .../rest/security/RealAclHBaseStorage.java  |  9 +-
 .../apache/kylin/rest/service/AclService.java   | 25 +++---
 .../apache/kylin/rest/service/CubeService.java  | 36 +++-
 .../apache/kylin/rest/service/QueryService.java | 24 +++--
 .../apache/kylin/rest/service/UserService.java  | 17 ++--
 .../kylin/storage/hbase/HBaseConnection.java| 44 -
 .../kylin/storage/hbase/HBaseResourceStore.java | 31 +++
 .../kylin/storage/hbase/HBaseStorage.java   |  3 +-
 .../storage/hbase/cube/SimpleHBaseStore.java| 20 ++---
 .../hbase/cube/v1/CubeSegmentTupleIterator.java | 11 +--
 .../storage/hbase/cube/v1/CubeStorageQuery.java |  6 +-
 .../hbase/cube/v1/RegionScannerAdapter.java | 10 ++-
 .../cube/v1/SerializedHBaseTupleIterator.java   |  4 +-
 .../observer/AggregateRegionObserver.java   |  4 +-
 .../observer/AggregationScanner.java| 14 ++-
 .../observer/ObserverAggregationCache.java  | 10 ++-
 .../coprocessor/observer/ObserverEnabler.java   |  4 +-
 .../hbase/cube/v2/CubeHBaseEndpointRPC.java | 13 +--
 .../storage/hbase/cube/v2/CubeHBaseScanRPC.java |  9 +-
 .../coprocessor/endpoint/CubeVisitService.java  |  4 +-
 .../storage/hbase/steps/CubeHTableUtil.java | 16 ++--
 .../storage/hbase/steps/DeprecatedGCStep.java   | 23 ++---
 .../storage/hbase/steps/HBaseCuboidWriter.java  |  7 +-
 .../hbase/steps/HBaseStreamingOutput.java   |  9 +-
 .../kylin/storage/hbase/steps/MergeGCStep.java  | 23 ++---
 .../storage/hbase/util/CleanHtableCLI.java  | 12 +--
 .../storage/hbase/util/CubeMigrationCLI.java| 36 
 .../hbase/util/CubeMigrationCheckCLI.java   | 17 ++--
 .../hbase/util/DeployCoprocessorCLI.java| 22 ++---
 .../hbase/util/ExtendCubeToHybridCLI.java   |  8 +-
 .../hbase/util/GridTableHBaseBenchmark.java | 34 +++
 .../kylin/storage/hbase/util/HBaseClean.java| 18 ++--
 .../hbase/util/HBaseRegionSizeCalculator.java   | 35 
 .../kylin/storage/hbase/util/HBaseUsage.java|  9 +-
 .../storage/hbase/util/HbaseStreamingInput.java | 30 +++
 .../hbase/util/HtableAlterMetadataCLI.java  |  9 +-
 .../storage/hbase/util/OrphanHBaseCleanJob.java | 19 ++--
 .../kylin/storage/hbase/util/PingHBaseCLI.java  | 15 ++--
 .../kylin/storage/hbase/util/RowCounterCLI.java | 11 +--
 .../storage/hbase/util/StorageCleanupJob.java   | 20 +++--
 .../storage/hbase/util/UpdateHTableHostCLI.java | 17 ++--
 .../observer/AggregateRegionObserverTest.java   | 26 ++
 .../v1/filter/TestFuzzyRowFilterV2EndToEnd.java |  5 +-
 .../org/apache/kylin/tool/CubeMigrationCLI.java | 36 
 .../kylin/tool/CubeMigrationCheckCLI.java   | 16 ++--
 .../kylin/tool/ExtendCubeToHybridCLI.java   |  8 +-
 .../apache/kylin/tool/StorageCleanupJob.java| 20 +++--
 53 files changed, 450 insertions(+), 500 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/02f37bd8/examples/test_case_data/sandbox/hbase-site.xml
--
diff --git a/examples/test_case_data/sandbox/hbase-site.xml 
b/examples/test_case_data/sandbox/hbase-site.xml
index 46d5345..734908e 100644
--- a/examples/test_case_data/sandbox/hbase-site.xml
+++ b/examples/test_case_data/sandbox/hbase-site.xml
@@ -190,22 +190,5 @@
 zookeeper.znode.parent
 /hbase-unsecure
 
-
-hbase.client.pause
-100
-General client pause value.  Used mostly as value to wait
-before running a retry of a failed get, region lookup, etc.
-See hbase.client.retries.number for description of how we backoff 
from
-this initial pause amount and how this pause works w/ 
retries.
-
-
-hbase.client.retries.number
-5
-Maximum retries.  Used as maximum for all retryable
-operations such as the getting of a cell's value, starting a row 
update,
-

[1/5] kylin git commit: KYLIN-1978: fix compatible issue on Ubuntu [Forced Update!]

[mahongbin]

Repository: kylin
Updated Branches:
  refs/heads/yang21-hbase1.x 1db21cd5f -> 02f37bd82 (forced update)


KYLIN-1978: fix compatible issue on Ubuntu

Signed-off-by: shaofengshi 


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/02068603
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/02068603
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/02068603

Branch: refs/heads/yang21-hbase1.x
Commit: 020686034f985a215476259caecfdc81a891e111
Parents: 150bab6
Author: Yiming Liu 
Authored: Sun Sep 18 10:35:17 2016 +0800
Committer: shaofengshi 
Committed: Tue Nov 29 16:19:11 2016 +0800

--
 build/bin/get-properties.sh | 9 ++---
 1 file changed, 2 insertions(+), 7 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/02068603/build/bin/get-properties.sh
--
diff --git a/build/bin/get-properties.sh b/build/bin/get-properties.sh
index 45bc49e..7ce4bd2 100755
--- a/build/bin/get-properties.sh
+++ b/build/bin/get-properties.sh
@@ -26,10 +26,5 @@ then
 fi
 
 IFS=$'\n'
-result=
-for i in `cat ${KYLIN_HOME}/conf/kylin.properties | grep -w "^$1" | grep -v 
'^#' | awk -F= '{ n = index($0,"="); print substr($0,n+1)}' | cut -c 1-`
-do
-   :
-   result=$i
-done
-echo "$result"
\ No newline at end of file
+result=`cat ${KYLIN_HOME}/conf/kylin.properties | grep -w "^$1" | grep -v '^#' 
| awk -F= '{ n = index($0,"="); print substr($0,n+1)}' | cut -c 1- |tail -1`
+echo $result

[3/5] kylin git commit: KYLIN-2240 Add a toggle to ignore all cube signature inconsistency temporally

[mahongbin]

 KYLIN-2240 Add a toggle to ignore all cube signature inconsistency temporally


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/80018874
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/80018874
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/80018874

Branch: refs/heads/yang21-hbase1.x
Commit: 80018874c26d17f57bd288654996d3b6508d8294
Parents: 076c77a
Author: Hongbin Ma 
Authored: Thu Dec 1 11:35:34 2016 +0800
Committer: Hongbin Ma 
Committed: Thu Dec 1 11:35:34 2016 +0800

--
 .../apache/kylin/common/KylinConfigBase.java| 39 ++---
 .../org/apache/kylin/cube/model/CubeDesc.java   | 90 +++-
 2 files changed, 76 insertions(+), 53 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/80018874/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
--
diff --git 
a/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java 
b/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
index a91e42b..f35f969 100644
--- a/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
+++ b/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
@@ -18,6 +18,13 @@
 
 package org.apache.kylin.common;
 
+import com.google.common.collect.Maps;
+import com.google.common.collect.Sets;
+import org.apache.commons.lang.StringUtils;
+import org.apache.kylin.common.util.CliCommandExecutor;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
 import java.io.File;
 import java.io.IOException;
 import java.io.Serializable;
@@ -28,14 +35,6 @@ import java.util.SortedSet;
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
 
-import org.apache.commons.lang.StringUtils;
-import org.apache.kylin.common.util.CliCommandExecutor;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import com.google.common.collect.Maps;
-import com.google.common.collect.Sets;
-
 /**
  * An abstract class to encapsulate access to a set of 'properties'.
  * Subclass can override methods in this class to extend the content of the 
'properties',
@@ -174,19 +173,25 @@ abstract public class KylinConfigBase implements 
Serializable {
 setProperty("kylin.storage.url", storageUrl);
 }
 
-/** was for route to hive, not used any more */
+/**
+ * was for route to hive, not used any more
+ */
 @Deprecated
 public String getHiveUrl() {
 return getOptional("hive.url", "");
 }
 
-/** was for route to hive, not used any more */
+/**
+ * was for route to hive, not used any more
+ */
 @Deprecated
 public String getHiveUser() {
 return getOptional("hive.user", "");
 }
 
-/** was for route to hive, not used any more */
+/**
+ * was for route to hive, not used any more
+ */
 @Deprecated
 public String getHivePassword() {
 return getOptional("hive.password", "");
@@ -202,7 +207,7 @@ abstract public class KylinConfigBase implements 
Serializable {
 
 public String[] getRealizationProviders() {
 return getOptionalStringArray("kylin.realization.providers", //
-new String[] { "org.apache.kylin.cube.CubeManager", 
"org.apache.kylin.storage.hybrid.HybridManager" });
+new String[]{"org.apache.kylin.cube.CubeManager", 
"org.apache.kylin.storage.hybrid.HybridManager"});
 }
 
 public CliCommandExecutor getCliCommandExecutor() throws IOException {
@@ -427,6 +432,10 @@ abstract public class KylinConfigBase implements 
Serializable {
 return 
Integer.parseInt(getOptional("kylin.cube.algorithm.auto.mapper.limit", "500"));
 }
 
+public boolean isIgnoreCubeSignatureInconsistency() {
+return 
Boolean.parseBoolean(getOptional("kylin.cube.ignore-signature-inconsistency", 
"false"));
+}
+
 @Deprecated
 public int getCubeAggrGroupMaxSize() {
 return Integer.parseInt(getOptional("kylin.cube.aggrgroup.max.size", 
"12"));
@@ -480,10 +489,11 @@ abstract public class KylinConfigBase implements 
Serializable {
 public float getCubeVisitTimeoutTimes() {
 return 
Float.parseFloat(getOptional("kylin.query.cube.visit.timeout.times", "1"));
 }
-
+
 public int getDerivedInThreshold() {
 return 
Integer.parseInt(getOptional("kylin.query.filter.derived_in.max", "20"));
 }
+
 public int getBadQueryStackTraceDepth() {
 return 
Integer.parseInt(getOptional("kylin.query.badquery.stacktrace.depth", "10"));
 }
@@ -569,7 +579,7 @@ abstract public class KylinConfigBase implements 
Serializable {
 }
 
 public int[] getQueryMetricsPercentilesIntervals() {
-String[] dft = { "60", 

[4/5] kylin git commit: KYLIN-1528 Create a branch for v1.5 with HBase 1.x API

[mahongbin]

http://git-wip-us.apache.org/repos/asf/kylin/blob/02f37bd8/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseStreamingOutput.java
--
diff --git 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseStreamingOutput.java
 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseStreamingOutput.java
index 0e95102..c59fb33 100644
--- 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseStreamingOutput.java
+++ 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseStreamingOutput.java
@@ -27,7 +27,8 @@ import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.fs.FSDataInputStream;
 import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.fs.Path;
-import org.apache.hadoop.hbase.client.HTableInterface;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Table;
 import org.apache.kylin.common.KylinConfig;
 import org.apache.kylin.common.persistence.ResourceStore;
 import org.apache.kylin.cube.CubeSegment;
@@ -56,7 +57,7 @@ public class HBaseStreamingOutput implements IStreamingOutput 
{
 try {
 CubeSegment cubeSegment = (CubeSegment) buildable;
 
-final HTableInterface hTable;
+final Table hTable;
 hTable = createHTable(cubeSegment);
 List cuboidWriters = Lists.newArrayList();
 cuboidWriters.add(new HBaseCuboidWriter(cubeSegment, hTable));
@@ -88,10 +89,10 @@ public class HBaseStreamingOutput implements 
IStreamingOutput {
 }
 }
 
-private HTableInterface createHTable(final CubeSegment cubeSegment) throws 
IOException {
+private Table createHTable(final CubeSegment cubeSegment) throws 
IOException {
 final String hTableName = cubeSegment.getStorageLocationIdentifier();
 CubeHTableUtil.createHTable(cubeSegment, null);
-final HTableInterface hTable = 
HBaseConnection.get(KylinConfig.getInstanceFromEnv().getStorageUrl()).getTable(hTableName);
+final Table hTable = 
HBaseConnection.get(KylinConfig.getInstanceFromEnv().getStorageUrl()).getTable(TableName.valueOf(hTableName));
 logger.info("hTable:" + hTableName + " for segment:" + 
cubeSegment.getName() + " created!");
 return hTable;
 }

http://git-wip-us.apache.org/repos/asf/kylin/blob/02f37bd8/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/MergeGCStep.java
--
diff --git 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/MergeGCStep.java
 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/MergeGCStep.java
index 5b2441c..2f7e164 100644
--- 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/MergeGCStep.java
+++ 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/MergeGCStep.java
@@ -24,11 +24,11 @@ import java.util.Collections;
 import java.util.List;
 
 import org.apache.commons.lang.StringUtils;
-import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.hbase.HTableDescriptor;
-import org.apache.hadoop.hbase.client.HBaseAdmin;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.Connection;
 import org.apache.kylin.common.KylinConfig;
-import org.apache.kylin.common.util.Bytes;
 import org.apache.kylin.job.exception.ExecuteException;
 import org.apache.kylin.job.execution.AbstractExecutable;
 import org.apache.kylin.job.execution.ExecutableContext;
@@ -69,19 +69,20 @@ public class MergeGCStep extends AbstractExecutable {
 List oldTables = getOldHTables();
 if (oldTables != null && oldTables.size() > 0) {
 String metadataUrlPrefix = 
KylinConfig.getInstanceFromEnv().getMetadataUrlPrefix();
-Configuration conf = 
HBaseConnection.getCurrentHBaseConfiguration();
-HBaseAdmin admin = null;
+Admin admin = null;
 try {
-admin = new HBaseAdmin(conf);
+Connection conn = 
HBaseConnection.get(KylinConfig.getInstanceFromEnv().getStorageUrl());
+admin = conn.getAdmin();
+
 for (String table : oldTables) {
-if (admin.tableExists(table)) {
-HTableDescriptor tableDescriptor = 
admin.getTableDescriptor(Bytes.toBytes(table));
+if (admin.tableExists(TableName.valueOf(table))) {
+HTableDescriptor tableDescriptor = 
admin.getTableDescriptor(TableName.valueOf((table)));
 String host = 
tableDescriptor.getValue(IRealizationConstants.HTableTag);
 if (metadataUrlPrefix.equalsIgnoreCase(host)) {
-if (admin.isTableEnabled(table)) {
-admin.disableTable(table);
+

[2/5] kylin git commit: org.apache.kylin.metadata.model.FunctionDesc#configuration should use default linkedhashmap to keep order

[mahongbin]

org.apache.kylin.metadata.model.FunctionDesc#configuration should use default 
linkedhashmap to keep order


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/076c77ae
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/076c77ae
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/076c77ae

Branch: refs/heads/yang21-hbase1.x
Commit: 076c77ae217ceb320e06e534b7149f1fc60075ee
Parents: 0206860
Author: Hongbin Ma 
Authored: Thu Dec 1 11:20:53 2016 +0800
Committer: Hongbin Ma 
Committed: Thu Dec 1 11:20:53 2016 +0800

--
 .../org/apache/kylin/common/util/BasicTest.java | 24 ++--
 .../apache/kylin/common/util/JacksonBean.java   | 16 +++
 .../apache/kylin/common/util/JacksonTest.java   | 19 ++---
 .../kylin/metadata/model/FunctionDesc.java  | 29 ++--
 4 files changed, 61 insertions(+), 27 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/076c77ae/core-common/src/test/java/org/apache/kylin/common/util/BasicTest.java
--
diff --git 
a/core-common/src/test/java/org/apache/kylin/common/util/BasicTest.java 
b/core-common/src/test/java/org/apache/kylin/common/util/BasicTest.java
index ee15832..ffdbd25 100644
--- a/core-common/src/test/java/org/apache/kylin/common/util/BasicTest.java
+++ b/core-common/src/test/java/org/apache/kylin/common/util/BasicTest.java
@@ -18,6 +18,12 @@
 
 package org.apache.kylin.common.util;
 
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import org.junit.Ignore;
+import org.junit.Test;
+import org.slf4j.LoggerFactory;
+
 import java.io.IOException;
 import java.nio.ByteBuffer;
 import java.text.DateFormat;
@@ -26,6 +32,7 @@ import java.util.ArrayList;
 import java.util.Calendar;
 import java.util.HashMap;
 import java.util.List;
+import java.util.Map;
 import java.util.concurrent.ExecutionException;
 import java.util.concurrent.ExecutorService;
 import java.util.concurrent.Executors;
@@ -33,13 +40,6 @@ import java.util.concurrent.Future;
 import java.util.concurrent.TimeUnit;
 import java.util.concurrent.TimeoutException;
 
-import org.junit.Ignore;
-import org.junit.Test;
-import org.slf4j.LoggerFactory;
-
-import com.google.common.collect.Lists;
-import com.google.common.collect.Maps;
-
 /**
  * 
  * Keep this test case to test basic java functionality
@@ -79,7 +79,15 @@ public class BasicTest {
 
 @Test
 public void testxx() throws InterruptedException {
-System.out.println( 0x8fL);
+Map a = Maps.newLinkedHashMap();
+a.put("1", "1");
+a.put("3", "3");
+a.put("2", "2");
+for (Map.Entry e : a.entrySet()) {
+System.out.println(e.getKey());
+}
+
+System.out.println(0x8fL);
 byte[] space = new byte[100];
 ByteBuffer buffer = ByteBuffer.wrap(space, 10, 20);
 buffer.put((byte) 1);

http://git-wip-us.apache.org/repos/asf/kylin/blob/076c77ae/core-common/src/test/java/org/apache/kylin/common/util/JacksonBean.java
--
diff --git 
a/core-common/src/test/java/org/apache/kylin/common/util/JacksonBean.java 
b/core-common/src/test/java/org/apache/kylin/common/util/JacksonBean.java
index 42357f2..6d1 100644
--- a/core-common/src/test/java/org/apache/kylin/common/util/JacksonBean.java
+++ b/core-common/src/test/java/org/apache/kylin/common/util/JacksonBean.java
@@ -23,6 +23,10 @@ import 
com.fasterxml.jackson.annotation.JsonAutoDetect.Visibility;
 import com.fasterxml.jackson.annotation.JsonInclude;
 import com.fasterxml.jackson.annotation.JsonProperty;
 
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.Map;
+
 @JsonAutoDetect(fieldVisibility = Visibility.NONE, getterVisibility = 
Visibility.NONE, isGetterVisibility = Visibility.NONE, setterVisibility = 
Visibility.NONE)
 public class JacksonBean {
 
@@ -32,6 +36,10 @@ public class JacksonBean {
 @JsonInclude(JsonInclude.Include.NON_NULL)
 private int b;
 
+@JsonProperty("configuration")
+@JsonInclude(JsonInclude.Include.NON_EMPTY)
+private Map configuration = new HashMap();
+
 public String getA() {
 return a;
 }
@@ -48,6 +56,14 @@ public class JacksonBean {
 this.b = b;
 }
 
+public Map getConfiguration() {
+return configuration;
+}
+
+public void setConfiguration(Map configuration) {
+this.configuration = configuration;
+}
+
 @Override
 public String toString() {
 return "JacksonBean{" + "a='" + a + '\'' + ", b=" + b + '}';

[2/2] kylin git commit: KYLIN-2240 Add a toggle to ignore all cube signature inconsistency temporally

[mahongbin]

 KYLIN-2240 Add a toggle to ignore all cube signature inconsistency temporally


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/28ba1eae
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/28ba1eae
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/28ba1eae

Branch: refs/heads/master
Commit: 28ba1eaeac8fd29ff3872f6d14604813a89b3a6e
Parents: 93bf0d0
Author: Hongbin Ma 
Authored: Thu Dec 1 11:35:34 2016 +0800
Committer: Hongbin Ma 
Committed: Thu Dec 1 11:46:00 2016 +0800

--
 .../java/org/apache/kylin/common/KylinConfigBase.java   | 12 
 .../main/java/org/apache/kylin/cube/model/CubeDesc.java |  6 +-
 2 files changed, 13 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/28ba1eae/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
--
diff --git 
a/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java 
b/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
index 766c04d..3c10826 100644
--- a/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
+++ b/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
@@ -79,7 +79,7 @@ abstract public class KylinConfigBase implements Serializable 
{
 protected KylinConfigBase(Properties props, boolean force) {
 this.properties = force ? props : BCC.check(props);
 }
-
+
 final protected String getOptional(String prop) {
 return getOptional(prop, null);
 }
@@ -161,7 +161,7 @@ abstract public class KylinConfigBase implements 
Serializable {
 public boolean isDevEnv() {
 return "DEV".equals(getOptional("kylin.env", "DEV"));
 }
-
+
 public String getDeployEnv() {
 return getOptional("kylin.env", "DEV");
 }
@@ -211,7 +211,7 @@ abstract public class KylinConfigBase implements 
Serializable {
 public Map getCubeCustomMeasureTypes() {
 return getPropertiesByPrefix("kylin.metadata.custom-measure-types.");
 }
-
+
 // 

 // DICTIONARY & SNAPSHOT
 // 

@@ -278,6 +278,10 @@ abstract public class KylinConfigBase implements 
Serializable {
 return 
Integer.parseInt(getOptional("kylin.cube.algorithm.inmem-split-limit", "500"));
 }
 
+public boolean isIgnoreCubeSignatureInconsistency() {
+return 
Boolean.parseBoolean(getOptional("kylin.cube.ignore-signature-inconsistency", 
"false"));
+}
+
 @Deprecated
 public int getCubeAggrGroupMaxSize() {
 return Integer.parseInt(getOptional("kylin.cube.aggrgroup.max-size", 
"12"));
@@ -298,7 +302,7 @@ abstract public class KylinConfigBase implements 
Serializable {
 public void setMaxBuildingSegments(int maxBuildingSegments) {
 setProperty("kylin.cube.max-building-segments", 
String.valueOf(maxBuildingSegments));
 }
-
+
 // 

 // JOB
 // 


http://git-wip-us.apache.org/repos/asf/kylin/blob/28ba1eae/core-cube/src/main/java/org/apache/kylin/cube/model/CubeDesc.java
--
diff --git a/core-cube/src/main/java/org/apache/kylin/cube/model/CubeDesc.java 
b/core-cube/src/main/java/org/apache/kylin/cube/model/CubeDesc.java
index 47063d0..327ce57 100644
--- a/core-cube/src/main/java/org/apache/kylin/cube/model/CubeDesc.java
+++ b/core-cube/src/main/java/org/apache/kylin/cube/model/CubeDesc.java
@@ -466,6 +466,11 @@ public class CubeDesc extends RootPersistentEntity 
implements IEngineAware {
  * @return
  */
 public boolean checkSignature() {
+if (this.getConfig().isIgnoreCubeSignatureInconsistency()) {
+logger.info("Skip checking cube signature");
+return true;
+}
+
 if (KylinVersion.getCurrentVersion().isCompatibleWith(new 
KylinVersion(getVersion())) && 
!KylinVersion.getCurrentVersion().isSignatureCompatibleWith(new 
KylinVersion(getVersion( {
 logger.info("checkSignature on {} is skipped as the its version is 
{} (not signature compatible but compatible) ", getName(), getVersion());
 return true;
@@ -1102,7 +1107,6 @@ public class CubeDesc extends RootPersistentEntity 
implements IEngineAware {
 return newCubeDesc;
 }
 
-
 private Collection ensureOrder(Collection c) {
 TreeSet set = new TreeSet();
 for (Object o : c)

[1/2] kylin git commit: org.apache.kylin.metadata.model.FunctionDesc#configuration should use default linkedhashmap to keep order

[mahongbin]

Repository: kylin
Updated Branches:
  refs/heads/master 30cb1ac67 -> 28ba1eaea


org.apache.kylin.metadata.model.FunctionDesc#configuration should use default 
linkedhashmap to keep order


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/93bf0d02
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/93bf0d02
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/93bf0d02

Branch: refs/heads/master
Commit: 93bf0d0214fe446d794b61e02fb18ff8097611d1
Parents: 30cb1ac
Author: Hongbin Ma 
Authored: Thu Dec 1 11:20:53 2016 +0800
Committer: Hongbin Ma 
Committed: Thu Dec 1 11:40:25 2016 +0800

--
 .../apache/kylin/common/util/JacksonBean.java| 16 
 .../apache/kylin/common/util/JacksonTest.java| 19 +++
 .../kylin/metadata/model/FunctionDesc.java   |  8 
 3 files changed, 35 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/93bf0d02/core-common/src/test/java/org/apache/kylin/common/util/JacksonBean.java
--
diff --git 
a/core-common/src/test/java/org/apache/kylin/common/util/JacksonBean.java 
b/core-common/src/test/java/org/apache/kylin/common/util/JacksonBean.java
index 42357f2..6d1 100644
--- a/core-common/src/test/java/org/apache/kylin/common/util/JacksonBean.java
+++ b/core-common/src/test/java/org/apache/kylin/common/util/JacksonBean.java
@@ -23,6 +23,10 @@ import 
com.fasterxml.jackson.annotation.JsonAutoDetect.Visibility;
 import com.fasterxml.jackson.annotation.JsonInclude;
 import com.fasterxml.jackson.annotation.JsonProperty;
 
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.Map;
+
 @JsonAutoDetect(fieldVisibility = Visibility.NONE, getterVisibility = 
Visibility.NONE, isGetterVisibility = Visibility.NONE, setterVisibility = 
Visibility.NONE)
 public class JacksonBean {
 
@@ -32,6 +36,10 @@ public class JacksonBean {
 @JsonInclude(JsonInclude.Include.NON_NULL)
 private int b;
 
+@JsonProperty("configuration")
+@JsonInclude(JsonInclude.Include.NON_EMPTY)
+private Map configuration = new HashMap();
+
 public String getA() {
 return a;
 }
@@ -48,6 +56,14 @@ public class JacksonBean {
 this.b = b;
 }
 
+public Map getConfiguration() {
+return configuration;
+}
+
+public void setConfiguration(Map configuration) {
+this.configuration = configuration;
+}
+
 @Override
 public String toString() {
 return "JacksonBean{" + "a='" + a + '\'' + ", b=" + b + '}';

http://git-wip-us.apache.org/repos/asf/kylin/blob/93bf0d02/core-common/src/test/java/org/apache/kylin/common/util/JacksonTest.java
--
diff --git 
a/core-common/src/test/java/org/apache/kylin/common/util/JacksonTest.java 
b/core-common/src/test/java/org/apache/kylin/common/util/JacksonTest.java
index 81be7eb..0d7097a 100644
--- a/core-common/src/test/java/org/apache/kylin/common/util/JacksonTest.java
+++ b/core-common/src/test/java/org/apache/kylin/common/util/JacksonTest.java
@@ -18,22 +18,33 @@
 
 package org.apache.kylin.common.util;
 
-import java.io.IOException;
-
 import org.junit.Test;
 
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+
 public class JacksonTest {
 @Test
 public void foo() throws IOException {
+HashMap a = new HashMap();
+a.put("1", "1");
+a.put("3", "3");
+a.put("2", "2");
+
+
 JacksonBean bean = new JacksonBean();
 bean.setA("valuea");
+bean.setConfiguration(a);
 
 String s = JsonUtil.writeValueAsString(bean);
 System.out.println(s);
 
-JacksonBean desBean = (JacksonBean) 
JsonUtil.readValue("{\"a\":\"valuea\"}", JacksonBean.class);
+JacksonBean desBean = (JacksonBean) 
JsonUtil.readValue("{\"a\":\"valuea\",\"b\":0,\"configuration\":{\"2\":\"2\",\"3\":\"3\",\"1\":\"1\"}}",
 JacksonBean.class);
+
 String x2 = JsonUtil.writeValueAsString(desBean);
-System.out.println(desBean);
 System.out.println(x2);
+
+System.out.println(desBean);
 }
 }

http://git-wip-us.apache.org/repos/asf/kylin/blob/93bf0d02/core-metadata/src/main/java/org/apache/kylin/metadata/model/FunctionDesc.java
--
diff --git 
a/core-metadata/src/main/java/org/apache/kylin/metadata/model/FunctionDesc.java 
b/core-metadata/src/main/java/org/apache/kylin/metadata/model/FunctionDesc.java
index ae7f805..7b41552 100644
--- 

kylin git commit: org.apache.kylin.metadata.model.FunctionDesc#configuration should use default linkedhashmap to keep order

[mahongbin]

Repository: kylin
Updated Branches:
  refs/heads/yang21 020686034 -> 076c77ae2


org.apache.kylin.metadata.model.FunctionDesc#configuration should use default 
linkedhashmap to keep order


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/076c77ae
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/076c77ae
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/076c77ae

Branch: refs/heads/yang21
Commit: 076c77ae217ceb320e06e534b7149f1fc60075ee
Parents: 0206860
Author: Hongbin Ma 
Authored: Thu Dec 1 11:20:53 2016 +0800
Committer: Hongbin Ma 
Committed: Thu Dec 1 11:20:53 2016 +0800

--
 .../org/apache/kylin/common/util/BasicTest.java | 24 ++--
 .../apache/kylin/common/util/JacksonBean.java   | 16 +++
 .../apache/kylin/common/util/JacksonTest.java   | 19 ++---
 .../kylin/metadata/model/FunctionDesc.java  | 29 ++--
 4 files changed, 61 insertions(+), 27 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/076c77ae/core-common/src/test/java/org/apache/kylin/common/util/BasicTest.java
--
diff --git 
a/core-common/src/test/java/org/apache/kylin/common/util/BasicTest.java 
b/core-common/src/test/java/org/apache/kylin/common/util/BasicTest.java
index ee15832..ffdbd25 100644
--- a/core-common/src/test/java/org/apache/kylin/common/util/BasicTest.java
+++ b/core-common/src/test/java/org/apache/kylin/common/util/BasicTest.java
@@ -18,6 +18,12 @@
 
 package org.apache.kylin.common.util;
 
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import org.junit.Ignore;
+import org.junit.Test;
+import org.slf4j.LoggerFactory;
+
 import java.io.IOException;
 import java.nio.ByteBuffer;
 import java.text.DateFormat;
@@ -26,6 +32,7 @@ import java.util.ArrayList;
 import java.util.Calendar;
 import java.util.HashMap;
 import java.util.List;
+import java.util.Map;
 import java.util.concurrent.ExecutionException;
 import java.util.concurrent.ExecutorService;
 import java.util.concurrent.Executors;
@@ -33,13 +40,6 @@ import java.util.concurrent.Future;
 import java.util.concurrent.TimeUnit;
 import java.util.concurrent.TimeoutException;
 
-import org.junit.Ignore;
-import org.junit.Test;
-import org.slf4j.LoggerFactory;
-
-import com.google.common.collect.Lists;
-import com.google.common.collect.Maps;
-
 /**
  * 
  * Keep this test case to test basic java functionality
@@ -79,7 +79,15 @@ public class BasicTest {
 
 @Test
 public void testxx() throws InterruptedException {
-System.out.println( 0x8fL);
+Map a = Maps.newLinkedHashMap();
+a.put("1", "1");
+a.put("3", "3");
+a.put("2", "2");
+for (Map.Entry e : a.entrySet()) {
+System.out.println(e.getKey());
+}
+
+System.out.println(0x8fL);
 byte[] space = new byte[100];
 ByteBuffer buffer = ByteBuffer.wrap(space, 10, 20);
 buffer.put((byte) 1);

http://git-wip-us.apache.org/repos/asf/kylin/blob/076c77ae/core-common/src/test/java/org/apache/kylin/common/util/JacksonBean.java
--
diff --git 
a/core-common/src/test/java/org/apache/kylin/common/util/JacksonBean.java 
b/core-common/src/test/java/org/apache/kylin/common/util/JacksonBean.java
index 42357f2..6d1 100644
--- a/core-common/src/test/java/org/apache/kylin/common/util/JacksonBean.java
+++ b/core-common/src/test/java/org/apache/kylin/common/util/JacksonBean.java
@@ -23,6 +23,10 @@ import 
com.fasterxml.jackson.annotation.JsonAutoDetect.Visibility;
 import com.fasterxml.jackson.annotation.JsonInclude;
 import com.fasterxml.jackson.annotation.JsonProperty;
 
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.Map;
+
 @JsonAutoDetect(fieldVisibility = Visibility.NONE, getterVisibility = 
Visibility.NONE, isGetterVisibility = Visibility.NONE, setterVisibility = 
Visibility.NONE)
 public class JacksonBean {
 
@@ -32,6 +36,10 @@ public class JacksonBean {
 @JsonInclude(JsonInclude.Include.NON_NULL)
 private int b;
 
+@JsonProperty("configuration")
+@JsonInclude(JsonInclude.Include.NON_EMPTY)
+private Map configuration = new HashMap();
+
 public String getA() {
 return a;
 }
@@ -48,6 +56,14 @@ public class JacksonBean {
 this.b = b;
 }
 
+public Map getConfiguration() {
+return configuration;
+}
+
+public void setConfiguration(Map configuration) {
+this.configuration = configuration;
+}
+
 @Override
 public String toString() {
 return 

[2/2] kylin git commit: minor, massin udf reverse

[mahongbin]

minor, massin udf reverse

Signed-off-by: Hongbin Ma 


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/b1b90adb
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/b1b90adb
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/b1b90adb

Branch: refs/heads/master
Commit: b1b90adbe896e808776caf49ad7172979cf860df
Parents: 875a593
Author: Roger Shi 
Authored: Wed Nov 30 13:35:12 2016 +0800
Committer: Hongbin Ma 
Committed: Wed Nov 30 13:49:16 2016 +0800

--
 .../metadata/filter/UDF/MassInTupleFilter.java  | 28 +++-
 1 file changed, 27 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/b1b90adb/core-metadata/src/main/java/org/apache/kylin/metadata/filter/UDF/MassInTupleFilter.java
--
diff --git 
a/core-metadata/src/main/java/org/apache/kylin/metadata/filter/UDF/MassInTupleFilter.java
 
b/core-metadata/src/main/java/org/apache/kylin/metadata/filter/UDF/MassInTupleFilter.java
index 3f0546c..29c5550 100644
--- 
a/core-metadata/src/main/java/org/apache/kylin/metadata/filter/UDF/MassInTupleFilter.java
+++ 
b/core-metadata/src/main/java/org/apache/kylin/metadata/filter/UDF/MassInTupleFilter.java
@@ -50,6 +50,7 @@ public class MassInTupleFilter extends FunctionTupleFilter {
 private String filterTableName;//key in MetadataManager.extFilterMap
 private String filterTableResourceIdentifier;//HDFS path, or hbase table 
name depending on FilterTableType
 private Functions.FilterTableType filterTableType;
+private boolean reverse = false;
 
 public MassInTupleFilter() {
 super(Lists. newArrayList(), 
TupleFilter.FilterOperatorEnum.MASSIN);
@@ -66,7 +67,18 @@ public class MassInTupleFilter extends FunctionTupleFilter {
 valueProvider = 
VALUE_PROVIDER_FACTORY.getProvider(filterTableType, 
filterTableResourceIdentifier, column);
 }
 boolean ret = valueProvider.getMassInValues().contains(colValue);
-return ret;
+return reverse ? !ret : ret;
+}
+
+@Override
+public TupleFilter reverse() {
+try {
+MassInTupleFilter result = (MassInTupleFilter) this.clone();
+result.setReverse(!this.isReverse());
+return result;
+} catch (CloneNotSupportedException e) {
+throw new UnsupportedOperationException(e);
+}
 }
 
 @Override
@@ -144,4 +156,18 @@ public class MassInTupleFilter extends FunctionTupleFilter 
{
 return false;
 }
 
+public boolean isReverse() {
+return reverse;
+}
+
+public void setReverse(boolean reverse) {
+this.reverse = reverse;
+}
+
+@Override
+protected Object clone() throws CloneNotSupportedException {
+MassInTupleFilter result = new MassInTupleFilter();
+result.setReverse(this.isReverse());
+return result;
+}
 }

[kylin] Git Push Summary

[mahongbin]

Repository: kylin
Updated Branches:
  refs/heads/beforepomupdate [deleted] 1ed00678f

kylin git commit: minor, trim build log

[mahongbin]

Repository: kylin
Updated Branches:
  refs/heads/master 6afcb2690 -> 8530ebd69


minor, trim build log

Revert "KYLIN-2210 call CubeStatsReader.print at SaveStatisticsStep"

This reverts commit 3ab966b650debb83eb219c9ed8d357d141466776.


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/8530ebd6
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/8530ebd6
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/8530ebd6

Branch: refs/heads/master
Commit: 8530ebd692b3362f7fb9eb07160c8c069d467fc4
Parents: 6afcb26
Author: Hongbin Ma 
Authored: Fri Nov 25 09:52:47 2016 +0800
Committer: Hongbin Ma 
Committed: Fri Nov 25 09:58:10 2016 +0800

--
 .../kylin/engine/mr/common/CubeStatsReader.java|  2 +-
 .../kylin/engine/mr/steps/SaveStatisticsStep.java  | 17 +++--
 2 files changed, 4 insertions(+), 15 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/8530ebd6/engine-mr/src/main/java/org/apache/kylin/engine/mr/common/CubeStatsReader.java
--
diff --git 
a/engine-mr/src/main/java/org/apache/kylin/engine/mr/common/CubeStatsReader.java
 
b/engine-mr/src/main/java/org/apache/kylin/engine/mr/common/CubeStatsReader.java
index c6839d6..c917cfb 100644
--- 
a/engine-mr/src/main/java/org/apache/kylin/engine/mr/common/CubeStatsReader.java
+++ 
b/engine-mr/src/main/java/org/apache/kylin/engine/mr/common/CubeStatsReader.java
@@ -227,7 +227,7 @@ public class CubeStatsReader {
 return ret;
 }
 
-public void print(PrintWriter out) {
+private void print(PrintWriter out) {
 Map cuboidRows = getCuboidRowEstimatesHLL();
 Map cuboidSizes = getCuboidSizeMap();
 List cuboids = new ArrayList(cuboidRows.keySet());

http://git-wip-us.apache.org/repos/asf/kylin/blob/8530ebd6/engine-mr/src/main/java/org/apache/kylin/engine/mr/steps/SaveStatisticsStep.java
--
diff --git 
a/engine-mr/src/main/java/org/apache/kylin/engine/mr/steps/SaveStatisticsStep.java
 
b/engine-mr/src/main/java/org/apache/kylin/engine/mr/steps/SaveStatisticsStep.java
index 79346a5..020c62c 100644
--- 
a/engine-mr/src/main/java/org/apache/kylin/engine/mr/steps/SaveStatisticsStep.java
+++ 
b/engine-mr/src/main/java/org/apache/kylin/engine/mr/steps/SaveStatisticsStep.java
@@ -19,8 +19,6 @@
 package org.apache.kylin.engine.mr.steps;
 
 import java.io.IOException;
-import java.io.PrintWriter;
-import java.io.StringWriter;
 import java.util.Random;
 
 import org.apache.hadoop.fs.FSDataInputStream;
@@ -31,8 +29,8 @@ import org.apache.kylin.common.KylinConfig;
 import org.apache.kylin.common.persistence.ResourceStore;
 import org.apache.kylin.cube.CubeSegment;
 import org.apache.kylin.engine.mr.CubingJob;
-import org.apache.kylin.engine.mr.CubingJob.AlgorithmEnum;
 import org.apache.kylin.engine.mr.HadoopUtil;
+import org.apache.kylin.engine.mr.CubingJob.AlgorithmEnum;
 import org.apache.kylin.engine.mr.common.BatchConstants;
 import org.apache.kylin.engine.mr.common.CubeStatsReader;
 import org.apache.kylin.job.exception.ExecuteException;
@@ -86,16 +84,6 @@ public class SaveStatisticsStep extends AbstractExecutable {
 
 private void decideCubingAlgorithm(CubeSegment seg, KylinConfig kylinConf) 
throws IOException {
 String algPref = kylinConf.getCubeAlgorithm();
-
-CubeStatsReader cubeStats = new CubeStatsReader(seg, kylinConf);
-StringWriter sw = new StringWriter();
-PrintWriter pw = new PrintWriter(sw);
-cubeStats.print(pw);
-pw.flush();
-pw.close();
-logger.info("Cube Stats Estimation for segment {} :", seg.toString());
-logger.info(sw.toString());
-
 AlgorithmEnum alg;
 if (AlgorithmEnum.INMEM.name().equalsIgnoreCase(algPref)) {
 alg = AlgorithmEnum.INMEM;
@@ -115,13 +103,14 @@ public class SaveStatisticsStep extends 
AbstractExecutable {
 } else if ("random".equalsIgnoreCase(algPref)) { // for testing
 alg = new Random().nextBoolean() ? AlgorithmEnum.INMEM : 
AlgorithmEnum.LAYER;
 } else { // the default
+CubeStatsReader cubeStats = new CubeStatsReader(seg, 
kylinConf);
 int mapperNumber = cubeStats.getMapperNumberOfFirstBuild();
 int mapperNumLimit = 
kylinConf.getCubeAlgorithmAutoMapperLimit();
 double mapperOverlapRatio = 
cubeStats.getMapperOverlapRatioOfFirstBuild();
 double overlapThreshold = 
kylinConf.getCubeAlgorithmAutoThreshold();
 logger.info("mapperNumber for " + seg + " is " + mapperNumber 
+ " and threshold is " + mapperNumLimit);
   

[1/2] kylin git commit: minor, fix kylin.sh version

[mahongbin]

Repository: kylin
Updated Branches:
  refs/heads/master 3e4e82f06 -> 83adb8eb8


minor, fix kylin.sh version


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/83adb8eb
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/83adb8eb
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/83adb8eb

Branch: refs/heads/master
Commit: 83adb8eb8700d388aa042d833e7fea9746282624
Parents: b8e764d
Author: Hongbin Ma 
Authored: Wed Nov 23 18:14:40 2016 +0800
Committer: Hongbin Ma 
Committed: Wed Nov 23 18:14:45 2016 +0800

--
 build/bin/kylin.sh | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/83adb8eb/build/bin/kylin.sh
--
diff --git a/build/bin/kylin.sh b/build/bin/kylin.sh
index b0627ef..d3aa4cb 100644
--- a/build/bin/kylin.sh
+++ b/build/bin/kylin.sh
@@ -151,6 +151,7 @@ then
 
 elif [ "$1" = "version" ]
 then
+retrieveDependency
 exec hbase 
-Dlog4j.configuration=file:${KYLIN_HOME}/conf/kylin-tools-log4j.properties 
org.apache.kylin.common.KylinVersion
 exit 0
 

[2/2] kylin git commit: KYLIN-2227 rename kylin-log4j.properties to kylin-tools-log4j.properties and move it to global conf folder

[mahongbin]

KYLIN-2227 rename kylin-log4j.properties to kylin-tools-log4j.properties and 
move it to global conf folder


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/b8e764d4
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/b8e764d4
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/b8e764d4

Branch: refs/heads/master
Commit: b8e764d434ab12b9fb7b902ac8dea0d6b792dec5
Parents: 3e4e82f
Author: Hongbin Ma 
Authored: Wed Nov 23 17:46:04 2016 +0800
Committer: Hongbin Ma 
Committed: Wed Nov 23 18:14:45 2016 +0800

--
 build/bin/kylin.sh  |  4 +--
 build/conf/kylin-tools-log4j.properties | 36 
 .../src/main/resources/kylin-log4j.properties   | 28 ---
 .../gridtable/AggregationCacheSpillTest.java|  1 -
 kylin-it/pom.xml|  6 ++--
 pom.xml |  2 +-
 .../java/org/apache/kylin/rest/DebugTomcat.java |  2 +-
 7 files changed, 43 insertions(+), 36 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/b8e764d4/build/bin/kylin.sh
--
diff --git a/build/bin/kylin.sh b/build/bin/kylin.sh
index fff6227..b0627ef 100644
--- a/build/bin/kylin.sh
+++ b/build/bin/kylin.sh
@@ -151,7 +151,7 @@ then
 
 elif [ "$1" = "version" ]
 then
-exec hbase -Dlog4j.configuration=kylin-log4j.properties 
org.apache.kylin.common.KylinVersion
+exec hbase 
-Dlog4j.configuration=file:${KYLIN_HOME}/conf/kylin-tools-log4j.properties 
org.apache.kylin.common.KylinVersion
 exit 0
 
 elif [ "$1" = "diag" ]
@@ -170,7 +170,7 @@ then
 then source ${dir}/setenv-tool.sh
 fi
 
-exec hbase ${KYLIN_EXTRA_START_OPTS} 
-Dkylin.hive.dependency=${hive_dependency} 
-Dkylin.hbase.dependency=${hbase_dependency} 
-Dlog4j.configuration=kylin-log4j.properties "$@"
+exec hbase ${KYLIN_EXTRA_START_OPTS} 
-Dkylin.hive.dependency=${hive_dependency} 
-Dkylin.hbase.dependency=${hbase_dependency} 
-Dlog4j.configuration=file:${KYLIN_HOME}/conf/kylin-tools-log4j.properties "$@"
 
 else
 quit "Usage: 'kylin.sh [-v] start' or 'kylin.sh [-v] stop'"

http://git-wip-us.apache.org/repos/asf/kylin/blob/b8e764d4/build/conf/kylin-tools-log4j.properties
--
diff --git a/build/conf/kylin-tools-log4j.properties 
b/build/conf/kylin-tools-log4j.properties
new file mode 100644
index 000..0ba7d2e
--- /dev/null
+++ b/build/conf/kylin-tools-log4j.properties
@@ -0,0 +1,36 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#  
+# http://www.apache.org/licenses/LICENSE-2.0
+#  
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+
+# the kylin-tools-log4j.properties is mainly for configuring log properties on 
kylin tools, including:
+#   1. tools launched by kylin.sh script, e.g. DeployCoprocessorCLI
+#   2. DebugTomcat
+#   3. others
+# 
+# It's called kylin-tools-log4j.properties so that it won't distract users 
from the other more important log4j config file: kylin-server-log4j.properties  
+# enable this by -Dlog4j.configuration=kylin-tools-log4j.properties
+
+log4j.rootLogger=INFO,stdout
+
+log4j.appender.stdout=org.apache.log4j.ConsoleAppender
+log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
+log4j.appender.stdout.layout.ConversionPattern=%d{ISO8601} %-5p [%t %c{1}:%L]: 
%m%n
+
+#log4j.logger.org.apache.hadoop=ERROR
+log4j.logger.org.apache.kylin=DEBUG
+log4j.logger.org.springframework=WARN

http://git-wip-us.apache.org/repos/asf/kylin/blob/b8e764d4/core-common/src/main/resources/kylin-log4j.properties
--
diff --git a/core-common/src/main/resources/kylin-log4j.properties 
b/core-common/src/main/resources/kylin-log4j.properties
deleted file mode 100644
index e7201b3..000
--- a/core-common/src/main/resources/kylin-log4j.properties
+++ /dev/null
@@ -1,28 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements.  See the NOTICE file 

kylin git commit: KYLIN-2224 fix this issue

[mahongbin]

Repository: kylin
Updated Branches:
  refs/heads/master 53b5a2752 -> 1ec1ffccd


KYLIN-2224 fix this issue

Signed-off-by: Hongbin Ma 


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/1ec1ffcc
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/1ec1ffcc
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/1ec1ffcc

Branch: refs/heads/master
Commit: 1ec1ffccd787036cbc89d0a549adcbb0469cb1ec
Parents: 53b5a27
Author: Cheng Wang 
Authored: Wed Nov 23 11:19:04 2016 +0800
Committer: Hongbin Ma 
Committed: Wed Nov 23 11:46:35 2016 +0800

--
 .../main/java/org/apache/kylin/cube/RawQueryLastHacker.java  | 5 +++--
 .../kylin/storage/gtrecord/GTCubeStorageQueryBase.java   | 2 +-
 .../apache/kylin/storage/hbase/cube/v1/CubeStorageQuery.java | 8 
 3 files changed, 8 insertions(+), 7 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/1ec1ffcc/core-cube/src/main/java/org/apache/kylin/cube/RawQueryLastHacker.java
--
diff --git 
a/core-cube/src/main/java/org/apache/kylin/cube/RawQueryLastHacker.java 
b/core-cube/src/main/java/org/apache/kylin/cube/RawQueryLastHacker.java
index 50c644e..b0a4823 100644
--- a/core-cube/src/main/java/org/apache/kylin/cube/RawQueryLastHacker.java
+++ b/core-cube/src/main/java/org/apache/kylin/cube/RawQueryLastHacker.java
@@ -24,6 +24,7 @@ import org.apache.kylin.metadata.model.MeasureDesc;
 import org.apache.kylin.metadata.model.ParameterDesc;
 import org.apache.kylin.metadata.model.TblColRef;
 import org.apache.kylin.metadata.realization.SQLDigest;
+import org.apache.kylin.metadata.tuple.TupleInfo;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -31,7 +32,7 @@ public class RawQueryLastHacker {
 
 private static final Logger logger = 
LoggerFactory.getLogger(RawQueryLastHacker.class);
 
-public static void hackNoAggregations(SQLDigest sqlDigest, CubeDesc 
cubeDesc) {
+public static void hackNoAggregations(SQLDigest sqlDigest, CubeDesc 
cubeDesc, TupleInfo tupleInfo) {
 if (!sqlDigest.isRawQuery) {
 return;
 }
@@ -45,7 +46,7 @@ public class RawQueryLastHacker {
 boolean isSelectAll = sqlDigest.allColumns.isEmpty() || 
sqlDigest.allColumns.equals(sqlDigest.filterColumns);
 for (TblColRef col : cubeDesc.listAllColumns()) {
 if (cubeDesc.listDimensionColumnsIncludingDerived().contains(col) 
|| isSelectAll) {
-if (col.getTable().equals(sqlDigest.factTable))
+if (tupleInfo.hasColumn(col))
 sqlDigest.allColumns.add(col);
 }
 }

http://git-wip-us.apache.org/repos/asf/kylin/blob/1ec1ffcc/core-storage/src/main/java/org/apache/kylin/storage/gtrecord/GTCubeStorageQueryBase.java
--
diff --git 
a/core-storage/src/main/java/org/apache/kylin/storage/gtrecord/GTCubeStorageQueryBase.java
 
b/core-storage/src/main/java/org/apache/kylin/storage/gtrecord/GTCubeStorageQueryBase.java
index 24dd2fb..9c74cca 100644
--- 
a/core-storage/src/main/java/org/apache/kylin/storage/gtrecord/GTCubeStorageQueryBase.java
+++ 
b/core-storage/src/main/java/org/apache/kylin/storage/gtrecord/GTCubeStorageQueryBase.java
@@ -77,7 +77,7 @@ public abstract class GTCubeStorageQueryBase implements 
IStorageQuery {
 sqlDigest.includeSubqueryJoinParticipants();
 
 //cope with queries with no aggregations
-RawQueryLastHacker.hackNoAggregations(sqlDigest, cubeDesc);
+RawQueryLastHacker.hackNoAggregations(sqlDigest, cubeDesc, 
returnTupleInfo);
 
 // Customized measure taking effect: e.g. allow custom measures to 
help raw queries
 notifyBeforeStorageQuery(sqlDigest);

http://git-wip-us.apache.org/repos/asf/kylin/blob/1ec1ffcc/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/cube/v1/CubeStorageQuery.java
--
diff --git 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/cube/v1/CubeStorageQuery.java
 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/cube/v1/CubeStorageQuery.java
index 3c73dc6..9af0faf 100644
--- 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/cube/v1/CubeStorageQuery.java
+++ 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/cube/v1/CubeStorageQuery.java
@@ -46,10 +46,10 @@ import org.apache.kylin.cube.RawQueryLastHacker;
 import org.apache.kylin.cube.cuboid.Cuboid;
 import org.apache.kylin.cube.kv.RowConstants;
 import org.apache.kylin.cube.model.CubeDesc;
+import org.apache.kylin.cube.model.CubeDesc.DeriveInfo;
 import org.apache.kylin.cube.model.HBaseColumnDesc;
 

kylin git commit: int deorecated show better

[mahongbin]

Repository: kylin
Updated Branches:
  refs/heads/master a058049c8 -> bce9ab822


int deorecated show better

Signed-off-by: Hongbin Ma 


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/bce9ab82
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/bce9ab82
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/bce9ab82

Branch: refs/heads/master
Commit: bce9ab822492772c7c2b42421941421dfb27f9f7
Parents: a058049
Author: luguosheng <550175...@qq.com>
Authored: Tue Nov 22 17:12:35 2016 +0800
Committer: Hongbin Ma 
Committed: Tue Nov 22 17:14:04 2016 +0800

--
 webapp/app/js/controllers/cubeEdit.js | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/bce9ab82/webapp/app/js/controllers/cubeEdit.js
--
diff --git a/webapp/app/js/controllers/cubeEdit.js 
b/webapp/app/js/controllers/cubeEdit.js
index 10f8185..888662d 100755
--- a/webapp/app/js/controllers/cubeEdit.js
+++ b/webapp/app/js/controllers/cubeEdit.js
@@ -49,7 +49,7 @@ KylinApp.controller('CubeEditCtrl', function ($scope, $q, 
$routeParams, $locatio
   var name = value;
   var typeVersion=+encodings[i];
   if(value=="int"){
-name = "int deprecated";
+name = "int (deprecated)";
   }
   if(/\d+/.test(""+typeVersion)&>1){
   for(var s=1;s<=typeVersion;s++){

[3/4] kylin git commit: KYLIN-1528 Create a branch for v1.5 with HBase 1.x API

[mahongbin]

KYLIN-1528 Create a branch for v1.5 with HBase 1.x API


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/e1b83470
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/e1b83470
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/e1b83470

Branch: refs/heads/yang21-cdh5.7
Commit: e1b83470be3e8f315ad6c5001b97e62d5682dd34
Parents: 150bab6
Author: shaofengshi 
Authored: Wed Mar 23 17:07:05 2016 +0800
Committer: Hongbin Ma 
Committed: Tue Nov 22 17:13:01 2016 +0800

--
 examples/test_case_data/sandbox/hbase-site.xml  | 19 +---
 .../kylin/provision/BuildCubeWithEngine.java| 13 ++-
 pom.xml | 12 +--
 .../kylin/rest/security/AclHBaseStorage.java|  4 +-
 .../rest/security/MockAclHBaseStorage.java  |  8 +-
 .../apache/kylin/rest/security/MockHTable.java  | 95 
 .../rest/security/RealAclHBaseStorage.java  |  9 +-
 .../apache/kylin/rest/service/AclService.java   | 25 +++---
 .../apache/kylin/rest/service/CubeService.java  | 36 +++-
 .../apache/kylin/rest/service/QueryService.java | 24 +++--
 .../apache/kylin/rest/service/UserService.java  | 17 ++--
 .../kylin/storage/hbase/HBaseConnection.java| 44 -
 .../kylin/storage/hbase/HBaseResourceStore.java | 31 +++
 .../kylin/storage/hbase/HBaseStorage.java   |  3 +-
 .../storage/hbase/cube/SimpleHBaseStore.java| 20 ++---
 .../hbase/cube/v1/CubeSegmentTupleIterator.java | 11 +--
 .../storage/hbase/cube/v1/CubeStorageQuery.java |  6 +-
 .../hbase/cube/v1/RegionScannerAdapter.java | 10 ++-
 .../cube/v1/SerializedHBaseTupleIterator.java   |  4 +-
 .../observer/AggregateRegionObserver.java   |  4 +-
 .../observer/AggregationScanner.java| 14 ++-
 .../observer/ObserverAggregationCache.java  | 10 ++-
 .../coprocessor/observer/ObserverEnabler.java   |  4 +-
 .../hbase/cube/v2/CubeHBaseEndpointRPC.java | 13 +--
 .../storage/hbase/cube/v2/CubeHBaseScanRPC.java |  9 +-
 .../coprocessor/endpoint/CubeVisitService.java  |  4 +-
 .../storage/hbase/steps/CubeHTableUtil.java | 16 ++--
 .../storage/hbase/steps/DeprecatedGCStep.java   | 23 ++---
 .../storage/hbase/steps/HBaseCuboidWriter.java  |  7 +-
 .../hbase/steps/HBaseStreamingOutput.java   |  9 +-
 .../kylin/storage/hbase/steps/MergeGCStep.java  | 23 ++---
 .../storage/hbase/util/CleanHtableCLI.java  | 12 +--
 .../storage/hbase/util/CubeMigrationCLI.java| 36 
 .../hbase/util/CubeMigrationCheckCLI.java   | 17 ++--
 .../hbase/util/DeployCoprocessorCLI.java| 22 ++---
 .../hbase/util/ExtendCubeToHybridCLI.java   |  8 +-
 .../hbase/util/GridTableHBaseBenchmark.java | 34 +++
 .../kylin/storage/hbase/util/HBaseClean.java| 18 ++--
 .../hbase/util/HBaseRegionSizeCalculator.java   | 35 
 .../kylin/storage/hbase/util/HBaseUsage.java|  9 +-
 .../storage/hbase/util/HbaseStreamingInput.java | 30 +++
 .../hbase/util/HtableAlterMetadataCLI.java  |  9 +-
 .../storage/hbase/util/OrphanHBaseCleanJob.java | 19 ++--
 .../kylin/storage/hbase/util/PingHBaseCLI.java  | 15 ++--
 .../kylin/storage/hbase/util/RowCounterCLI.java | 11 +--
 .../storage/hbase/util/StorageCleanupJob.java   | 20 +++--
 .../storage/hbase/util/UpdateHTableHostCLI.java | 17 ++--
 .../observer/AggregateRegionObserverTest.java   | 26 ++
 .../v1/filter/TestFuzzyRowFilterV2EndToEnd.java |  5 +-
 .../org/apache/kylin/tool/CubeMigrationCLI.java | 36 
 .../kylin/tool/CubeMigrationCheckCLI.java   | 16 ++--
 .../kylin/tool/ExtendCubeToHybridCLI.java   |  8 +-
 .../apache/kylin/tool/StorageCleanupJob.java| 20 +++--
 53 files changed, 450 insertions(+), 500 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/e1b83470/examples/test_case_data/sandbox/hbase-site.xml
--
diff --git a/examples/test_case_data/sandbox/hbase-site.xml 
b/examples/test_case_data/sandbox/hbase-site.xml
index 46d5345..734908e 100644
--- a/examples/test_case_data/sandbox/hbase-site.xml
+++ b/examples/test_case_data/sandbox/hbase-site.xml
@@ -190,22 +190,5 @@
 zookeeper.znode.parent
 /hbase-unsecure
 
-
-hbase.client.pause
-100
-General client pause value.  Used mostly as value to wait
-before running a retry of a failed get, region lookup, etc.
-See hbase.client.retries.number for description of how we backoff 
from
-this initial pause amount and how this pause works w/ 
retries.
-
-
-hbase.client.retries.number
-5
-Maximum retries.  Used as maximum for all retryable
-operations such as the getting of a cell's value, starting a row 
update,
- 

[2/4] kylin git commit: KYLIN-1528 Create a branch for v1.5 with HBase 1.x API

[mahongbin]

http://git-wip-us.apache.org/repos/asf/kylin/blob/e1b83470/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseStreamingOutput.java
--
diff --git 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseStreamingOutput.java
 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseStreamingOutput.java
index 0e95102..c59fb33 100644
--- 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseStreamingOutput.java
+++ 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseStreamingOutput.java
@@ -27,7 +27,8 @@ import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.fs.FSDataInputStream;
 import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.fs.Path;
-import org.apache.hadoop.hbase.client.HTableInterface;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Table;
 import org.apache.kylin.common.KylinConfig;
 import org.apache.kylin.common.persistence.ResourceStore;
 import org.apache.kylin.cube.CubeSegment;
@@ -56,7 +57,7 @@ public class HBaseStreamingOutput implements IStreamingOutput 
{
 try {
 CubeSegment cubeSegment = (CubeSegment) buildable;
 
-final HTableInterface hTable;
+final Table hTable;
 hTable = createHTable(cubeSegment);
 List cuboidWriters = Lists.newArrayList();
 cuboidWriters.add(new HBaseCuboidWriter(cubeSegment, hTable));
@@ -88,10 +89,10 @@ public class HBaseStreamingOutput implements 
IStreamingOutput {
 }
 }
 
-private HTableInterface createHTable(final CubeSegment cubeSegment) throws 
IOException {
+private Table createHTable(final CubeSegment cubeSegment) throws 
IOException {
 final String hTableName = cubeSegment.getStorageLocationIdentifier();
 CubeHTableUtil.createHTable(cubeSegment, null);
-final HTableInterface hTable = 
HBaseConnection.get(KylinConfig.getInstanceFromEnv().getStorageUrl()).getTable(hTableName);
+final Table hTable = 
HBaseConnection.get(KylinConfig.getInstanceFromEnv().getStorageUrl()).getTable(TableName.valueOf(hTableName));
 logger.info("hTable:" + hTableName + " for segment:" + 
cubeSegment.getName() + " created!");
 return hTable;
 }

http://git-wip-us.apache.org/repos/asf/kylin/blob/e1b83470/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/MergeGCStep.java
--
diff --git 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/MergeGCStep.java
 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/MergeGCStep.java
index 5b2441c..2f7e164 100644
--- 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/MergeGCStep.java
+++ 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/MergeGCStep.java
@@ -24,11 +24,11 @@ import java.util.Collections;
 import java.util.List;
 
 import org.apache.commons.lang.StringUtils;
-import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.hbase.HTableDescriptor;
-import org.apache.hadoop.hbase.client.HBaseAdmin;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.Connection;
 import org.apache.kylin.common.KylinConfig;
-import org.apache.kylin.common.util.Bytes;
 import org.apache.kylin.job.exception.ExecuteException;
 import org.apache.kylin.job.execution.AbstractExecutable;
 import org.apache.kylin.job.execution.ExecutableContext;
@@ -69,19 +69,20 @@ public class MergeGCStep extends AbstractExecutable {
 List oldTables = getOldHTables();
 if (oldTables != null && oldTables.size() > 0) {
 String metadataUrlPrefix = 
KylinConfig.getInstanceFromEnv().getMetadataUrlPrefix();
-Configuration conf = 
HBaseConnection.getCurrentHBaseConfiguration();
-HBaseAdmin admin = null;
+Admin admin = null;
 try {
-admin = new HBaseAdmin(conf);
+Connection conn = 
HBaseConnection.get(KylinConfig.getInstanceFromEnv().getStorageUrl());
+admin = conn.getAdmin();
+
 for (String table : oldTables) {
-if (admin.tableExists(table)) {
-HTableDescriptor tableDescriptor = 
admin.getTableDescriptor(Bytes.toBytes(table));
+if (admin.tableExists(TableName.valueOf(table))) {
+HTableDescriptor tableDescriptor = 
admin.getTableDescriptor(TableName.valueOf((table)));
 String host = 
tableDescriptor.getValue(IRealizationConstants.HTableTag);
 if (metadataUrlPrefix.equalsIgnoreCase(host)) {
-if (admin.isTableEnabled(table)) {
-admin.disableTable(table);
+

[1/4] kylin git commit: revert KYLIN-2199 because KYLIN-2191 is not a major fix [Forced Update!]

[mahongbin]

Repository: kylin
Updated Branches:
  refs/heads/yang21-cdh5.7 229cffdbd -> ac2d7c41d (forced update)


revert KYLIN-2199 because KYLIN-2191 is not a major fix


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/150bab61
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/150bab61
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/150bab61

Branch: refs/heads/yang21-cdh5.7
Commit: 150bab61a77c0900bf8d43f569b710d8b9177f0d
Parents: d4715b5
Author: Hongbin Ma 
Authored: Tue Nov 22 15:32:31 2016 +0800
Committer: Hongbin Ma 
Committed: Tue Nov 22 15:51:08 2016 +0800

--
 .../gridtable/DimEncodingPreserveOrderTest.java |   4 +-
 .../dimension/DimensionEncodingFactory.java |   4 -
 .../apache/kylin/dimension/IntegerDimEnc.java   |   3 +-
 .../apache/kylin/dimension/IntegerDimEncV2.java | 228 ---
 .../kylin/dimension/IntegerDimEncTest.java  |  16 +-
 5 files changed, 11 insertions(+), 244 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/150bab61/core-cube/src/test/java/org/apache/kylin/gridtable/DimEncodingPreserveOrderTest.java
--
diff --git 
a/core-cube/src/test/java/org/apache/kylin/gridtable/DimEncodingPreserveOrderTest.java
 
b/core-cube/src/test/java/org/apache/kylin/gridtable/DimEncodingPreserveOrderTest.java
index d572e56..87d37be 100644
--- 
a/core-cube/src/test/java/org/apache/kylin/gridtable/DimEncodingPreserveOrderTest.java
+++ 
b/core-cube/src/test/java/org/apache/kylin/gridtable/DimEncodingPreserveOrderTest.java
@@ -27,7 +27,7 @@ import org.apache.kylin.common.util.ByteArray;
 import org.apache.kylin.common.util.Bytes;
 import org.apache.kylin.dimension.DimensionEncoding;
 import org.apache.kylin.dimension.FixedLenHexDimEnc;
-import org.apache.kylin.dimension.IntegerDimEncV2;
+import org.apache.kylin.dimension.IntegerDimEnc;
 import org.apache.kylin.dimension.OneMoreByteVLongDimEnc;
 import org.junit.BeforeClass;
 import org.junit.Test;
@@ -99,7 +99,7 @@ public class DimEncodingPreserveOrderTest {
 @Test
 public void testVLongDimEncPreserveOrder() {
 for (int i = 1; i <= successValue.size(); i++) {
-IntegerDimEncV2 enc = new IntegerDimEncV2(i);
+IntegerDimEnc enc = new IntegerDimEnc(i);
 List encodedValues = Lists.newArrayList();
 for (long value : successValue.get(i - 1)) {
 encodedValues.add(encode(enc, value));

http://git-wip-us.apache.org/repos/asf/kylin/blob/150bab61/core-metadata/src/main/java/org/apache/kylin/dimension/DimensionEncodingFactory.java
--
diff --git 
a/core-metadata/src/main/java/org/apache/kylin/dimension/DimensionEncodingFactory.java
 
b/core-metadata/src/main/java/org/apache/kylin/dimension/DimensionEncodingFactory.java
index b617a19..2b6f894 100644
--- 
a/core-metadata/src/main/java/org/apache/kylin/dimension/DimensionEncodingFactory.java
+++ 
b/core-metadata/src/main/java/org/apache/kylin/dimension/DimensionEncodingFactory.java
@@ -117,10 +117,6 @@ public abstract class DimensionEncodingFactory {
 map.put(Pair.newPair(IntegerDimEnc.ENCODING_NAME, 
value.getCurrentVersion()), value);
 }
 {
-IntegerDimEncV2.Factory value = new IntegerDimEncV2.Factory();
-map.put(Pair.newPair(IntegerDimEncV2.ENCODING_NAME, 
value.getCurrentVersion()), value);
-}
-{
 FixedLenHexDimEnc.Factory value = new 
FixedLenHexDimEnc.Factory();
 map.put(Pair.newPair(FixedLenHexDimEnc.ENCODING_NAME, 
value.getCurrentVersion()), value);
 }

http://git-wip-us.apache.org/repos/asf/kylin/blob/150bab61/core-metadata/src/main/java/org/apache/kylin/dimension/IntegerDimEnc.java
--
diff --git 
a/core-metadata/src/main/java/org/apache/kylin/dimension/IntegerDimEnc.java 
b/core-metadata/src/main/java/org/apache/kylin/dimension/IntegerDimEnc.java
index 983af9a..bcd6283 100644
--- a/core-metadata/src/main/java/org/apache/kylin/dimension/IntegerDimEnc.java
+++ b/core-metadata/src/main/java/org/apache/kylin/dimension/IntegerDimEnc.java
@@ -37,7 +37,6 @@ import org.slf4j.LoggerFactory;
  * -2^(8*N-1) is not supported because the slot is reserved for null values.
  * -2^(8*N-1) will be encoded with warn, and its output will be null
  */
-@Deprecated//due to a fatal bug (KYLIN-2191)
 public class IntegerDimEnc extends DimensionEncoding {
 private static final long serialVersionUID = 1L;
 
@@ -132,7 +131,7 @@ public class IntegerDimEnc extends DimensionEncoding {
 
 //only take useful bytes
 

[2/3] kylin git commit: KYLIN-1528 Create a branch for v1.5 with HBase 1.x API

[mahongbin]

http://git-wip-us.apache.org/repos/asf/kylin/blob/e1b83470/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseStreamingOutput.java
--
diff --git 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseStreamingOutput.java
 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseStreamingOutput.java
index 0e95102..c59fb33 100644
--- 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseStreamingOutput.java
+++ 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseStreamingOutput.java
@@ -27,7 +27,8 @@ import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.fs.FSDataInputStream;
 import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.fs.Path;
-import org.apache.hadoop.hbase.client.HTableInterface;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Table;
 import org.apache.kylin.common.KylinConfig;
 import org.apache.kylin.common.persistence.ResourceStore;
 import org.apache.kylin.cube.CubeSegment;
@@ -56,7 +57,7 @@ public class HBaseStreamingOutput implements IStreamingOutput 
{
 try {
 CubeSegment cubeSegment = (CubeSegment) buildable;
 
-final HTableInterface hTable;
+final Table hTable;
 hTable = createHTable(cubeSegment);
 List cuboidWriters = Lists.newArrayList();
 cuboidWriters.add(new HBaseCuboidWriter(cubeSegment, hTable));
@@ -88,10 +89,10 @@ public class HBaseStreamingOutput implements 
IStreamingOutput {
 }
 }
 
-private HTableInterface createHTable(final CubeSegment cubeSegment) throws 
IOException {
+private Table createHTable(final CubeSegment cubeSegment) throws 
IOException {
 final String hTableName = cubeSegment.getStorageLocationIdentifier();
 CubeHTableUtil.createHTable(cubeSegment, null);
-final HTableInterface hTable = 
HBaseConnection.get(KylinConfig.getInstanceFromEnv().getStorageUrl()).getTable(hTableName);
+final Table hTable = 
HBaseConnection.get(KylinConfig.getInstanceFromEnv().getStorageUrl()).getTable(TableName.valueOf(hTableName));
 logger.info("hTable:" + hTableName + " for segment:" + 
cubeSegment.getName() + " created!");
 return hTable;
 }

http://git-wip-us.apache.org/repos/asf/kylin/blob/e1b83470/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/MergeGCStep.java
--
diff --git 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/MergeGCStep.java
 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/MergeGCStep.java
index 5b2441c..2f7e164 100644
--- 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/MergeGCStep.java
+++ 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/MergeGCStep.java
@@ -24,11 +24,11 @@ import java.util.Collections;
 import java.util.List;
 
 import org.apache.commons.lang.StringUtils;
-import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.hbase.HTableDescriptor;
-import org.apache.hadoop.hbase.client.HBaseAdmin;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.Connection;
 import org.apache.kylin.common.KylinConfig;
-import org.apache.kylin.common.util.Bytes;
 import org.apache.kylin.job.exception.ExecuteException;
 import org.apache.kylin.job.execution.AbstractExecutable;
 import org.apache.kylin.job.execution.ExecutableContext;
@@ -69,19 +69,20 @@ public class MergeGCStep extends AbstractExecutable {
 List oldTables = getOldHTables();
 if (oldTables != null && oldTables.size() > 0) {
 String metadataUrlPrefix = 
KylinConfig.getInstanceFromEnv().getMetadataUrlPrefix();
-Configuration conf = 
HBaseConnection.getCurrentHBaseConfiguration();
-HBaseAdmin admin = null;
+Admin admin = null;
 try {
-admin = new HBaseAdmin(conf);
+Connection conn = 
HBaseConnection.get(KylinConfig.getInstanceFromEnv().getStorageUrl());
+admin = conn.getAdmin();
+
 for (String table : oldTables) {
-if (admin.tableExists(table)) {
-HTableDescriptor tableDescriptor = 
admin.getTableDescriptor(Bytes.toBytes(table));
+if (admin.tableExists(TableName.valueOf(table))) {
+HTableDescriptor tableDescriptor = 
admin.getTableDescriptor(TableName.valueOf((table)));
 String host = 
tableDescriptor.getValue(IRealizationConstants.HTableTag);
 if (metadataUrlPrefix.equalsIgnoreCase(host)) {
-if (admin.isTableEnabled(table)) {
-admin.disableTable(table);
+

[3/3] kylin git commit: KYLIN-1528 Create a branch for v1.5 with HBase 1.x API

[mahongbin]

KYLIN-1528 Create a branch for v1.5 with HBase 1.x API


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/e1b83470
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/e1b83470
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/e1b83470

Branch: refs/heads/yang21-hbase1.x
Commit: e1b83470be3e8f315ad6c5001b97e62d5682dd34
Parents: 150bab6
Author: shaofengshi 
Authored: Wed Mar 23 17:07:05 2016 +0800
Committer: Hongbin Ma 
Committed: Tue Nov 22 17:13:01 2016 +0800

--
 examples/test_case_data/sandbox/hbase-site.xml  | 19 +---
 .../kylin/provision/BuildCubeWithEngine.java| 13 ++-
 pom.xml | 12 +--
 .../kylin/rest/security/AclHBaseStorage.java|  4 +-
 .../rest/security/MockAclHBaseStorage.java  |  8 +-
 .../apache/kylin/rest/security/MockHTable.java  | 95 
 .../rest/security/RealAclHBaseStorage.java  |  9 +-
 .../apache/kylin/rest/service/AclService.java   | 25 +++---
 .../apache/kylin/rest/service/CubeService.java  | 36 +++-
 .../apache/kylin/rest/service/QueryService.java | 24 +++--
 .../apache/kylin/rest/service/UserService.java  | 17 ++--
 .../kylin/storage/hbase/HBaseConnection.java| 44 -
 .../kylin/storage/hbase/HBaseResourceStore.java | 31 +++
 .../kylin/storage/hbase/HBaseStorage.java   |  3 +-
 .../storage/hbase/cube/SimpleHBaseStore.java| 20 ++---
 .../hbase/cube/v1/CubeSegmentTupleIterator.java | 11 +--
 .../storage/hbase/cube/v1/CubeStorageQuery.java |  6 +-
 .../hbase/cube/v1/RegionScannerAdapter.java | 10 ++-
 .../cube/v1/SerializedHBaseTupleIterator.java   |  4 +-
 .../observer/AggregateRegionObserver.java   |  4 +-
 .../observer/AggregationScanner.java| 14 ++-
 .../observer/ObserverAggregationCache.java  | 10 ++-
 .../coprocessor/observer/ObserverEnabler.java   |  4 +-
 .../hbase/cube/v2/CubeHBaseEndpointRPC.java | 13 +--
 .../storage/hbase/cube/v2/CubeHBaseScanRPC.java |  9 +-
 .../coprocessor/endpoint/CubeVisitService.java  |  4 +-
 .../storage/hbase/steps/CubeHTableUtil.java | 16 ++--
 .../storage/hbase/steps/DeprecatedGCStep.java   | 23 ++---
 .../storage/hbase/steps/HBaseCuboidWriter.java  |  7 +-
 .../hbase/steps/HBaseStreamingOutput.java   |  9 +-
 .../kylin/storage/hbase/steps/MergeGCStep.java  | 23 ++---
 .../storage/hbase/util/CleanHtableCLI.java  | 12 +--
 .../storage/hbase/util/CubeMigrationCLI.java| 36 
 .../hbase/util/CubeMigrationCheckCLI.java   | 17 ++--
 .../hbase/util/DeployCoprocessorCLI.java| 22 ++---
 .../hbase/util/ExtendCubeToHybridCLI.java   |  8 +-
 .../hbase/util/GridTableHBaseBenchmark.java | 34 +++
 .../kylin/storage/hbase/util/HBaseClean.java| 18 ++--
 .../hbase/util/HBaseRegionSizeCalculator.java   | 35 
 .../kylin/storage/hbase/util/HBaseUsage.java|  9 +-
 .../storage/hbase/util/HbaseStreamingInput.java | 30 +++
 .../hbase/util/HtableAlterMetadataCLI.java  |  9 +-
 .../storage/hbase/util/OrphanHBaseCleanJob.java | 19 ++--
 .../kylin/storage/hbase/util/PingHBaseCLI.java  | 15 ++--
 .../kylin/storage/hbase/util/RowCounterCLI.java | 11 +--
 .../storage/hbase/util/StorageCleanupJob.java   | 20 +++--
 .../storage/hbase/util/UpdateHTableHostCLI.java | 17 ++--
 .../observer/AggregateRegionObserverTest.java   | 26 ++
 .../v1/filter/TestFuzzyRowFilterV2EndToEnd.java |  5 +-
 .../org/apache/kylin/tool/CubeMigrationCLI.java | 36 
 .../kylin/tool/CubeMigrationCheckCLI.java   | 16 ++--
 .../kylin/tool/ExtendCubeToHybridCLI.java   |  8 +-
 .../apache/kylin/tool/StorageCleanupJob.java| 20 +++--
 53 files changed, 450 insertions(+), 500 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/e1b83470/examples/test_case_data/sandbox/hbase-site.xml
--
diff --git a/examples/test_case_data/sandbox/hbase-site.xml 
b/examples/test_case_data/sandbox/hbase-site.xml
index 46d5345..734908e 100644
--- a/examples/test_case_data/sandbox/hbase-site.xml
+++ b/examples/test_case_data/sandbox/hbase-site.xml
@@ -190,22 +190,5 @@
 zookeeper.znode.parent
 /hbase-unsecure
 
-
-hbase.client.pause
-100
-General client pause value.  Used mostly as value to wait
-before running a retry of a failed get, region lookup, etc.
-See hbase.client.retries.number for description of how we backoff 
from
-this initial pause amount and how this pause works w/ 
retries.
-
-
-hbase.client.retries.number
-5
-Maximum retries.  Used as maximum for all retryable
-operations such as the getting of a cell's value, starting a row 
update,
-   

[1/3] kylin git commit: revert KYLIN-2199 because KYLIN-2191 is not a major fix [Forced Update!]

[mahongbin]

Repository: kylin
Updated Branches:
  refs/heads/yang21-hbase1.x edb9d54e9 -> e1b83470b (forced update)


revert KYLIN-2199 because KYLIN-2191 is not a major fix


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/150bab61
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/150bab61
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/150bab61

Branch: refs/heads/yang21-hbase1.x
Commit: 150bab61a77c0900bf8d43f569b710d8b9177f0d
Parents: d4715b5
Author: Hongbin Ma 
Authored: Tue Nov 22 15:32:31 2016 +0800
Committer: Hongbin Ma 
Committed: Tue Nov 22 15:51:08 2016 +0800

--
 .../gridtable/DimEncodingPreserveOrderTest.java |   4 +-
 .../dimension/DimensionEncodingFactory.java |   4 -
 .../apache/kylin/dimension/IntegerDimEnc.java   |   3 +-
 .../apache/kylin/dimension/IntegerDimEncV2.java | 228 ---
 .../kylin/dimension/IntegerDimEncTest.java  |  16 +-
 5 files changed, 11 insertions(+), 244 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/150bab61/core-cube/src/test/java/org/apache/kylin/gridtable/DimEncodingPreserveOrderTest.java
--
diff --git 
a/core-cube/src/test/java/org/apache/kylin/gridtable/DimEncodingPreserveOrderTest.java
 
b/core-cube/src/test/java/org/apache/kylin/gridtable/DimEncodingPreserveOrderTest.java
index d572e56..87d37be 100644
--- 
a/core-cube/src/test/java/org/apache/kylin/gridtable/DimEncodingPreserveOrderTest.java
+++ 
b/core-cube/src/test/java/org/apache/kylin/gridtable/DimEncodingPreserveOrderTest.java
@@ -27,7 +27,7 @@ import org.apache.kylin.common.util.ByteArray;
 import org.apache.kylin.common.util.Bytes;
 import org.apache.kylin.dimension.DimensionEncoding;
 import org.apache.kylin.dimension.FixedLenHexDimEnc;
-import org.apache.kylin.dimension.IntegerDimEncV2;
+import org.apache.kylin.dimension.IntegerDimEnc;
 import org.apache.kylin.dimension.OneMoreByteVLongDimEnc;
 import org.junit.BeforeClass;
 import org.junit.Test;
@@ -99,7 +99,7 @@ public class DimEncodingPreserveOrderTest {
 @Test
 public void testVLongDimEncPreserveOrder() {
 for (int i = 1; i <= successValue.size(); i++) {
-IntegerDimEncV2 enc = new IntegerDimEncV2(i);
+IntegerDimEnc enc = new IntegerDimEnc(i);
 List encodedValues = Lists.newArrayList();
 for (long value : successValue.get(i - 1)) {
 encodedValues.add(encode(enc, value));

http://git-wip-us.apache.org/repos/asf/kylin/blob/150bab61/core-metadata/src/main/java/org/apache/kylin/dimension/DimensionEncodingFactory.java
--
diff --git 
a/core-metadata/src/main/java/org/apache/kylin/dimension/DimensionEncodingFactory.java
 
b/core-metadata/src/main/java/org/apache/kylin/dimension/DimensionEncodingFactory.java
index b617a19..2b6f894 100644
--- 
a/core-metadata/src/main/java/org/apache/kylin/dimension/DimensionEncodingFactory.java
+++ 
b/core-metadata/src/main/java/org/apache/kylin/dimension/DimensionEncodingFactory.java
@@ -117,10 +117,6 @@ public abstract class DimensionEncodingFactory {
 map.put(Pair.newPair(IntegerDimEnc.ENCODING_NAME, 
value.getCurrentVersion()), value);
 }
 {
-IntegerDimEncV2.Factory value = new IntegerDimEncV2.Factory();
-map.put(Pair.newPair(IntegerDimEncV2.ENCODING_NAME, 
value.getCurrentVersion()), value);
-}
-{
 FixedLenHexDimEnc.Factory value = new 
FixedLenHexDimEnc.Factory();
 map.put(Pair.newPair(FixedLenHexDimEnc.ENCODING_NAME, 
value.getCurrentVersion()), value);
 }

http://git-wip-us.apache.org/repos/asf/kylin/blob/150bab61/core-metadata/src/main/java/org/apache/kylin/dimension/IntegerDimEnc.java
--
diff --git 
a/core-metadata/src/main/java/org/apache/kylin/dimension/IntegerDimEnc.java 
b/core-metadata/src/main/java/org/apache/kylin/dimension/IntegerDimEnc.java
index 983af9a..bcd6283 100644
--- a/core-metadata/src/main/java/org/apache/kylin/dimension/IntegerDimEnc.java
+++ b/core-metadata/src/main/java/org/apache/kylin/dimension/IntegerDimEnc.java
@@ -37,7 +37,6 @@ import org.slf4j.LoggerFactory;
  * -2^(8*N-1) is not supported because the slot is reserved for null values.
  * -2^(8*N-1) will be encoded with warn, and its output will be null
  */
-@Deprecated//due to a fatal bug (KYLIN-2191)
 public class IntegerDimEnc extends DimensionEncoding {
 private static final long serialVersionUID = 1L;
 
@@ -132,7 +131,7 @@ public class IntegerDimEnc extends DimensionEncoding {
 
 //only take useful bytes

kylin git commit: encoding version show better

[mahongbin]

Repository: kylin
Updated Branches:
  refs/heads/master 1c730b91b -> a058049c8


encoding version show better

Signed-off-by: Hongbin Ma 


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/a058049c
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/a058049c
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/a058049c

Branch: refs/heads/master
Commit: a058049c8824db19c0a7bb724fc7ed3b920ff2d6
Parents: 1c730b9
Author: luguosheng <550175...@qq.com>
Authored: Tue Nov 22 16:30:22 2016 +0800
Committer: Hongbin Ma 
Committed: Tue Nov 22 16:50:34 2016 +0800

--
 webapp/app/js/controllers/cubeEdit.js | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/a058049c/webapp/app/js/controllers/cubeEdit.js
--
diff --git a/webapp/app/js/controllers/cubeEdit.js 
b/webapp/app/js/controllers/cubeEdit.js
index 9e2a3c9..10f8185 100755
--- a/webapp/app/js/controllers/cubeEdit.js
+++ b/webapp/app/js/controllers/cubeEdit.js
@@ -49,12 +49,12 @@ KylinApp.controller('CubeEditCtrl', function ($scope, $q, 
$routeParams, $locatio
   var name = value;
   var typeVersion=+encodings[i];
   if(value=="int"){
-name = "int(deprecated)";
+name = "int deprecated";
   }
   if(/\d+/.test(""+typeVersion)&>1){
   for(var s=1;s<=typeVersion;s++){
 $scope.store.supportedEncoding.push({
-  "name":name+" 
(v"+s+")"+(s==typeVersion&>1?"(suggest)":""),
+  "name":name+" 
(v"+s+","+(s==typeVersion&>1?"suggest)":")"),
   "value":value+"[v"+s+"]",
   "version":typeVersion,
   "baseValue":value,

kylin git commit: revert KYLIN-2199 because KYLIN-2191 is not a major fix

[mahongbin]

Repository: kylin
Updated Branches:
  refs/heads/master 8132cab31 -> 1c730b91b


revert KYLIN-2199 because KYLIN-2191 is not a major fix


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/1c730b91
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/1c730b91
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/1c730b91

Branch: refs/heads/master
Commit: 1c730b91b3c0847873dbbf74106647b663be1942
Parents: 8132cab
Author: Hongbin Ma 
Authored: Tue Nov 22 15:32:31 2016 +0800
Committer: Hongbin Ma 
Committed: Tue Nov 22 15:32:31 2016 +0800

--
 .../gridtable/DimEncodingPreserveOrderTest.java |   4 +-
 .../dimension/DimensionEncodingFactory.java |   4 -
 .../apache/kylin/dimension/IntegerDimEnc.java   |   3 +-
 .../apache/kylin/dimension/IntegerDimEncV2.java | 218 ---
 .../kylin/dimension/IntegerDimEncTest.java  |  16 +-
 5 files changed, 11 insertions(+), 234 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/1c730b91/core-cube/src/test/java/org/apache/kylin/gridtable/DimEncodingPreserveOrderTest.java
--
diff --git 
a/core-cube/src/test/java/org/apache/kylin/gridtable/DimEncodingPreserveOrderTest.java
 
b/core-cube/src/test/java/org/apache/kylin/gridtable/DimEncodingPreserveOrderTest.java
index 7d6af24..1866079 100644
--- 
a/core-cube/src/test/java/org/apache/kylin/gridtable/DimEncodingPreserveOrderTest.java
+++ 
b/core-cube/src/test/java/org/apache/kylin/gridtable/DimEncodingPreserveOrderTest.java
@@ -26,7 +26,7 @@ import java.util.List;
 import org.apache.kylin.common.util.ByteArray;
 import org.apache.kylin.dimension.DimensionEncoding;
 import org.apache.kylin.dimension.FixedLenHexDimEnc;
-import org.apache.kylin.dimension.IntegerDimEncV2;
+import org.apache.kylin.dimension.IntegerDimEnc;
 import org.apache.kylin.dimension.OneMoreByteVLongDimEnc;
 import org.junit.BeforeClass;
 import org.junit.Test;
@@ -98,7 +98,7 @@ public class DimEncodingPreserveOrderTest {
 @Test
 public void testVLongDimEncPreserveOrder() {
 for (int i = 1; i <= successValue.size(); i++) {
-IntegerDimEncV2 enc = new IntegerDimEncV2(i);
+IntegerDimEnc enc = new IntegerDimEnc(i);
 List encodedValues = Lists.newArrayList();
 for (long value : successValue.get(i - 1)) {
 encodedValues.add(encode(enc, value));

http://git-wip-us.apache.org/repos/asf/kylin/blob/1c730b91/core-metadata/src/main/java/org/apache/kylin/dimension/DimensionEncodingFactory.java
--
diff --git 
a/core-metadata/src/main/java/org/apache/kylin/dimension/DimensionEncodingFactory.java
 
b/core-metadata/src/main/java/org/apache/kylin/dimension/DimensionEncodingFactory.java
index 843f4cd..4954ead 100644
--- 
a/core-metadata/src/main/java/org/apache/kylin/dimension/DimensionEncodingFactory.java
+++ 
b/core-metadata/src/main/java/org/apache/kylin/dimension/DimensionEncodingFactory.java
@@ -117,10 +117,6 @@ public abstract class DimensionEncodingFactory {
 map.put(Pair.newPair(IntegerDimEnc.ENCODING_NAME, 
value.getCurrentVersion()), value);
 }
 {
-IntegerDimEncV2.Factory value = new IntegerDimEncV2.Factory();
-map.put(Pair.newPair(IntegerDimEncV2.ENCODING_NAME, 
value.getCurrentVersion()), value);
-}
-{
 FixedLenHexDimEnc.Factory value = new 
FixedLenHexDimEnc.Factory();
 map.put(Pair.newPair(FixedLenHexDimEnc.ENCODING_NAME, 
value.getCurrentVersion()), value);
 }

http://git-wip-us.apache.org/repos/asf/kylin/blob/1c730b91/core-metadata/src/main/java/org/apache/kylin/dimension/IntegerDimEnc.java
--
diff --git 
a/core-metadata/src/main/java/org/apache/kylin/dimension/IntegerDimEnc.java 
b/core-metadata/src/main/java/org/apache/kylin/dimension/IntegerDimEnc.java
index 44d0e73..090dc83 100644
--- a/core-metadata/src/main/java/org/apache/kylin/dimension/IntegerDimEnc.java
+++ b/core-metadata/src/main/java/org/apache/kylin/dimension/IntegerDimEnc.java
@@ -36,7 +36,6 @@ import org.slf4j.LoggerFactory;
  * -2^(8*N-1) is not supported because the slot is reserved for null values.
  * -2^(8*N-1) will be encoded with warn, and its output will be null
  */
-@Deprecated//due to a fatal bug (KYLIN-2191) and the limitation of not able to 
represent -2^(8N-1)
 public class IntegerDimEnc extends DimensionEncoding {
 private static final long serialVersionUID = 1L;
 
@@ -122,7 +121,7 @@ public class IntegerDimEnc extends DimensionEncoding {
 
 //only take useful bytes
   

kylin git commit: revert KYLIN-2199 because KYLIN-2191 is not a major fix

[mahongbin]

Repository: kylin
Updated Branches:
  refs/heads/yang21 d4715b559 -> 150bab61a


revert KYLIN-2199 because KYLIN-2191 is not a major fix


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/150bab61
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/150bab61
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/150bab61

Branch: refs/heads/yang21
Commit: 150bab61a77c0900bf8d43f569b710d8b9177f0d
Parents: d4715b5
Author: Hongbin Ma 
Authored: Tue Nov 22 15:32:31 2016 +0800
Committer: Hongbin Ma 
Committed: Tue Nov 22 15:51:08 2016 +0800

--
 .../gridtable/DimEncodingPreserveOrderTest.java |   4 +-
 .../dimension/DimensionEncodingFactory.java |   4 -
 .../apache/kylin/dimension/IntegerDimEnc.java   |   3 +-
 .../apache/kylin/dimension/IntegerDimEncV2.java | 228 ---
 .../kylin/dimension/IntegerDimEncTest.java  |  16 +-
 5 files changed, 11 insertions(+), 244 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/150bab61/core-cube/src/test/java/org/apache/kylin/gridtable/DimEncodingPreserveOrderTest.java
--
diff --git 
a/core-cube/src/test/java/org/apache/kylin/gridtable/DimEncodingPreserveOrderTest.java
 
b/core-cube/src/test/java/org/apache/kylin/gridtable/DimEncodingPreserveOrderTest.java
index d572e56..87d37be 100644
--- 
a/core-cube/src/test/java/org/apache/kylin/gridtable/DimEncodingPreserveOrderTest.java
+++ 
b/core-cube/src/test/java/org/apache/kylin/gridtable/DimEncodingPreserveOrderTest.java
@@ -27,7 +27,7 @@ import org.apache.kylin.common.util.ByteArray;
 import org.apache.kylin.common.util.Bytes;
 import org.apache.kylin.dimension.DimensionEncoding;
 import org.apache.kylin.dimension.FixedLenHexDimEnc;
-import org.apache.kylin.dimension.IntegerDimEncV2;
+import org.apache.kylin.dimension.IntegerDimEnc;
 import org.apache.kylin.dimension.OneMoreByteVLongDimEnc;
 import org.junit.BeforeClass;
 import org.junit.Test;
@@ -99,7 +99,7 @@ public class DimEncodingPreserveOrderTest {
 @Test
 public void testVLongDimEncPreserveOrder() {
 for (int i = 1; i <= successValue.size(); i++) {
-IntegerDimEncV2 enc = new IntegerDimEncV2(i);
+IntegerDimEnc enc = new IntegerDimEnc(i);
 List encodedValues = Lists.newArrayList();
 for (long value : successValue.get(i - 1)) {
 encodedValues.add(encode(enc, value));

http://git-wip-us.apache.org/repos/asf/kylin/blob/150bab61/core-metadata/src/main/java/org/apache/kylin/dimension/DimensionEncodingFactory.java
--
diff --git 
a/core-metadata/src/main/java/org/apache/kylin/dimension/DimensionEncodingFactory.java
 
b/core-metadata/src/main/java/org/apache/kylin/dimension/DimensionEncodingFactory.java
index b617a19..2b6f894 100644
--- 
a/core-metadata/src/main/java/org/apache/kylin/dimension/DimensionEncodingFactory.java
+++ 
b/core-metadata/src/main/java/org/apache/kylin/dimension/DimensionEncodingFactory.java
@@ -117,10 +117,6 @@ public abstract class DimensionEncodingFactory {
 map.put(Pair.newPair(IntegerDimEnc.ENCODING_NAME, 
value.getCurrentVersion()), value);
 }
 {
-IntegerDimEncV2.Factory value = new IntegerDimEncV2.Factory();
-map.put(Pair.newPair(IntegerDimEncV2.ENCODING_NAME, 
value.getCurrentVersion()), value);
-}
-{
 FixedLenHexDimEnc.Factory value = new 
FixedLenHexDimEnc.Factory();
 map.put(Pair.newPair(FixedLenHexDimEnc.ENCODING_NAME, 
value.getCurrentVersion()), value);
 }

http://git-wip-us.apache.org/repos/asf/kylin/blob/150bab61/core-metadata/src/main/java/org/apache/kylin/dimension/IntegerDimEnc.java
--
diff --git 
a/core-metadata/src/main/java/org/apache/kylin/dimension/IntegerDimEnc.java 
b/core-metadata/src/main/java/org/apache/kylin/dimension/IntegerDimEnc.java
index 983af9a..bcd6283 100644
--- a/core-metadata/src/main/java/org/apache/kylin/dimension/IntegerDimEnc.java
+++ b/core-metadata/src/main/java/org/apache/kylin/dimension/IntegerDimEnc.java
@@ -37,7 +37,6 @@ import org.slf4j.LoggerFactory;
  * -2^(8*N-1) is not supported because the slot is reserved for null values.
  * -2^(8*N-1) will be encoded with warn, and its output will be null
  */
-@Deprecated//due to a fatal bug (KYLIN-2191)
 public class IntegerDimEnc extends DimensionEncoding {
 private static final long serialVersionUID = 1L;
 
@@ -132,7 +131,7 @@ public class IntegerDimEnc extends DimensionEncoding {
 
 //only take useful bytes
 integer = integer & 

[1/2] kylin git commit: encoding at version tips

[mahongbin]

Repository: kylin
Updated Branches:
  refs/heads/master 58991331d -> 8132cab31


encoding at version tips

Signed-off-by: Hongbin Ma 


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/12221a32
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/12221a32
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/12221a32

Branch: refs/heads/master
Commit: 12221a327f7e70c1aa0401c6f9b30d761d1b58b4
Parents: 5899133
Author: luguosheng <550175...@qq.com>
Authored: Sat Nov 19 14:50:52 2016 +0800
Committer: Hongbin Ma 
Committed: Tue Nov 22 13:59:56 2016 +0800

--
 webapp/app/js/controllers/cubeAdvanceSetting.js |  76 +++--
 webapp/app/js/controllers/cubeEdit.js   |  90 +--
 webapp/app/js/controllers/cubeMeasures.js   | 109 ---
 webapp/app/js/services/cubes.js |   2 +-
 webapp/app/js/utils/utils.js|  25 +
 5 files changed, 191 insertions(+), 111 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/12221a32/webapp/app/js/controllers/cubeAdvanceSetting.js
--
diff --git a/webapp/app/js/controllers/cubeAdvanceSetting.js 
b/webapp/app/js/controllers/cubeAdvanceSetting.js
index 0ba321c..f2b36fe 100644
--- a/webapp/app/js/controllers/cubeAdvanceSetting.js
+++ b/webapp/app/js/controllers/cubeAdvanceSetting.js
@@ -19,40 +19,46 @@
 'use strict';
 
 KylinApp.controller('CubeAdvanceSettingCtrl', function ($scope, 
$modal,cubeConfig,MetaModel,cubesManager,CubeDescModel,SweetAlert) {
-$scope.cubesManager = cubesManager;
-
-
+  $scope.cubesManager = cubesManager;
+  $scope.getTypeVersion=function(typename){
+var searchResult=/\[v(\d+)\]/.exec(typename);
+if(searchResult&){
+  return searchResult.length&[1]||1;
+}else{
+  return 1;
+}
+  }
+  $scope.removeVersion=function(typename){
+if(typename){
+  return typename.replace(/\[v\d+\]/g,"");
+}
+return "";
+  }
+  var needLengthKeyList=['fixed_length','fixed_length_hex','int','integer'];
   //rowkey
   $scope.convertedRowkeys = [];
   angular.forEach($scope.cubeMetaFrame.rowkey.rowkey_columns,function(item){
 //var _isDictionaries = item.encoding === "dict"?"true":"false";
-var _isFixedLength = item.encoding.substring(0,12) === 
"fixed_length"?"true":"false";//fixed_length:12
-var _isIntegerLength = item.encoding.substring(0,7) === 
"integer"?"true":"false";
-var _isIntLength = item.encoding.substring(0,3) === "int"?"true":"false";
+//var version=$scope.getTypeVersion(encoding);
+item.encoding=$scope.removeVersion(item.encoding);
+//var _isFixedLength = item.encoding.substring(0,12) === 
"fixed_length"?"true":"false";//fixed_length:12
+//var _isIntegerLength = item.encoding.substring(0,7) === 
"integer"?"true":"false";
+//var _isIntLength = item.encoding.substring(0,3) === "int"?"true":"false";
 var _encoding = item.encoding;
 var _valueLength ;
-if(_isFixedLength !=="false"){
-  _valueLength = item.encoding.substring(13,item.encoding.length);
-  _encoding = "fixed_length";
-}
-if(_isIntLength!="false" && _isIntegerLength=="false" ){
-  _valueLength = item.encoding.substring(4,item.encoding.length);
-  _encoding = "int";
-}
-
-if(_isIntegerLength!="false" ){
-  _valueLength = item.encoding.substring(8,item.encoding.length);
-  _encoding = "integer";
+var baseKey=item.encoding.replace(/:\d+/,'');
+if(needLengthKeyList.indexOf(baseKey)>=-1){
+  var result=/:(\d+)/.exec(item.encoding);
+  _valueLength=result?result[1]:0;
 }
-
+_encoding=baseKey;
 var rowkeyObj = {
   column:item.column,
-  encoding:_encoding,
+  
encoding:_encoding+(item.encoding_version?"[v"+item.encoding_version+"]":"[v1]"),
   valueLength:_valueLength,
-  isShardBy:item.isShardBy
-
+  isShardBy:item.isShardBy,
+  encoding_version:item.encoding_version||1
 }
-
 $scope.convertedRowkeys.push(rowkeyObj);
 
   })
@@ -61,28 +67,26 @@ KylinApp.controller('CubeAdvanceSettingCtrl', function 
($scope, $modal,cubeConfi
   $scope.rule={
 shardColumnAvailable:true
   }
-
+  var 
checkedlen=$scope.cubeMetaFrame.rowkey.rowkey_columns&&$scope.cubeMetaFrame.rowkey.rowkey_columns.length||0;
+  for(var i=0;i

[2/2] kylin git commit: encoding add filter by column type

[mahongbin]

encoding add filter by column type

Signed-off-by: Hongbin Ma 


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/8132cab3
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/8132cab3
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/8132cab3

Branch: refs/heads/master
Commit: 8132cab31fc084cf6b83ab29884e6821d86bf249
Parents: 12221a3
Author: luguosheng <550175...@qq.com>
Authored: Sat Nov 19 16:49:41 2016 +0800
Committer: Hongbin Ma 
Committed: Tue Nov 22 14:00:12 2016 +0800

--
 webapp/app/js/controllers/cubeEdit.js | 51 +++---
 webapp/app/js/controllers/cubeMeasures.js | 33 +
 webapp/app/js/utils/utils.js  | 21 ---
 3 files changed, 70 insertions(+), 35 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/8132cab3/webapp/app/js/controllers/cubeEdit.js
--
diff --git a/webapp/app/js/controllers/cubeEdit.js 
b/webapp/app/js/controllers/cubeEdit.js
index a663d6f..9e2a3c9 100755
--- a/webapp/app/js/controllers/cubeEdit.js
+++ b/webapp/app/js/controllers/cubeEdit.js
@@ -77,33 +77,40 @@ KylinApp.controller('CubeEditCtrl', function ($scope, $q, 
$routeParams, $locatio
   },function(e){
 $scope.store.supportedEncoding = $scope.cubeConfig.encodings;
   })
-
+  $scope.createFilter=function(type){
+ if(type.indexOf("varchar")<=0){
+   return ['fixed_length_hex'];
+ }else if(type!="date"){
+   return ['date'];
+ }else if(type!="time"&!="datetime"&!="timestamp"){
+   return ['time'];
+ }else{
+   return [];
+ }
+  }
   $scope.getEncodings =function (name){
-var type = TableModel.columnNameTypeMap[name]||'';
-var encodings =[];
-for(var k=0;k<$scope.store.supportedEncoding.length;k++){
-  encodings.push($scope.store.supportedEncoding[k]);
-}
+var filterName=name;
+var type = TableModel.columnNameTypeMap[filterName]||'';
+var encodings =$scope.store.supportedEncoding,filterEncoding;
+var filerList=$scope.createFilter(type);
 if($scope.isEdit){
-  for(var s=0;s<$scope.cubeMetaFrame.rowkey.rowkey_columns.length;s++){
-if(name==$scope.cubeMetaFrame.rowkey.rowkey_columns[s].column){
-  var 
version=$scope.cubeMetaFrame.rowkey.rowkey_columns[s].encoding_version;
-  
encodings=VdmUtil.changeObjectListValueByFilter(encodings,'value',$scope.cubeMetaFrame.rowkey.rowkey_columns[s].encoding.replace(/:\d+/,"")+(version?"[v"+version+"]":"[v1]"),'suggest',true)
+  if($scope.cubeMetaFrame.rowkey.rowkey_columns&){
+for(var s=0;s<$scope.cubeMetaFrame.rowkey.rowkey_columns.length;s++){
+  if(filterName==$scope.cubeMetaFrame.rowkey.rowkey_columns[s].column){
+var 
version=$scope.cubeMetaFrame.rowkey.rowkey_columns[s].encoding_version;
+
filterEncoding=VdmUtil.getFilterObjectListByOrFilterVal(encodings,'value',$scope.cubeMetaFrame.rowkey.rowkey_columns[s].encoding.replace(/:\d+/,"")+(version?"[v"+version+"]":"[v1]"),'suggest',true)
+  }
 }
+  }else{
+
filterEncoding=VdmUtil.getFilterObjectListByOrFilterVal(encodings,'suggest',true);
   }
+}else{
+  
filterEncoding=VdmUtil.getFilterObjectListByOrFilterVal(encodings,'suggest',true);
 }
-//if(type.indexOf("varchar")!==-1){
-//
filterEncodings=VdmUtil.getObjectListByFilterVal(encodings,'baseValue',['fixed_length_hex','fixed_length'],'suggest',true);
-//}
-//else if(type=="date"){
-//filterEncodings = VdmUtil.getObjectListByFilterVal(encodings, 
'baseValue', 'date','suggest',true);
-//}else if(type=="time"||type=="datetime"||type=="timestamp"){
-//filterEncodings = VdmUtil.getObjectListByFilterVal(encodings, 
'baseValue', 'time','suggest',true);
-//}else{
-encodings=VdmUtil.getObjectListByFilterVal(encodings, '', 
'','suggest',true);
-//}
-
-return encodings;
+for(var f=0;f

kylin git commit: KYLIN-2221 rethink on KYLIN-1684

[mahongbin]

Repository: kylin
Updated Branches:
  refs/heads/master 51830456f -> 58991331d


KYLIN-2221 rethink on KYLIN-1684


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/58991331
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/58991331
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/58991331

Branch: refs/heads/master
Commit: 58991331d276e127695893c2041bce37934c836f
Parents: 5183045
Author: Hongbin Ma 
Authored: Tue Nov 22 13:42:46 2016 +0800
Committer: Hongbin Ma 
Committed: Tue Nov 22 13:42:57 2016 +0800

--
 .../apache/kylin/common/KylinConfigBase.java| 22 +---
 .../gtrecord/GTCubeStorageQueryBase.java| 15 ++---
 2 files changed, 20 insertions(+), 17 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/58991331/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
--
diff --git 
a/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java 
b/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
index c7dd8a8..3252245 100644
--- a/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
+++ b/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
@@ -18,13 +18,6 @@
 
 package org.apache.kylin.common;
 
-import com.google.common.collect.Maps;
-import com.google.common.collect.Sets;
-import org.apache.commons.lang.StringUtils;
-import org.apache.kylin.common.util.CliCommandExecutor;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
 import java.io.File;
 import java.io.IOException;
 import java.io.Serializable;
@@ -35,6 +28,14 @@ import java.util.SortedSet;
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
 
+import org.apache.commons.lang.StringUtils;
+import org.apache.kylin.common.util.CliCommandExecutor;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Maps;
+import com.google.common.collect.Sets;
+
 /**
  * An abstract class to encapsulate access to a set of 'properties'.
  * Subclass can override methods in this class to extend the content of the 
'properties',
@@ -210,7 +211,7 @@ abstract public class KylinConfigBase implements 
Serializable {
 
 public String[] getRealizationProviders() {
 return getOptionalStringArray("kylin.realization.providers", //
-new String[] {"org.apache.kylin.cube.CubeManager", 
"org.apache.kylin.storage.hybrid.HybridManager"});
+new String[] { "org.apache.kylin.cube.CubeManager", 
"org.apache.kylin.storage.hybrid.HybridManager" });
 }
 
 public CliCommandExecutor getCliCommandExecutor() throws IOException {
@@ -490,6 +491,11 @@ abstract public class KylinConfigBase implements 
Serializable {
 return Float.parseFloat(getOptional("kylin.hbase.hfile.size.gb", 
"2.0"));
 }
 
+//check KYLIN-1684, in most cases keep the default value
+public boolean isSkippingEmptySegments() {
+return Boolean.valueOf(getOptional("kylin.query.skip-empty-segments", 
"true"));
+}
+
 public int getStoragePushDownLimitMax() {
 return Integer.parseInt(getOptional("kylin.query.pushdown.limit.max", 
"1"));
 }

http://git-wip-us.apache.org/repos/asf/kylin/blob/58991331/core-storage/src/main/java/org/apache/kylin/storage/gtrecord/GTCubeStorageQueryBase.java
--
diff --git 
a/core-storage/src/main/java/org/apache/kylin/storage/gtrecord/GTCubeStorageQueryBase.java
 
b/core-storage/src/main/java/org/apache/kylin/storage/gtrecord/GTCubeStorageQueryBase.java
index e715120..24dd2fb 100644
--- 
a/core-storage/src/main/java/org/apache/kylin/storage/gtrecord/GTCubeStorageQueryBase.java
+++ 
b/core-storage/src/main/java/org/apache/kylin/storage/gtrecord/GTCubeStorageQueryBase.java
@@ -125,16 +125,13 @@ public abstract class GTCubeStorageQueryBase implements 
IStorageQuery {
 List scanners = Lists.newArrayList();
 for (CubeSegment cubeSeg : 
cubeInstance.getSegments(SegmentStatusEnum.READY)) {
 CubeSegmentScanner scanner;
-try {
-scanner = new CubeSegmentScanner(cubeSeg, cuboid, dimensionsD, 
groupsD, metrics, filterD, context, getGTStorage());
-} catch (IllegalArgumentException ex) {
-// ref KYLIN-1967, real empty segment can trigger dictionary 
exception -- IllegalArgumentException: Value not exists!
-if (cubeSeg.getInputRecords() == 0) {
-logger.warn("cube segment {} input record is 0, skip it 
still", cubeSeg);
-continue;
-}
-throw ex;
+
+if 

[3/6] kylin git commit: add utilsjs link

[mahongbin]

add utilsjs link

Signed-off-by: Hongbin Ma 


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/d4715b55
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/d4715b55
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/d4715b55

Branch: refs/heads/yang21-cdh5.7
Commit: d4715b559dda73457610f41374fc0042ffdbddf2
Parents: 258353f
Author: luguosheng <550175...@qq.com>
Authored: Sat Nov 19 19:44:53 2016 +0800
Committer: Hongbin Ma 
Committed: Tue Nov 22 09:39:42 2016 +0800

--
 webapp/app/index.html | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/d4715b55/webapp/app/index.html
--
diff --git a/webapp/app/index.html b/webapp/app/index.html
index 8fb2cb9..ad881d5 100644
--- a/webapp/app/index.html
+++ b/webapp/app/index.html
@@ -163,7 +163,7 @@
 
 
 
-
+
 
 
 

[6/6] kylin git commit: KYLIN-1672 support kylin on cdh 5.7

[mahongbin]

KYLIN-1672 support kylin on cdh 5.7

Signed-off-by: Li Yang 


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/229cffdb
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/229cffdb
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/229cffdb

Branch: refs/heads/yang21-cdh5.7
Commit: 229cffdbd5586fba349e370b0e1195345e43703d
Parents: edb9d54
Author: Lynne Jiang 
Authored: Mon May 16 03:33:27 2016 -0700
Committer: Hongbin Ma 
Committed: Tue Nov 22 12:53:42 2016 +0800

--
 build/conf/kylin.properties |   3 +
 dev-support/test_all_against_hdp_2_2_4_2_2.sh   |   0
 .../kylin/engine/mr/steps/MockupMapContext.java |  15 +-
 examples/test_case_data/sandbox/core-site.xml   | 146 +++---
 examples/test_case_data/sandbox/hbase-site.xml  | 162 ++
 examples/test_case_data/sandbox/hdfs-site.xml   | 259 ++
 examples/test_case_data/sandbox/mapred-site.xml | 398 ++-
 examples/test_case_data/sandbox/yarn-site.xml   | 496 ++-
 pom.xml |  16 +-
 server/pom.xml  |  36 ++
 .../storage/hbase/steps/MockupMapContext.java   |  19 +-
 tool/pom.xml|  12 +
 12 files changed, 431 insertions(+), 1131 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/229cffdb/build/conf/kylin.properties
--
diff --git a/build/conf/kylin.properties b/build/conf/kylin.properties
index ed86bdb..ebd8cde 100644
--- a/build/conf/kylin.properties
+++ b/build/conf/kylin.properties
@@ -118,6 +118,9 @@ kylin.job.mapreduce.mapper.input.rows=100
 
 kylin.job.step.timeout=7200
 
+# for secure cdh, filtering hive dependency is risky, so filter nothing
+kylin.job.dependency.filterlist=[^,]+
+
 ### CUBE ###
 
 # 'auto', 'inmem', 'layer' or 'random' for testing

http://git-wip-us.apache.org/repos/asf/kylin/blob/229cffdb/dev-support/test_all_against_hdp_2_2_4_2_2.sh
--
diff --git a/dev-support/test_all_against_hdp_2_2_4_2_2.sh 
b/dev-support/test_all_against_hdp_2_2_4_2_2.sh
old mode 100644
new mode 100755

http://git-wip-us.apache.org/repos/asf/kylin/blob/229cffdb/engine-mr/src/test/java/org/apache/kylin/engine/mr/steps/MockupMapContext.java
--
diff --git 
a/engine-mr/src/test/java/org/apache/kylin/engine/mr/steps/MockupMapContext.java
 
b/engine-mr/src/test/java/org/apache/kylin/engine/mr/steps/MockupMapContext.java
index 847071d..9900465 100644
--- 
a/engine-mr/src/test/java/org/apache/kylin/engine/mr/steps/MockupMapContext.java
+++ 
b/engine-mr/src/test/java/org/apache/kylin/engine/mr/steps/MockupMapContext.java
@@ -77,6 +77,7 @@ public class MockupMapContext {
 outKV[0] = key;
 outKV[1] = value;
 }
+
 }
 
 @Override
@@ -99,6 +100,7 @@ public class MockupMapContext {
 throw new NotImplementedException();
 }
 
+
 @Override
 public float getProgress() {
 throw new NotImplementedException();
@@ -195,17 +197,17 @@ public class MockupMapContext {
 }
 
 @Override
-public RawComparator getSortComparator() {
+public boolean userClassesTakesPrecedence() {
 throw new NotImplementedException();
 }
 
 @Override
-public String getJar() {
+public RawComparator getSortComparator() {
 throw new NotImplementedException();
 }
 
 @Override
-public RawComparator getGroupingComparator() {
+public String getJar() {
 throw new NotImplementedException();
 }
 
@@ -221,7 +223,7 @@ public class MockupMapContext {
 
 @Override
 public boolean getProfileEnabled() {
-throw new NotImplementedException();
+return false;
 }
 
 @Override
@@ -308,6 +310,11 @@ public class MockupMapContext {
 public RawComparator getCombinerKeyGroupingComparator() {
 throw new NotImplementedException();
 }
+
+@Override
+public RawComparator getGroupingComparator() {
+return null;
+}
 });
 }
 }

http://git-wip-us.apache.org/repos/asf/kylin/blob/229cffdb/examples/test_case_data/sandbox/core-site.xml
--
diff --git a/examples/test_case_data/sandbox/core-site.xml 

[2/6] kylin git commit: encoding add filter by column type

[mahongbin]

encoding add filter by column type

Signed-off-by: Hongbin Ma 


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/258353f5
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/258353f5
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/258353f5

Branch: refs/heads/yang21-cdh5.7
Commit: 258353f5718ce007313fd723b3ba62cb82c1d9f4
Parents: 577e6be
Author: luguosheng <550175...@qq.com>
Authored: Sat Nov 19 16:49:41 2016 +0800
Committer: Hongbin Ma 
Committed: Tue Nov 22 09:39:26 2016 +0800

--
 webapp/app/js/controllers/cubeEdit.js | 51 +++---
 webapp/app/js/controllers/cubeMeasures.js | 33 +
 webapp/app/js/utils/utils.js  | 21 ---
 3 files changed, 70 insertions(+), 35 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/258353f5/webapp/app/js/controllers/cubeEdit.js
--
diff --git a/webapp/app/js/controllers/cubeEdit.js 
b/webapp/app/js/controllers/cubeEdit.js
index 5c5d9c7..3282974 100755
--- a/webapp/app/js/controllers/cubeEdit.js
+++ b/webapp/app/js/controllers/cubeEdit.js
@@ -77,33 +77,40 @@ KylinApp.controller('CubeEditCtrl', function ($scope, $q, 
$routeParams, $locatio
   },function(e){
 $scope.store.supportedEncoding = $scope.cubeConfig.encodings;
   })
-
+  $scope.createFilter=function(type){
+ if(type.indexOf("varchar")<=0){
+   return ['fixed_length_hex'];
+ }else if(type!="date"){
+   return ['date'];
+ }else if(type!="time"&!="datetime"&!="timestamp"){
+   return ['time'];
+ }else{
+   return [];
+ }
+  }
   $scope.getEncodings =function (name){
-var type = TableModel.columnNameTypeMap[name]||'';
-var encodings =[];
-for(var k=0;k<$scope.store.supportedEncoding.length;k++){
-  encodings.push($scope.store.supportedEncoding[k]);
-}
+var filterName=name;
+var type = TableModel.columnNameTypeMap[filterName]||'';
+var encodings =$scope.store.supportedEncoding,filterEncoding;
+var filerList=$scope.createFilter(type);
 if($scope.isEdit){
-  for(var s=0;s<$scope.cubeMetaFrame.rowkey.rowkey_columns.length;s++){
-if(name==$scope.cubeMetaFrame.rowkey.rowkey_columns[s].column){
-  var 
version=$scope.cubeMetaFrame.rowkey.rowkey_columns[s].encoding_version;
-  
encodings=VdmUtil.changeObjectListValueByFilter(encodings,'value',$scope.cubeMetaFrame.rowkey.rowkey_columns[s].encoding.replace(/:\d+/,"")+(version?"[v"+version+"]":"[v1]"),'suggest',true)
+  if($scope.cubeMetaFrame.rowkey.rowkey_columns&){
+for(var s=0;s<$scope.cubeMetaFrame.rowkey.rowkey_columns.length;s++){
+  if(filterName==$scope.cubeMetaFrame.rowkey.rowkey_columns[s].column){
+var 
version=$scope.cubeMetaFrame.rowkey.rowkey_columns[s].encoding_version;
+
filterEncoding=VdmUtil.getFilterObjectListByOrFilterVal(encodings,'value',$scope.cubeMetaFrame.rowkey.rowkey_columns[s].encoding.replace(/:\d+/,"")+(version?"[v"+version+"]":"[v1]"),'suggest',true)
+  }
 }
+  }else{
+
filterEncoding=VdmUtil.getFilterObjectListByOrFilterVal(encodings,'suggest',true);
   }
+}else{
+  
filterEncoding=VdmUtil.getFilterObjectListByOrFilterVal(encodings,'suggest',true);
+}
+for(var f=0;f

[5/6] kylin git commit: KYLIN-1528 Create a branch for v1.5 with HBase 1.x API

[mahongbin]

KYLIN-1528 Create a branch for v1.5 with HBase 1.x API


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/edb9d54e
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/edb9d54e
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/edb9d54e

Branch: refs/heads/yang21-cdh5.7
Commit: edb9d54e90639e507ddba33214747f3eb445677a
Parents: d4715b5
Author: shaofengshi 
Authored: Wed Mar 23 17:07:05 2016 +0800
Committer: Hongbin Ma 
Committed: Tue Nov 22 12:52:10 2016 +0800

--
 examples/test_case_data/sandbox/hbase-site.xml  | 19 +---
 .../kylin/provision/BuildCubeWithEngine.java| 13 ++-
 pom.xml | 12 +--
 .../kylin/rest/security/AclHBaseStorage.java|  4 +-
 .../rest/security/MockAclHBaseStorage.java  |  8 +-
 .../apache/kylin/rest/security/MockHTable.java  | 95 
 .../rest/security/RealAclHBaseStorage.java  |  9 +-
 .../apache/kylin/rest/service/AclService.java   | 25 +++---
 .../apache/kylin/rest/service/CubeService.java  | 36 +++-
 .../apache/kylin/rest/service/QueryService.java | 24 +++--
 .../apache/kylin/rest/service/UserService.java  | 17 ++--
 .../kylin/storage/hbase/HBaseConnection.java| 44 -
 .../kylin/storage/hbase/HBaseResourceStore.java | 31 +++
 .../kylin/storage/hbase/HBaseStorage.java   |  3 +-
 .../storage/hbase/cube/SimpleHBaseStore.java| 20 ++---
 .../hbase/cube/v1/CubeSegmentTupleIterator.java | 11 +--
 .../storage/hbase/cube/v1/CubeStorageQuery.java |  6 +-
 .../hbase/cube/v1/RegionScannerAdapter.java | 10 ++-
 .../cube/v1/SerializedHBaseTupleIterator.java   |  4 +-
 .../observer/AggregateRegionObserver.java   |  4 +-
 .../observer/AggregationScanner.java| 14 ++-
 .../observer/ObserverAggregationCache.java  | 10 ++-
 .../coprocessor/observer/ObserverEnabler.java   |  4 +-
 .../hbase/cube/v2/CubeHBaseEndpointRPC.java | 13 +--
 .../storage/hbase/cube/v2/CubeHBaseScanRPC.java |  9 +-
 .../coprocessor/endpoint/CubeVisitService.java  |  4 +-
 .../storage/hbase/steps/CubeHTableUtil.java | 16 ++--
 .../storage/hbase/steps/DeprecatedGCStep.java   | 23 ++---
 .../storage/hbase/steps/HBaseCuboidWriter.java  |  7 +-
 .../hbase/steps/HBaseStreamingOutput.java   |  9 +-
 .../kylin/storage/hbase/steps/MergeGCStep.java  | 23 ++---
 .../storage/hbase/util/CleanHtableCLI.java  | 12 +--
 .../storage/hbase/util/CubeMigrationCLI.java| 36 
 .../hbase/util/CubeMigrationCheckCLI.java   | 17 ++--
 .../hbase/util/DeployCoprocessorCLI.java| 22 ++---
 .../hbase/util/ExtendCubeToHybridCLI.java   |  8 +-
 .../hbase/util/GridTableHBaseBenchmark.java | 34 +++
 .../kylin/storage/hbase/util/HBaseClean.java| 18 ++--
 .../hbase/util/HBaseRegionSizeCalculator.java   | 35 
 .../kylin/storage/hbase/util/HBaseUsage.java|  9 +-
 .../storage/hbase/util/HbaseStreamingInput.java | 30 +++
 .../hbase/util/HtableAlterMetadataCLI.java  |  9 +-
 .../storage/hbase/util/OrphanHBaseCleanJob.java | 19 ++--
 .../kylin/storage/hbase/util/PingHBaseCLI.java  | 15 ++--
 .../kylin/storage/hbase/util/RowCounterCLI.java | 11 +--
 .../storage/hbase/util/StorageCleanupJob.java   | 20 +++--
 .../storage/hbase/util/UpdateHTableHostCLI.java | 17 ++--
 .../observer/AggregateRegionObserverTest.java   | 26 ++
 .../v1/filter/TestFuzzyRowFilterV2EndToEnd.java |  5 +-
 .../org/apache/kylin/tool/CubeMigrationCLI.java | 36 
 .../kylin/tool/CubeMigrationCheckCLI.java   | 16 ++--
 .../kylin/tool/ExtendCubeToHybridCLI.java   |  8 +-
 .../apache/kylin/tool/StorageCleanupJob.java| 20 +++--
 53 files changed, 450 insertions(+), 500 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/edb9d54e/examples/test_case_data/sandbox/hbase-site.xml
--
diff --git a/examples/test_case_data/sandbox/hbase-site.xml 
b/examples/test_case_data/sandbox/hbase-site.xml
index 46d5345..734908e 100644
--- a/examples/test_case_data/sandbox/hbase-site.xml
+++ b/examples/test_case_data/sandbox/hbase-site.xml
@@ -190,22 +190,5 @@
 zookeeper.znode.parent
 /hbase-unsecure
 
-
-hbase.client.pause
-100
-General client pause value.  Used mostly as value to wait
-before running a retry of a failed get, region lookup, etc.
-See hbase.client.retries.number for description of how we backoff 
from
-this initial pause amount and how this pause works w/ 
retries.
-
-
-hbase.client.retries.number
-5
-Maximum retries.  Used as maximum for all retryable
-operations such as the getting of a cell's value, starting a row 
update,
- 

[1/6] kylin git commit: encoding at version tips [Forced Update!]

[mahongbin]

Repository: kylin
Updated Branches:
  refs/heads/yang21-cdh5.7 2e166ad79 -> 229cffdbd (forced update)


encoding at version tips

Signed-off-by: Hongbin Ma 


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/577e6be2
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/577e6be2
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/577e6be2

Branch: refs/heads/yang21-cdh5.7
Commit: 577e6be2d08faf821061d037df8f1576ec2c5d57
Parents: 61dd19f
Author: luguosheng <550175...@qq.com>
Authored: Sat Nov 19 14:50:52 2016 +0800
Committer: Hongbin Ma 
Committed: Tue Nov 22 09:38:59 2016 +0800

--
 webapp/app/js/controllers/cubeAdvanceSetting.js |  68 -
 webapp/app/js/controllers/cubeEdit.js   |  90 ++---
 webapp/app/js/controllers/cubeMeasures.js   | 101 +--
 webapp/app/js/services/cubes.js |   2 +-
 webapp/app/js/utils/utils.js|  45 +
 5 files changed, 211 insertions(+), 95 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/577e6be2/webapp/app/js/controllers/cubeAdvanceSetting.js
--
diff --git a/webapp/app/js/controllers/cubeAdvanceSetting.js 
b/webapp/app/js/controllers/cubeAdvanceSetting.js
index be90f65..f2b36fe 100644
--- a/webapp/app/js/controllers/cubeAdvanceSetting.js
+++ b/webapp/app/js/controllers/cubeAdvanceSetting.js
@@ -19,34 +19,46 @@
 'use strict';
 
 KylinApp.controller('CubeAdvanceSettingCtrl', function ($scope, 
$modal,cubeConfig,MetaModel,cubesManager,CubeDescModel,SweetAlert) {
-$scope.cubesManager = cubesManager;
-
-
+  $scope.cubesManager = cubesManager;
+  $scope.getTypeVersion=function(typename){
+var searchResult=/\[v(\d+)\]/.exec(typename);
+if(searchResult&){
+  return searchResult.length&[1]||1;
+}else{
+  return 1;
+}
+  }
+  $scope.removeVersion=function(typename){
+if(typename){
+  return typename.replace(/\[v\d+\]/g,"");
+}
+return "";
+  }
+  var needLengthKeyList=['fixed_length','fixed_length_hex','int','integer'];
   //rowkey
   $scope.convertedRowkeys = [];
   angular.forEach($scope.cubeMetaFrame.rowkey.rowkey_columns,function(item){
 //var _isDictionaries = item.encoding === "dict"?"true":"false";
-var _isFixedLength = item.encoding.substring(0,12) === 
"fixed_length"?"true":"false";//fixed_length:12
-var _isIntLength = item.encoding.substring(0,3) === 
"int"?"true":"false";//fixed_length:12
+//var version=$scope.getTypeVersion(encoding);
+item.encoding=$scope.removeVersion(item.encoding);
+//var _isFixedLength = item.encoding.substring(0,12) === 
"fixed_length"?"true":"false";//fixed_length:12
+//var _isIntegerLength = item.encoding.substring(0,7) === 
"integer"?"true":"false";
+//var _isIntLength = item.encoding.substring(0,3) === "int"?"true":"false";
 var _encoding = item.encoding;
 var _valueLength ;
-if(_isFixedLength !=="false"){
-  _valueLength = item.encoding.substring(13,item.encoding.length);
-  _encoding = "fixed_length";
-}
-if(_isIntLength!="false"){
-  _valueLength = item.encoding.substring(4,item.encoding.length);
-  _encoding = "int";
+var baseKey=item.encoding.replace(/:\d+/,'');
+if(needLengthKeyList.indexOf(baseKey)>=-1){
+  var result=/:(\d+)/.exec(item.encoding);
+  _valueLength=result?result[1]:0;
 }
-
+_encoding=baseKey;
 var rowkeyObj = {
   column:item.column,
-  encoding:_encoding,
+  
encoding:_encoding+(item.encoding_version?"[v"+item.encoding_version+"]":"[v1]"),
   valueLength:_valueLength,
-  isShardBy:item.isShardBy
-
+  isShardBy:item.isShardBy,
+  encoding_version:item.encoding_version||1
 }
-
 $scope.convertedRowkeys.push(rowkeyObj);
 
   })
@@ -55,26 +67,26 @@ KylinApp.controller('CubeAdvanceSettingCtrl', function 
($scope, $modal,cubeConfi
   $scope.rule={
 shardColumnAvailable:true
   }
-
+  var 
checkedlen=$scope.cubeMetaFrame.rowkey.rowkey_columns&&$scope.cubeMetaFrame.rowkey.rowkey_columns.length||0;
+  for(var i=0;i

[4/6] kylin git commit: KYLIN-1528 Create a branch for v1.5 with HBase 1.x API

[mahongbin]

http://git-wip-us.apache.org/repos/asf/kylin/blob/edb9d54e/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseStreamingOutput.java
--
diff --git 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseStreamingOutput.java
 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseStreamingOutput.java
index 0e95102..c59fb33 100644
--- 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseStreamingOutput.java
+++ 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseStreamingOutput.java
@@ -27,7 +27,8 @@ import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.fs.FSDataInputStream;
 import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.fs.Path;
-import org.apache.hadoop.hbase.client.HTableInterface;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Table;
 import org.apache.kylin.common.KylinConfig;
 import org.apache.kylin.common.persistence.ResourceStore;
 import org.apache.kylin.cube.CubeSegment;
@@ -56,7 +57,7 @@ public class HBaseStreamingOutput implements IStreamingOutput 
{
 try {
 CubeSegment cubeSegment = (CubeSegment) buildable;
 
-final HTableInterface hTable;
+final Table hTable;
 hTable = createHTable(cubeSegment);
 List cuboidWriters = Lists.newArrayList();
 cuboidWriters.add(new HBaseCuboidWriter(cubeSegment, hTable));
@@ -88,10 +89,10 @@ public class HBaseStreamingOutput implements 
IStreamingOutput {
 }
 }
 
-private HTableInterface createHTable(final CubeSegment cubeSegment) throws 
IOException {
+private Table createHTable(final CubeSegment cubeSegment) throws 
IOException {
 final String hTableName = cubeSegment.getStorageLocationIdentifier();
 CubeHTableUtil.createHTable(cubeSegment, null);
-final HTableInterface hTable = 
HBaseConnection.get(KylinConfig.getInstanceFromEnv().getStorageUrl()).getTable(hTableName);
+final Table hTable = 
HBaseConnection.get(KylinConfig.getInstanceFromEnv().getStorageUrl()).getTable(TableName.valueOf(hTableName));
 logger.info("hTable:" + hTableName + " for segment:" + 
cubeSegment.getName() + " created!");
 return hTable;
 }

http://git-wip-us.apache.org/repos/asf/kylin/blob/edb9d54e/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/MergeGCStep.java
--
diff --git 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/MergeGCStep.java
 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/MergeGCStep.java
index 5b2441c..2f7e164 100644
--- 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/MergeGCStep.java
+++ 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/MergeGCStep.java
@@ -24,11 +24,11 @@ import java.util.Collections;
 import java.util.List;
 
 import org.apache.commons.lang.StringUtils;
-import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.hbase.HTableDescriptor;
-import org.apache.hadoop.hbase.client.HBaseAdmin;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.Connection;
 import org.apache.kylin.common.KylinConfig;
-import org.apache.kylin.common.util.Bytes;
 import org.apache.kylin.job.exception.ExecuteException;
 import org.apache.kylin.job.execution.AbstractExecutable;
 import org.apache.kylin.job.execution.ExecutableContext;
@@ -69,19 +69,20 @@ public class MergeGCStep extends AbstractExecutable {
 List oldTables = getOldHTables();
 if (oldTables != null && oldTables.size() > 0) {
 String metadataUrlPrefix = 
KylinConfig.getInstanceFromEnv().getMetadataUrlPrefix();
-Configuration conf = 
HBaseConnection.getCurrentHBaseConfiguration();
-HBaseAdmin admin = null;
+Admin admin = null;
 try {
-admin = new HBaseAdmin(conf);
+Connection conn = 
HBaseConnection.get(KylinConfig.getInstanceFromEnv().getStorageUrl());
+admin = conn.getAdmin();
+
 for (String table : oldTables) {
-if (admin.tableExists(table)) {
-HTableDescriptor tableDescriptor = 
admin.getTableDescriptor(Bytes.toBytes(table));
+if (admin.tableExists(TableName.valueOf(table))) {
+HTableDescriptor tableDescriptor = 
admin.getTableDescriptor(TableName.valueOf((table)));
 String host = 
tableDescriptor.getValue(IRealizationConstants.HTableTag);
 if (metadataUrlPrefix.equalsIgnoreCase(host)) {
-if (admin.isTableEnabled(table)) {
-admin.disableTable(table);
+

[5/5] kylin git commit: KYLIN-1528 Create a branch for v1.5 with HBase 1.x API

[mahongbin]

KYLIN-1528 Create a branch for v1.5 with HBase 1.x API


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/edb9d54e
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/edb9d54e
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/edb9d54e

Branch: refs/heads/yang21-hbase1.x
Commit: edb9d54e90639e507ddba33214747f3eb445677a
Parents: d4715b5
Author: shaofengshi 
Authored: Wed Mar 23 17:07:05 2016 +0800
Committer: Hongbin Ma 
Committed: Tue Nov 22 12:52:10 2016 +0800

--
 examples/test_case_data/sandbox/hbase-site.xml  | 19 +---
 .../kylin/provision/BuildCubeWithEngine.java| 13 ++-
 pom.xml | 12 +--
 .../kylin/rest/security/AclHBaseStorage.java|  4 +-
 .../rest/security/MockAclHBaseStorage.java  |  8 +-
 .../apache/kylin/rest/security/MockHTable.java  | 95 
 .../rest/security/RealAclHBaseStorage.java  |  9 +-
 .../apache/kylin/rest/service/AclService.java   | 25 +++---
 .../apache/kylin/rest/service/CubeService.java  | 36 +++-
 .../apache/kylin/rest/service/QueryService.java | 24 +++--
 .../apache/kylin/rest/service/UserService.java  | 17 ++--
 .../kylin/storage/hbase/HBaseConnection.java| 44 -
 .../kylin/storage/hbase/HBaseResourceStore.java | 31 +++
 .../kylin/storage/hbase/HBaseStorage.java   |  3 +-
 .../storage/hbase/cube/SimpleHBaseStore.java| 20 ++---
 .../hbase/cube/v1/CubeSegmentTupleIterator.java | 11 +--
 .../storage/hbase/cube/v1/CubeStorageQuery.java |  6 +-
 .../hbase/cube/v1/RegionScannerAdapter.java | 10 ++-
 .../cube/v1/SerializedHBaseTupleIterator.java   |  4 +-
 .../observer/AggregateRegionObserver.java   |  4 +-
 .../observer/AggregationScanner.java| 14 ++-
 .../observer/ObserverAggregationCache.java  | 10 ++-
 .../coprocessor/observer/ObserverEnabler.java   |  4 +-
 .../hbase/cube/v2/CubeHBaseEndpointRPC.java | 13 +--
 .../storage/hbase/cube/v2/CubeHBaseScanRPC.java |  9 +-
 .../coprocessor/endpoint/CubeVisitService.java  |  4 +-
 .../storage/hbase/steps/CubeHTableUtil.java | 16 ++--
 .../storage/hbase/steps/DeprecatedGCStep.java   | 23 ++---
 .../storage/hbase/steps/HBaseCuboidWriter.java  |  7 +-
 .../hbase/steps/HBaseStreamingOutput.java   |  9 +-
 .../kylin/storage/hbase/steps/MergeGCStep.java  | 23 ++---
 .../storage/hbase/util/CleanHtableCLI.java  | 12 +--
 .../storage/hbase/util/CubeMigrationCLI.java| 36 
 .../hbase/util/CubeMigrationCheckCLI.java   | 17 ++--
 .../hbase/util/DeployCoprocessorCLI.java| 22 ++---
 .../hbase/util/ExtendCubeToHybridCLI.java   |  8 +-
 .../hbase/util/GridTableHBaseBenchmark.java | 34 +++
 .../kylin/storage/hbase/util/HBaseClean.java| 18 ++--
 .../hbase/util/HBaseRegionSizeCalculator.java   | 35 
 .../kylin/storage/hbase/util/HBaseUsage.java|  9 +-
 .../storage/hbase/util/HbaseStreamingInput.java | 30 +++
 .../hbase/util/HtableAlterMetadataCLI.java  |  9 +-
 .../storage/hbase/util/OrphanHBaseCleanJob.java | 19 ++--
 .../kylin/storage/hbase/util/PingHBaseCLI.java  | 15 ++--
 .../kylin/storage/hbase/util/RowCounterCLI.java | 11 +--
 .../storage/hbase/util/StorageCleanupJob.java   | 20 +++--
 .../storage/hbase/util/UpdateHTableHostCLI.java | 17 ++--
 .../observer/AggregateRegionObserverTest.java   | 26 ++
 .../v1/filter/TestFuzzyRowFilterV2EndToEnd.java |  5 +-
 .../org/apache/kylin/tool/CubeMigrationCLI.java | 36 
 .../kylin/tool/CubeMigrationCheckCLI.java   | 16 ++--
 .../kylin/tool/ExtendCubeToHybridCLI.java   |  8 +-
 .../apache/kylin/tool/StorageCleanupJob.java| 20 +++--
 53 files changed, 450 insertions(+), 500 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kylin/blob/edb9d54e/examples/test_case_data/sandbox/hbase-site.xml
--
diff --git a/examples/test_case_data/sandbox/hbase-site.xml 
b/examples/test_case_data/sandbox/hbase-site.xml
index 46d5345..734908e 100644
--- a/examples/test_case_data/sandbox/hbase-site.xml
+++ b/examples/test_case_data/sandbox/hbase-site.xml
@@ -190,22 +190,5 @@
 zookeeper.znode.parent
 /hbase-unsecure
 
-
-hbase.client.pause
-100
-General client pause value.  Used mostly as value to wait
-before running a retry of a failed get, region lookup, etc.
-See hbase.client.retries.number for description of how we backoff 
from
-this initial pause amount and how this pause works w/ 
retries.
-
-
-hbase.client.retries.number
-5
-Maximum retries.  Used as maximum for all retryable
-operations such as the getting of a cell's value, starting a row 
update,
-