drill-site git commit: minor edit

2018-01-18 Thread bridgetb 
Repository: drill-site
Updated Branches:
  refs/heads/asf-site 589b3552f -> bc41d8e02


minor edit


Project: http://git-wip-us.apache.org/repos/asf/drill-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/drill-site/commit/bc41d8e0
Tree: http://git-wip-us.apache.org/repos/asf/drill-site/tree/bc41d8e0
Diff: http://git-wip-us.apache.org/repos/asf/drill-site/diff/bc41d8e0

Branch: refs/heads/asf-site
Commit: bc41d8e0282f0cb793e122b66e0ca957ecd248ec
Parents: 589b355
Author: Bridget Bevens 
Authored: Thu Jan 18 17:55:30 2018 -0800
Committer: Bridget Bevens 
Committed: Thu Jan 18 17:55:30 2018 -0800

--
 docs/using-jdbc-with-squirrel-on-windows/index.html | 2 +-
 feed.xml| 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/drill-site/blob/bc41d8e0/docs/using-jdbc-with-squirrel-on-windows/index.html
--
diff --git a/docs/using-jdbc-with-squirrel-on-windows/index.html 
b/docs/using-jdbc-with-squirrel-on-windows/index.html
index 5284deb..fb5ba0c 100644
--- a/docs/using-jdbc-with-squirrel-on-windows/index.html
+++ b/docs/using-jdbc-with-squirrel-on-windows/index.html
@@ -1144,7 +1144,7 @@
   
 You can use the JDBC driver to connect to Drill from SQuirrel and 
run SQL queries. Verify that your system meets the prerequisites and then 
follow the steps listed.  
 
-Note: Drill interprets a wildcard in a SQL query as a 
multiline comment (/.../), which SQuirreL supports. As a result, SQuirreL tries 
to remove what it interprets as comments and submits an abruptly truncated 
query. To resolve this issue in SQuirreL, go to the menu and select 
File  New Session Properties  Tab SQL and uncheck the 
“Remove multi-line comment (/.../) from SQL before sending to 
database option located in the lower portion of the window. See https://sourceforge.net/p/squirrel-sql/bugs/1300/;>https://sourceforge.net/p/squirrel-sql/bugs/1300/
 for more information. 
+Note: Drill interprets a wildcard in an SQL query as a 
multiline comment (/.../), which SQuirreL supports. As a result, SQuirreL tries 
to remove what it interprets as comments and submits an abruptly truncated 
query. To resolve this issue in SQuirreL, go to the menu and select 
File  New Session Properties  Tab SQL and uncheck the 
“Remove multi-line comment (/.../) from SQL before sending to 
database option located in the lower portion of the window. See https://sourceforge.net/p/squirrel-sql/bugs/1300/;>https://sourceforge.net/p/squirrel-sql/bugs/1300/
 for more information. 
 
 Prerequisites
 

http://git-wip-us.apache.org/repos/asf/drill-site/blob/bc41d8e0/feed.xml
--
diff --git a/feed.xml b/feed.xml
index cdc7796..e941b1f 100644
--- a/feed.xml
+++ b/feed.xml
@@ -6,8 +6,8 @@
 
 /
 
-Thu, 18 Jan 2018 17:41:04 -0800
-Thu, 18 Jan 2018 17:41:04 -0800
+Thu, 18 Jan 2018 17:52:59 -0800
+Thu, 18 Jan 2018 17:52:59 -0800
 Jekyll v2.5.2

drill git commit: minor edit

2018-01-18 Thread bridgetb 
Repository: drill
Updated Branches:
  refs/heads/gh-pages f416850a0 -> d5bc6b0ac


minor edit


Project: http://git-wip-us.apache.org/repos/asf/drill/repo
Commit: http://git-wip-us.apache.org/repos/asf/drill/commit/d5bc6b0a
Tree: http://git-wip-us.apache.org/repos/asf/drill/tree/d5bc6b0a
Diff: http://git-wip-us.apache.org/repos/asf/drill/diff/d5bc6b0a

Branch: refs/heads/gh-pages
Commit: d5bc6b0ac64e69d7b226ab550678601327de7138
Parents: f416850
Author: Bridget Bevens 
Authored: Thu Jan 18 17:49:56 2018 -0800
Committer: Bridget Bevens 
Committed: Thu Jan 18 17:49:56 2018 -0800

--
 .../020-using-jdbc-with-squirrel-on-windows.md   | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/drill/blob/d5bc6b0a/_docs/odbc-jdbc-interfaces/020-using-jdbc-with-squirrel-on-windows.md
--
diff --git 
a/_docs/odbc-jdbc-interfaces/020-using-jdbc-with-squirrel-on-windows.md 
b/_docs/odbc-jdbc-interfaces/020-using-jdbc-with-squirrel-on-windows.md
index 0190172..31bd8a7 100644
--- a/_docs/odbc-jdbc-interfaces/020-using-jdbc-with-squirrel-on-windows.md
+++ b/_docs/odbc-jdbc-interfaces/020-using-jdbc-with-squirrel-on-windows.md
@@ -1,11 +1,11 @@
 ---
 title: "Using JDBC with SQuirreL on Windows"
-date: 2018-01-19 01:36:03 UTC
+date: 2018-01-19 01:49:57 UTC
 parent: "ODBC/JDBC Interfaces"
 ---
 You can use the JDBC driver to connect to Drill from SQuirrel and run SQL 
queries. Verify that your system meets the prerequisites and then follow the 
steps listed.  
 
-**Note:** Drill interprets a wildcard in a SQL query as a multiline comment 
(/.../), which SQuirreL supports. As a result, SQuirreL tries to remove what it 
interprets as comments and submits an abruptly truncated query. To resolve this 
issue in SQuirreL, go to the menu and select **File > New Session Properties > 
Tab SQL** and uncheck the *“Remove multi-line comment (/.../) from SQL before 
sending to database"* option located in the lower portion of the window. See 
[https://sourceforge.net/p/squirrel-sql/bugs/1300](https://sourceforge.net/p/squirrel-sql/bugs/1300/)/
 for more information. 
+**Note:** Drill interprets a wildcard in an SQL query as a multiline comment 
(/.../), which SQuirreL supports. As a result, SQuirreL tries to remove what it 
interprets as comments and submits an abruptly truncated query. To resolve this 
issue in SQuirreL, go to the menu and select **File > New Session Properties > 
Tab SQL** and uncheck the *“Remove multi-line comment (/.../) from SQL before 
sending to database"* option located in the lower portion of the window. See 
[https://sourceforge.net/p/squirrel-sql/bugs/1300](https://sourceforge.net/p/squirrel-sql/bugs/1300/)/
 for more information. 
 
 ## Prerequisites

[16/18] drill git commit: DRILL-3993: Changes for CALCITE-2018

2018-01-18 Thread amansinha 
DRILL-3993: Changes for CALCITE-2018


Project: http://git-wip-us.apache.org/repos/asf/drill/repo
Commit: http://git-wip-us.apache.org/repos/asf/drill/commit/22d0f7e5
Tree: http://git-wip-us.apache.org/repos/asf/drill/tree/22d0f7e5
Diff: http://git-wip-us.apache.org/repos/asf/drill/diff/22d0f7e5

Branch: refs/heads/master
Commit: 22d0f7e57b84fcb034624579f68b638ffdad6b7e
Parents: 6580674
Author: Volodymyr Vysotskyi 
Authored: Tue Dec 26 15:03:59 2017 +0200
Committer: Volodymyr Vysotskyi 
Committed: Tue Jan 16 12:10:13 2018 +0200

--
 .../cost/DrillRelMdDistinctRowCount.java| 31 ++--
 .../exec/planner/cost/DrillRelMdRowCount.java   |  2 +-
 .../drill/exec/planner/sql/SqlConverter.java| 12 ++--
 3 files changed, 33 insertions(+), 12 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/drill/blob/22d0f7e5/exec/java-exec/src/main/java/org/apache/drill/exec/planner/cost/DrillRelMdDistinctRowCount.java
--
diff --git 
a/exec/java-exec/src/main/java/org/apache/drill/exec/planner/cost/DrillRelMdDistinctRowCount.java
 
b/exec/java-exec/src/main/java/org/apache/drill/exec/planner/cost/DrillRelMdDistinctRowCount.java
index 5d4e4f6..ab7fb87 100644
--- 
a/exec/java-exec/src/main/java/org/apache/drill/exec/planner/cost/DrillRelMdDistinctRowCount.java
+++ 
b/exec/java-exec/src/main/java/org/apache/drill/exec/planner/cost/DrillRelMdDistinctRowCount.java
@@ -17,7 +17,9 @@
  */
 package org.apache.drill.exec.planner.cost;
 
+import org.apache.calcite.plan.volcano.RelSubset;
 import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Join;
 import org.apache.calcite.rel.metadata.ReflectiveRelMetadataProvider;
 import org.apache.calcite.rel.metadata.RelMdDistinctRowCount;
 import org.apache.calcite.rel.metadata.RelMetadataProvider;
@@ -27,7 +29,7 @@ import org.apache.calcite.util.BuiltInMethod;
 import org.apache.calcite.util.ImmutableBitSet;
 import org.apache.drill.exec.planner.logical.DrillScanRel;
 
-public class DrillRelMdDistinctRowCount extends RelMdDistinctRowCount{
+public class DrillRelMdDistinctRowCount extends RelMdDistinctRowCount {
   private static final DrillRelMdDistinctRowCount INSTANCE =
   new DrillRelMdDistinctRowCount();
 
@@ -35,18 +37,29 @@ public class DrillRelMdDistinctRowCount extends 
RelMdDistinctRowCount{
   ReflectiveRelMetadataProvider.reflectiveSource(
   BuiltInMethod.DISTINCT_ROW_COUNT.method, INSTANCE);
 
+  /**
+   * We need to override this method since Calcite and Drill calculate
+   * joined row count in different ways. It helps avoid a case when
+   * at the first time was used Drill join row count but at the second time
+   * Calcite row count was used. It may happen when
+   * {@link RelMdDistinctRowCount#getDistinctRowCount(Join, RelMetadataQuery,
+   * ImmutableBitSet, RexNode)} method is used and after that used
+   * another getDistinctRowCount method for parent rel, which just uses
+   * row count of input rel node (our join rel).
+   * It causes cost increase of best rel node when
+   * {@link RelSubset#propagateCostImprovements} is called.
+   *
+   * This is a part of the fix for CALCITE-2018.
+   */
   @Override
-  public Double getDistinctRowCount(RelNode rel, RelMetadataQuery mq, 
ImmutableBitSet groupKey, RexNode predicate) {
-if (rel instanceof DrillScanRel) {
-  return getDistinctRowCount((DrillScanRel) rel, groupKey, predicate);
-} else {
-  return super.getDistinctRowCount(rel, mq, groupKey, predicate);
-}
+  public Double getDistinctRowCount(Join rel, RelMetadataQuery mq,
+  ImmutableBitSet groupKey, RexNode predicate) {
+return getDistinctRowCount((RelNode) rel, mq, groupKey, predicate);
   }
 
-  private Double getDistinctRowCount(DrillScanRel scan, ImmutableBitSet 
groupKey, RexNode predicate) {
+  public Double getDistinctRowCount(DrillScanRel scan, RelMetadataQuery mq,
+  ImmutableBitSet groupKey, RexNode predicate) {
 // Consistent with the estimation of Aggregate row count in RelMdRowCount 
: distinctRowCount = rowCount * 10%.
 return scan.getRows() * 0.1;
   }
-
 }

http://git-wip-us.apache.org/repos/asf/drill/blob/22d0f7e5/exec/java-exec/src/main/java/org/apache/drill/exec/planner/cost/DrillRelMdRowCount.java
--
diff --git 
a/exec/java-exec/src/main/java/org/apache/drill/exec/planner/cost/DrillRelMdRowCount.java
 
b/exec/java-exec/src/main/java/org/apache/drill/exec/planner/cost/DrillRelMdRowCount.java
index 5cc2c6a..7f15fb3 100644
--- 
a/exec/java-exec/src/main/java/org/apache/drill/exec/planner/cost/DrillRelMdRowCount.java
+++ 
b/exec/java-exec/src/main/java/org/apache/drill/exec/planner/cost/DrillRelMdRowCount.java
@@ -32,7 +32,7 @@ import

[02/18] drill git commit: DRILL-3993: Add check for OOM in HashAgg

2018-01-18 Thread amansinha 
DRILL-3993: Add check for OOM in HashAgg


Project: http://git-wip-us.apache.org/repos/asf/drill/repo
Commit: http://git-wip-us.apache.org/repos/asf/drill/commit/d06a7cbb
Tree: http://git-wip-us.apache.org/repos/asf/drill/tree/d06a7cbb
Diff: http://git-wip-us.apache.org/repos/asf/drill/diff/d06a7cbb

Branch: refs/heads/master
Commit: d06a7cbbd7e5046e5017ffeb010b8b3c74123184
Parents: 3c9093e
Author: Volodymyr Vysotskyi 
Authored: Fri Dec 29 17:36:08 2017 +0200
Committer: Volodymyr Vysotskyi 
Committed: Tue Jan 16 12:10:13 2018 +0200

--
 .../physical/impl/aggregate/HashAggTemplate.java | 15 +++
 1 file changed, 15 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/drill/blob/d06a7cbb/exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggTemplate.java
--
diff --git 
a/exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggTemplate.java
 
b/exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggTemplate.java
index 4a81f3c..4b43b22 100644
--- 
a/exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggTemplate.java
+++ 
b/exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggTemplate.java
@@ -1280,6 +1280,8 @@ public abstract class HashAggTemplate implements 
HashAggregator {
   logger.trace("Reserved memory runs short, trying to {} a partition and 
retry Hash Table put() again.",
 is1stPhase ? "early return" : "spill");
 
+  checkForSpillPossibility(currentPartition);
+
   doSpill(currentPartition); // spill to free some memory
 
   retrySameIndex = true;
@@ -1303,6 +1305,8 @@ public abstract class HashAggTemplate implements 
HashAggregator {
   long memDiff = allocator.getAllocatedMemory() - allocatedBeforeHTput;
   if ( memDiff > 0 ) { logger.warn("Leak: HashTable put() OOM left behind 
{} bytes allocated",memDiff); }
 
+  checkForSpillPossibility(currentPartition);
+
   doSpill(currentPartition); // spill to free some memory
 
   retrySameIndex = true;
@@ -1379,6 +1383,17 @@ public abstract class HashAggTemplate implements 
HashAggregator {
 }
   }
 
+  /**
+   * Checks that spill is possible, otherwise throws {@link 
OutOfMemoryException}.
+   *
+   * @param currentPartition the partition that hit the memory limit
+   */
+  private void checkForSpillPossibility(int currentPartition) {
+if (chooseAPartitionToFlush(currentPartition, true) < 0) {
+  throw new OutOfMemoryException(getOOMErrorMsg("AGGR"));
+}
+  }
+
   private void spillIfNeeded(int currentPartition) { 
spillIfNeeded(currentPartition, false);}
   private void doSpill(int currentPartition) { spillIfNeeded(currentPartition, 
true);}
   /**

[08/18] drill git commit: DRILL-3993: Fix failed tests after Calcite update

2018-01-18 Thread amansinha 
DRILL-3993: Fix failed tests after Calcite update

- fix temporary table errors according to updated logic;
- fixed errors when we trying to make select from hbase table with schema name 
in query (example: "SELECT row_key FROM hbase.TestTableNullStr) from hbase 
schema (did "USE hbase" before). Added test for it;
- added fix for views which were created on Calcite 1.4 and test for it.


Project: http://git-wip-us.apache.org/repos/asf/drill/repo
Commit: http://git-wip-us.apache.org/repos/asf/drill/commit/9274cb92
Tree: http://git-wip-us.apache.org/repos/asf/drill/tree/9274cb92
Diff: http://git-wip-us.apache.org/repos/asf/drill/diff/9274cb92

Branch: refs/heads/master
Commit: 9274cb9204c6ebf5bd2d13fe4d02af5cebb48fa5
Parents: 0a525aa
Author: Roman Kulyk 
Authored: Thu Nov 2 18:22:36 2017 +
Committer: Volodymyr Vysotskyi 
Committed: Tue Jan 16 12:10:13 2018 +0200

--
 .../apache/drill/hbase/TestHBaseQueries.java|  7 ++
 .../org/apache/drill/exec/dotdrill/View.java|  7 +-
 .../drill/exec/planner/sql/SqlConverter.java| 99 ++--
 .../apache/drill/exec/rpc/user/UserSession.java |  9 ++
 .../org/apache/drill/exec/sql/TestCTTAS.java| 15 +++
 .../apache/drill/exec/sql/TestViewSupport.java  | 10 ++
 .../view/view_from_calcite_1_4.view.drill   | 10 ++
 7 files changed, 127 insertions(+), 30 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/drill/blob/9274cb92/contrib/storage-hbase/src/test/java/org/apache/drill/hbase/TestHBaseQueries.java
--
diff --git 
a/contrib/storage-hbase/src/test/java/org/apache/drill/hbase/TestHBaseQueries.java
 
b/contrib/storage-hbase/src/test/java/org/apache/drill/hbase/TestHBaseQueries.java
index c3ee7d9..ee839c5 100644
--- 
a/contrib/storage-hbase/src/test/java/org/apache/drill/hbase/TestHBaseQueries.java
+++ 
b/contrib/storage-hbase/src/test/java/org/apache/drill/hbase/TestHBaseQueries.java
@@ -102,4 +102,11 @@ public class TestHBaseQueries extends BaseHBaseTest {
 }
   }
 
+  @Test
+  public void testSelectFromSchema() throws Exception {
+setColumnWidths(new int[] {8, 15});
+test("USE hbase");
+runHBaseSQLVerifyCount("SELECT row_key\n"
++ " FROM hbase.TestTableNullStr t WHERE row_key='a1'", 1);
+  }
 }

http://git-wip-us.apache.org/repos/asf/drill/blob/9274cb92/exec/java-exec/src/main/java/org/apache/drill/exec/dotdrill/View.java
--
diff --git 
a/exec/java-exec/src/main/java/org/apache/drill/exec/dotdrill/View.java 
b/exec/java-exec/src/main/java/org/apache/drill/exec/dotdrill/View.java
index 2b69f00..3524d73 100644
--- a/exec/java-exec/src/main/java/org/apache/drill/exec/dotdrill/View.java
+++ b/exec/java-exec/src/main/java/org/apache/drill/exec/dotdrill/View.java
@@ -71,8 +71,11 @@ public class View {
 @JsonProperty("endUnit")TimeUnit endUnit,
 @JsonProperty("fractionalSecondPrecision")  Integer 
fractionalSecondPrecision,
 @JsonProperty("isNullable") Boolean isNullable) {
-  this.name = name;
-  this.type = type;
+  // Fix for views which were created on Calcite 1.4.
+  // After Calcite upgrade star "*" was changed on dynamic star "**"
+  // and type of star was changed to SqlTypeName.DYNAMIC_STAR
+  this.name = "*".equals(name) ? "**" : name;
+  this.type = "*".equals(name) && type == SqlTypeName.ANY ? 
SqlTypeName.DYNAMIC_STAR : type;
   this.precision = precision;
   this.scale = scale;
   this.intervalQualifier =

http://git-wip-us.apache.org/repos/asf/drill/blob/9274cb92/exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/SqlConverter.java
--
diff --git 
a/exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/SqlConverter.java
 
b/exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/SqlConverter.java
index f900587..8224d97 100644
--- 
a/exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/SqlConverter.java
+++ 
b/exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/SqlConverter.java
@@ -26,8 +26,6 @@ import com.google.common.collect.ImmutableList;
 import com.google.common.collect.Lists;
 import com.google.common.collect.Sets;
 import org.apache.calcite.adapter.java.JavaTypeFactory;
-import org.apache.calcite.avatica.util.Casing;
-import org.apache.calcite.avatica.util.Quoting;
 import org.apache.calcite.jdbc.CalciteSchema;
 import org.apache.calcite.jdbc.JavaTypeFactoryImpl;
 import org.apache.calcite.plan.ConventionTraitDef;
@@ -37,7 +35,6 @@ import org.apache.calcite.plan.RelOptTable;
 import org.apache.calcite.plan.volcano.VolcanoPlanner;
 import

[10/18] drill git commit: DRILL-3993: Changes after code review.

2018-01-18 Thread amansinha 
DRILL-3993: Changes after code review.


Project: http://git-wip-us.apache.org/repos/asf/drill/repo
Commit: http://git-wip-us.apache.org/repos/asf/drill/commit/663518e8
Tree: http://git-wip-us.apache.org/repos/asf/drill/tree/663518e8
Diff: http://git-wip-us.apache.org/repos/asf/drill/diff/663518e8

Branch: refs/heads/master
Commit: 663518e8ae1a05e29895a3994213a1e9819c955b
Parents: 688e043
Author: Volodymyr Vysotskyi 
Authored: Thu Dec 21 15:01:50 2017 +0200
Committer: Volodymyr Vysotskyi 
Committed: Tue Jan 16 12:10:13 2018 +0200

--
 .../apache/drill/exec/store/hbase/HBaseSchemaFactory.java| 8 
 .../main/java/org/apache/drill/exec/store/jdbc/JdbcPrel.java | 2 +-
 .../exec/store/openTSDB/schema/OpenTSDBSchemaFactory.java| 8 
 .../java/org/apache/drill/exec/planner/sql/SqlConverter.java | 8 +---
 exec/jdbc-all/pom.xml| 2 +-
 5 files changed, 11 insertions(+), 17 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/drill/blob/663518e8/contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/HBaseSchemaFactory.java
--
diff --git 
a/contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/HBaseSchemaFactory.java
 
b/contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/HBaseSchemaFactory.java
index 810daaa..b8e825b 100644
--- 
a/contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/HBaseSchemaFactory.java
+++ 
b/contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/HBaseSchemaFactory.java
@@ -75,10 +75,10 @@ public class HBaseSchemaFactory implements SchemaFactory {
   try {
 return new DrillHBaseTable(schemaName, plugin, scanSpec);
   } catch (Exception e) {
-// Calcite firstly is looking for a table in the default schema, if a 
table was not found,
-// it is looking in root schema.
-// If a table does not exist, a query will fail at validation stage,
-// so the error should not be thrown there.
+// Calcite firstly looks for a table in the default schema, if the 
table was not found,
+// it looks in the root schema.
+// If the table does not exist, a query will fail at validation stage,
+// so the error should not be thrown here.
 logger.warn("Failure while loading table '{}' for database '{}'.", 
name, schemaName, e.getCause());
 return null;
   }

http://git-wip-us.apache.org/repos/asf/drill/blob/663518e8/contrib/storage-jdbc/src/main/java/org/apache/drill/exec/store/jdbc/JdbcPrel.java
--
diff --git 
a/contrib/storage-jdbc/src/main/java/org/apache/drill/exec/store/jdbc/JdbcPrel.java
 
b/contrib/storage-jdbc/src/main/java/org/apache/drill/exec/store/jdbc/JdbcPrel.java
index e335c08..63752fa 100644
--- 
a/contrib/storage-jdbc/src/main/java/org/apache/drill/exec/store/jdbc/JdbcPrel.java
+++ 
b/contrib/storage-jdbc/src/main/java/org/apache/drill/exec/store/jdbc/JdbcPrel.java
@@ -62,7 +62,7 @@ public class JdbcPrel extends AbstractRelNode implements Prel 
{
 (JavaTypeFactory) getCluster().getTypeFactory());
 final JdbcImplementor.Result result =
 jdbcImplementor.visitChild(0, input.accept(new SubsetRemover()));
-sql = result.asSelect().toSqlString(dialect).getSql();
+sql = result.asStatement().toSqlString(dialect).getSql();
 rowType = input.getRowType();
   }
 

http://git-wip-us.apache.org/repos/asf/drill/blob/663518e8/contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/schema/OpenTSDBSchemaFactory.java
--
diff --git 
a/contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/schema/OpenTSDBSchemaFactory.java
 
b/contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/schema/OpenTSDBSchemaFactory.java
index ca93445..3b86a95 100644
--- 
a/contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/schema/OpenTSDBSchemaFactory.java
+++ 
b/contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/schema/OpenTSDBSchemaFactory.java
@@ -64,10 +64,10 @@ public class OpenTSDBSchemaFactory implements SchemaFactory 
{
   try {
 return new DrillOpenTSDBTable(schemaName, plugin, new 
Schema(plugin.getClient(), name), scanSpec);
   } catch (Exception e) {
-// Calcite firstly is looking for a table in the default schema, if a 
table was not found,
-// it is looking in root schema.
-// If a table does not exist, a query will fail at validation stage,
-// so the error should not be thrown there.
+// Calcite firstly looks for a table

[14/18] drill git commit: DRILL-3993: Resolve conflicts

2018-01-18 Thread amansinha 
DRILL-3993: Resolve conflicts


Project: http://git-wip-us.apache.org/repos/asf/drill/repo
Commit: http://git-wip-us.apache.org/repos/asf/drill/commit/3f0e517f
Tree: http://git-wip-us.apache.org/repos/asf/drill/tree/3f0e517f
Diff: http://git-wip-us.apache.org/repos/asf/drill/diff/3f0e517f

Branch: refs/heads/master
Commit: 3f0e517fb62a1ebad92fb473e787d343152920d6
Parents: 90b5e63
Author: Volodymyr Vysotskyi 
Authored: Tue Jan 9 12:44:12 2018 +0200
Committer: Volodymyr Vysotskyi 
Committed: Tue Jan 16 12:10:13 2018 +0200

--
 .../org/apache/drill/exec/planner/physical/SubsetTransformer.java  | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/drill/blob/3f0e517f/exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/SubsetTransformer.java
--
diff --git 
a/exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/SubsetTransformer.java
 
b/exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/SubsetTransformer.java
index 3562d5c..a871c3a 100644
--- 
a/exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/SubsetTransformer.java
+++ 
b/exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/SubsetTransformer.java
@@ -70,7 +70,7 @@ public abstract class SubsetTransformer
 
 //2, convert the candidateSet to targeted taitSets
 for (RelTraitSet traitSet: traitSets) {
-  RelNode newRel = RelOptRule.convert(candidateSet, traitSet);
+  RelNode newRel = RelOptRule.convert(candidateSet, traitSet.simplify());
   if(transformedRels.contains(newRel)) {
 continue;
   }

[18/18] drill git commit: DRILL-3993: Fix typo in DrillRelBuilder Javadoc

2018-01-18 Thread amansinha 
DRILL-3993: Fix typo in DrillRelBuilder Javadoc


Project: http://git-wip-us.apache.org/repos/asf/drill/repo
Commit: http://git-wip-us.apache.org/repos/asf/drill/commit/9e944c97
Tree: http://git-wip-us.apache.org/repos/asf/drill/tree/9e944c97
Diff: http://git-wip-us.apache.org/repos/asf/drill/diff/9e944c97

Branch: refs/heads/master
Commit: 9e944c97ee6f6c0d1705f09d531af35deed2e310
Parents: de3889a
Author: Volodymyr Vysotskyi 
Authored: Wed Jan 17 21:50:34 2018 +0200
Committer: Volodymyr Vysotskyi 
Committed: Thu Jan 18 12:56:05 2018 +0200

--
 .../main/java/org/apache/drill/exec/planner/DrillRelBuilder.java   | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/drill/blob/9e944c97/exec/java-exec/src/main/java/org/apache/drill/exec/planner/DrillRelBuilder.java
--
diff --git 
a/exec/java-exec/src/main/java/org/apache/drill/exec/planner/DrillRelBuilder.java
 
b/exec/java-exec/src/main/java/org/apache/drill/exec/planner/DrillRelBuilder.java
index 3923ba3..9010e30 100644
--- 
a/exec/java-exec/src/main/java/org/apache/drill/exec/planner/DrillRelBuilder.java
+++ 
b/exec/java-exec/src/main/java/org/apache/drill/exec/planner/DrillRelBuilder.java
@@ -38,7 +38,7 @@ public class DrillRelBuilder extends RelBuilder {
 
   /**
* Original method {@link RelBuilder#empty} returns empty values rel.
-   * In the order to preserve dara row types, filter with false predicate is 
created.
+   * In the order to preserve data row types, filter with false predicate is 
created.
*/
   @Override
   public RelBuilder empty() {

[12/18] drill git commit: DRILL-3993: Changes to support Calcite 1.13

2018-01-18 Thread amansinha 
http://git-wip-us.apache.org/repos/asf/drill/blob/9fabb612/exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/TypeInferenceUtils.java
--
diff --git 
a/exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/TypeInferenceUtils.java
 
b/exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/TypeInferenceUtils.java
index 523b721..ddf48ed 100644
--- 
a/exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/TypeInferenceUtils.java
+++ 
b/exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/TypeInferenceUtils.java
@@ -35,6 +35,7 @@ import org.apache.calcite.sql.SqlOperatorBinding;
 import org.apache.calcite.sql.fun.SqlAvgAggFunction;
 import org.apache.calcite.sql.parser.SqlParserPos;
 import org.apache.calcite.sql.type.SqlReturnTypeInference;
+import org.apache.calcite.sql.type.SqlTypeFamily;
 import org.apache.calcite.sql.type.SqlTypeName;
 
 import org.apache.drill.common.expression.ExpressionPosition;
@@ -73,7 +74,7 @@ public class TypeInferenceUtils {
   .put(TypeProtos.MinorType.TIMESTAMP, SqlTypeName.TIMESTAMP)
   .put(TypeProtos.MinorType.VARBINARY, SqlTypeName.VARBINARY)
   .put(TypeProtos.MinorType.INTERVALYEAR, SqlTypeName.INTERVAL_YEAR_MONTH)
-  .put(TypeProtos.MinorType.INTERVALDAY, SqlTypeName.INTERVAL_DAY_TIME)
+  .put(TypeProtos.MinorType.INTERVALDAY, SqlTypeName.INTERVAL_DAY)
   .put(TypeProtos.MinorType.MAP, SqlTypeName.MAP)
   .put(TypeProtos.MinorType.LIST, SqlTypeName.ARRAY)
   .put(TypeProtos.MinorType.LATE, SqlTypeName.ANY)
@@ -97,8 +98,19 @@ public class TypeInferenceUtils {
   .put(SqlTypeName.TIME, TypeProtos.MinorType.TIME)
   .put(SqlTypeName.TIMESTAMP, TypeProtos.MinorType.TIMESTAMP)
   .put(SqlTypeName.VARBINARY, TypeProtos.MinorType.VARBINARY)
+  .put(SqlTypeName.INTERVAL_YEAR, TypeProtos.MinorType.INTERVALYEAR)
   .put(SqlTypeName.INTERVAL_YEAR_MONTH, TypeProtos.MinorType.INTERVALYEAR)
-  .put(SqlTypeName.INTERVAL_DAY_TIME, TypeProtos.MinorType.INTERVALDAY)
+  .put(SqlTypeName.INTERVAL_MONTH, TypeProtos.MinorType.INTERVALYEAR)
+  .put(SqlTypeName.INTERVAL_DAY, TypeProtos.MinorType.INTERVALDAY)
+  .put(SqlTypeName.INTERVAL_DAY_HOUR, TypeProtos.MinorType.INTERVALDAY)
+  .put(SqlTypeName.INTERVAL_DAY_MINUTE, TypeProtos.MinorType.INTERVALDAY)
+  .put(SqlTypeName.INTERVAL_DAY_SECOND, TypeProtos.MinorType.INTERVALDAY)
+  .put(SqlTypeName.INTERVAL_HOUR, TypeProtos.MinorType.INTERVALDAY)
+  .put(SqlTypeName.INTERVAL_HOUR_MINUTE, TypeProtos.MinorType.INTERVALDAY)
+  .put(SqlTypeName.INTERVAL_HOUR_SECOND, TypeProtos.MinorType.INTERVALDAY)
+  .put(SqlTypeName.INTERVAL_MINUTE, TypeProtos.MinorType.INTERVALDAY)
+  .put(SqlTypeName.INTERVAL_MINUTE_SECOND, 
TypeProtos.MinorType.INTERVALDAY)
+  .put(SqlTypeName.INTERVAL_SECOND, TypeProtos.MinorType.INTERVALDAY)
 
   // SqlTypeName.CHAR is the type for Literals in Calcite, Drill treats 
Literals as VARCHAR also
   .put(SqlTypeName.CHAR, TypeProtos.MinorType.VARCHAR)
@@ -734,13 +746,13 @@ public class TypeInferenceUtils {
  SqlTypeName 
sqlTypeName,
  boolean 
isNullable) {
 RelDataType type;
-if (sqlTypeName == SqlTypeName.INTERVAL_DAY_TIME) {
+if (sqlTypeName.getFamily() == SqlTypeFamily.INTERVAL_DAY_TIME) {
   type = typeFactory.createSqlIntervalType(
   new SqlIntervalQualifier(
   TimeUnit.DAY,
   TimeUnit.MINUTE,
   SqlParserPos.ZERO));
-} else if (sqlTypeName == SqlTypeName.INTERVAL_YEAR_MONTH) {
+} else if (sqlTypeName.getFamily() == SqlTypeFamily.INTERVAL_YEAR_MONTH) {
   type = typeFactory.createSqlIntervalType(
   new SqlIntervalQualifier(
   TimeUnit.YEAR,

http://git-wip-us.apache.org/repos/asf/drill/blob/9fabb612/exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/DefaultSqlHandler.java
--
diff --git 
a/exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/DefaultSqlHandler.java
 
b/exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/DefaultSqlHandler.java
index 9515b3b..93113cb 100644
--- 
a/exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/DefaultSqlHandler.java
+++ 
b/exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/DefaultSqlHandler.java
@@ -23,6 +23,10 @@ import java.util.Collection;
 import java.util.List;
 import java.util.concurrent.TimeUnit;
 
+import com.google.common.collect.ImmutableList;
+import org.apache.calcite.plan.RelOptCostImpl;
+import org.apache.calcite.plan.RelOptLattice;
+import org.apache.calcite.plan.RelOptMaterialization;
 import org.apache.calcite.plan.RelOptPlanner;
 import org.apache.calcite.plan.RelOptRule;

[04/18] drill git commit: DRILL-3993: Fix unit test failures connected with support Calcite 1.13

2018-01-18 Thread amansinha 
DRILL-3993: Fix unit test failures connected with support Calcite 1.13

- Use root schema as default for describe table statement.
Fix TestOpenTSDBPlugin.testDescribe() and 
TestInfoSchemaOnHiveStorage.varCharMaxLengthAndDecimalPrecisionInInfoSchema() 
unit tests.
- Modify expected results for tests:
TestPreparedStatementProvider.invalidQueryValidationError();
TestProjectPushDown.testTPCH1();
TestProjectPushDown.testTPCH3();
TestStorageBasedHiveAuthorization.selectUser1_db_u0_only();
TestStorageBasedHiveAuthorization.selectUser0_db_u1g1_only()
- Fix TestCTAS.whenTableQueryColumnHasStarAndTableFiledListIsSpecified(), 
TestViewSupport.createViewWhenViewQueryColumnHasStarAndViewFiledListIsSpecified(),
 TestInbuiltHiveUDFs.testIf(), testDisableUtf8SupportInQueryString unit tests.
- Fix UnsupportedOperationException and NPE for jdbc tests.
- Fix AssertionError: Conversion to relational algebra failed to preserve 
datatypes

*DrillCompoundIdentifier:
According to the changes, made in [CALCITE-546], star Identifier is replaced by 
empty string during parsing the query. Since Drill uses its own 
DrillCompoundIdentifier, it should also replace star by empty string before 
creating SqlIdentifier instance to avoid further errors connected with star 
column. see SqlIdentifier.isStar() method.

*SqlConverter:
In [CALCITE-1417] added simplification of expressions which should be projected 
every time when a new project rel node is created using RelBuilder. It causes 
assertion errors connected with types nullability. This hook was set to false 
to avoid project expressions simplification. See usage of this hook and 
RelBuilder.project() method.

In Drill the type nullability of the function depends on only the nullability 
of its arguments. In some cases, a function may return null value even if it 
had non-nullable arguments. When Calice simplifies expressions, it checks that 
the type of the result is the same as the type of the expression. Otherwise, 
makeCast() method is called. But when a function returns null literal, this 
cast does nothing, even when the function has a non-nullable type. So to avoid 
this issue, method makeCast() was overridden.

*DrillAvgVarianceConvertlet:
Problem with sum0 and specific changes in old Calcite (it is CALCITE-777). (see 
HistogramShuttle.visitCall method) Changes were made to avoid changes in 
Calcite.

*SqlConverter, DescribeTableHandler, ShowTablesHandler:
New Calcite tries to combine both default and specified workspaces during the 
query validation. In some cases, for example, when describe table statement is 
used, Calcite tries to find INFORMATION_SCHEMA in the schema used as default. 
When it does not find the schema, it tries to find a table with such name. For 
some storage plugins, such as opentsdb and hbase, when a table was not found, 
the error is thrown, and the query fails. To avoid this issue, default schema 
was changed to root schema for validation stage for describe table and show 
tables queries.


Project: http://git-wip-us.apache.org/repos/asf/drill/repo
Commit: http://git-wip-us.apache.org/repos/asf/drill/commit/3896a582
Tree: http://git-wip-us.apache.org/repos/asf/drill/tree/3896a582
Diff: http://git-wip-us.apache.org/repos/asf/drill/diff/3896a582

Branch: refs/heads/master
Commit: 3896a58243f310c5d9466a98edc205b61f9dd2e7
Parents: 9fabb61
Author: Volodymyr Vysotskyi 
Authored: Fri Nov 3 12:18:09 2017 +
Committer: Volodymyr Vysotskyi 
Committed: Tue Jan 16 12:10:13 2018 +0200

--
 .../drill/exec/fn/hive/TestInbuiltHiveUDFs.java |  2 +-
 .../hive/TestStorageBasedHiveAuthorization.java |  6 +-
 .../planner/sql/DrillAvgVarianceConvertlet.java |  8 +--
 .../sql/DrillCalciteSqlAggFunctionWrapper.java  |  6 ++
 .../drill/exec/planner/sql/SqlConverter.java| 53 +-
 .../planner/sql/handlers/DefaultSqlHandler.java |  4 +-
 .../sql/handlers/DescribeTableHandler.java  | 16 -
 .../planner/sql/handlers/ShowTablesHandler.java | 15 
 .../planner/sql/handlers/SqlHandlerUtil.java|  4 +-
 .../sql/parser/DrillCompoundIdentifier.java | 72 ++--
 .../ischema/InfoSchemaRecordGenerator.java  |  3 +-
 .../org/apache/drill/TestProjectPushDown.java   |  4 +-
 .../drill/TestUtf8SupportInQueryString.java | 24 +++
 .../prepare/TestPreparedStatementProvider.java  |  5 +-
 .../apache/drill/jdbc/impl/DrillMetaImpl.java   | 14 ++--
 .../jdbc/impl/DrillPreparedStatementImpl.java   |  8 ++-
 ...l2489CallsAfterCloseThrowExceptionsTest.java | 45 +---
 ...69UnsupportedReportsUseSqlExceptionTest.java | 15 ++--
 18 files changed, 189 insertions(+), 115 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/drill/blob/3896a582/contrib/storage-hive/core/src/test/java/org/apache/drill/exec/fn/hive/TestInbuiltHiveUDFs.java

[03/18] drill git commit: DRILL-3993: Fix aggregate exchange rules for the cases when aggregate rel node contains several calls

2018-01-18 Thread amansinha 
DRILL-3993: Fix aggregate exchange rules for the cases when aggregate rel node 
contains several calls


Project: http://git-wip-us.apache.org/repos/asf/drill/repo
Commit: http://git-wip-us.apache.org/repos/asf/drill/commit/3c9093e3
Tree: http://git-wip-us.apache.org/repos/asf/drill/tree/3c9093e3
Diff: http://git-wip-us.apache.org/repos/asf/drill/diff/3c9093e3

Branch: refs/heads/master
Commit: 3c9093e32a095bd40832bcd8fe67ab20898537c4
Parents: 22d0f7e
Author: Volodymyr Vysotskyi 
Authored: Thu Jan 4 16:05:53 2018 +0200
Committer: Volodymyr Vysotskyi 
Committed: Tue Jan 16 12:10:13 2018 +0200

--
 .../exec/planner/physical/AggPrelBase.java  | 24 ++
 .../exec/planner/physical/AggPruleBase.java | 26 
 .../exec/planner/physical/HashAggPrule.java | 22 +++--
 .../exec/planner/physical/StreamAggPrule.java   | 25 ++-
 4 files changed, 69 insertions(+), 28 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/drill/blob/3c9093e3/exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/AggPrelBase.java
--
diff --git 
a/exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/AggPrelBase.java
 
b/exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/AggPrelBase.java
index 8c69930..232473b 100644
--- 
a/exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/AggPrelBase.java
+++ 
b/exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/AggPrelBase.java
@@ -1,4 +1,4 @@
-/**
+/*
  * Licensed to the Apache Software Foundation (ASF) under one
  * or more contributor license agreements.  See the NOTICE file
  * distributed with this work for additional information
@@ -31,7 +31,6 @@ import org.apache.drill.common.logical.data.NamedExpression;
 import org.apache.drill.exec.planner.common.DrillAggregateRelBase;
 import org.apache.drill.exec.planner.physical.visitor.PrelVisitor;
 import org.apache.calcite.rel.core.AggregateCall;
-import org.apache.calcite.rel.core.Aggregate;
 import org.apache.calcite.rel.InvalidRelException;
 import org.apache.calcite.rel.RelNode;
 import org.apache.calcite.plan.RelOptCluster;
@@ -44,14 +43,13 @@ import org.apache.calcite.sql.SqlKind;
 import org.apache.calcite.sql.type.OperandTypes;
 import org.apache.calcite.sql.type.ReturnTypes;
 
-import java.util.BitSet;
 import java.util.Collections;
 import java.util.Iterator;
 import java.util.List;
 
 public abstract class AggPrelBase extends DrillAggregateRelBase implements 
Prel {
 
-  public static enum OperatorPhase {PHASE_1of1, PHASE_1of2, PHASE_2of2};
+  public enum OperatorPhase {PHASE_1of1, PHASE_1of2, PHASE_2of2}
 
   protected OperatorPhase operPhase = OperatorPhase.PHASE_1of1 ; // default 
phase
   protected List keys = Lists.newArrayList();
@@ -70,11 +68,14 @@ public abstract class AggPrelBase extends 
DrillAggregateRelBase implements Prel
 
 public SqlSumCountAggFunction(RelDataType type) {
   super("$SUM0",
+  null,
   SqlKind.OTHER_FUNCTION,
   ReturnTypes.BIGINT, // use the inferred return type of 
SqlCountAggFunction
   null,
   OperandTypes.NUMERIC,
-  SqlFunctionCategory.NUMERIC);
+  SqlFunctionCategory.NUMERIC,
+  false,
+  false);
 
   this.type = type;
 }
@@ -143,20 +144,24 @@ public abstract class AggPrelBase extends 
DrillAggregateRelBase implements Prel
   // If we are doing a COUNT aggregate in Phase1of2, then in Phase2of2 
we should SUM the COUNTs,
   SqlAggFunction sumAggFun = new 
SqlSumCountAggFunction(aggCall.e.getType());
   AggregateCall newAggCall =
-  new AggregateCall(
+  AggregateCall.create(
   sumAggFun,
   aggCall.e.isDistinct(),
+  aggCall.e.isApproximate(),
   Collections.singletonList(aggExprOrdinal),
+  aggCall.e.filterArg,
   aggCall.e.getType(),
   aggCall.e.getName());
 
   phase2AggCallList.add(newAggCall);
 } else {
   AggregateCall newAggCall =
-  new AggregateCall(
+  AggregateCall.create(
   aggCall.e.getAggregation(),
   aggCall.e.isDistinct(),
+  aggCall.e.isApproximate(),
   Collections.singletonList(aggExprOrdinal),
+  aggCall.e.filterArg,
   aggCall.e.getType(),
   aggCall.e.getName());
 
@@ -174,10 +179,9 @@ public abstract class AggPrelBase extends 
DrillAggregateRelBase implements Prel
 
 // for count(1).
 if (args.isEmpty()) {
-  args.add(new ValueExpressions.LongExpression(1l));
+  args.add(new

[13/18] drill git commit: DRILL-3993: Changes to support Calcite 1.13

2018-01-18 Thread amansinha 
DRILL-3993: Changes to support Calcite 1.13

- fixed all compiling errors (main changes were: Maven changes, chenges RelNode 
-> RelRoot, implementing some new methods from updated interfaces, chenges some 
literals, logger changes);
- fixed unexpected column errors, validation errors and assertion errors after 
Calcite update;
- fixed describe table/schema statement according to updated logic;
- added fixes with time-intervals;
- changed precision of BINARY to 65536 (was 1048576) according to updated logic 
(Calcite overrides bigger precision to own maxPrecision);
- ignored some incorrect tests with DRILL-3244;
- changed "Table not found" message to "Object not found within" according to 
new Calcite changes.


Project: http://git-wip-us.apache.org/repos/asf/drill/repo
Commit: http://git-wip-us.apache.org/repos/asf/drill/commit/9fabb612
Tree: http://git-wip-us.apache.org/repos/asf/drill/tree/9fabb612
Diff: http://git-wip-us.apache.org/repos/asf/drill/diff/9fabb612

Branch: refs/heads/master
Commit: 9fabb612f16f6f541b3bde68ad7d734cad26df33
Parents: 450e670
Author: Roman Kulyk 
Authored: Tue Aug 29 14:10:24 2017 +
Committer: Volodymyr Vysotskyi 
Committed: Tue Jan 16 12:10:13 2018 +0200

--
 contrib/storage-hive/hive-exec-shade/pom.xml|   6 +-
 .../exec/store/jdbc/JdbcExpressionCheck.java|  20 +++-
 .../apache/drill/exec/store/jdbc/JdbcPrel.java  |   2 +-
 exec/java-exec/pom.xml  |   4 +
 exec/java-exec/src/main/codegen/data/Parser.tdd |  23 -
 .../src/main/codegen/includes/parserImpls.ftl   |  25 -
 .../org/apache/drill/exec/dotdrill/View.java|  15 ++-
 .../planner/FileSystemPartitionDescriptor.java  |   6 +-
 .../exec/planner/common/DrillStoreRelBase.java  |   9 +-
 .../exec/planner/common/DrillValuesRelBase.java |  13 ++-
 .../planner/logical/DrillAggregateRule.java |  10 +-
 .../exec/planner/logical/DrillConditions.java   |   8 +-
 .../planner/logical/DrillConstExecutor.java |  14 ++-
 .../exec/planner/logical/DrillJoinRule.java |  12 +--
 .../drill/exec/planner/logical/DrillOptiq.java  |  32 +-
 .../logical/DrillProjectSetOpTransposeRule.java |   5 +-
 .../logical/DrillPushProjectPastFilterRule.java |   7 +-
 .../logical/DrillPushProjectPastJoinRule.java   |   5 +-
 .../logical/DrillReduceAggregatesRule.java  |   7 +-
 .../logical/DrillReduceExpressionsRule.java |  25 ++---
 .../exec/planner/logical/DrillRelFactories.java |  11 +-
 .../exec/planner/logical/DrillStoreRel.java |   9 +-
 .../exec/planner/logical/DrillUnionAllRule.java |   8 +-
 .../exec/planner/logical/DrillViewTable.java|   9 +-
 .../exec/planner/physical/HashAggPrel.java  |   5 -
 .../exec/planner/physical/HashAggPrule.java |   7 +-
 .../exec/planner/physical/HashJoinPrel.java |   1 -
 .../exec/planner/physical/HashJoinPrule.java|   9 +-
 .../exec/planner/physical/MergeJoinPrule.java   |   6 +-
 .../planner/physical/NestedLoopJoinPrule.java   |   6 +-
 .../exec/planner/physical/StreamAggPrule.java   |   6 +-
 .../exec/planner/physical/UnionAllPrule.java|  13 ++-
 .../planner/physical/UnionDistinctPrule.java|  14 +--
 .../exec/planner/physical/WindowPrule.java  |   8 +-
 .../physical/visitor/TopProjectVisitor.java |   2 +-
 .../sql/DrillCalciteSqlAggFunctionWrapper.java  |   5 +-
 .../sql/DrillCalciteSqlFunctionWrapper.java |   5 +-
 .../sql/DrillCalciteSqlOperatorWrapper.java |   5 +-
 .../exec/planner/sql/DrillParserConfig.java |  12 +++
 .../drill/exec/planner/sql/DrillSqlWorker.java  |  15 +++
 .../drill/exec/planner/sql/SqlConverter.java|  41 
 .../exec/planner/sql/TypeInferenceUtils.java|  20 +++-
 .../planner/sql/handlers/DefaultSqlHandler.java |  29 --
 .../sql/handlers/DescribeSchemaHandler.java |   2 +-
 .../sql/handlers/DescribeTableHandler.java  |   5 +-
 .../planner/sql/handlers/FindLimit0Visitor.java |  29 +-
 .../sql/parser/CompoundIdentifierConverter.java |   3 +-
 .../sql/parser/DrillSqlDescribeTable.java   |  73 +
 .../planner/sql/parser/SqlDescribeSchema.java   |  81 ---
 .../planner/sql/parser/SqlDescribeTable.java| 102 ---
 .../exec/planner/torel/ConversionContext.java   |   7 +-
 .../planner/types/RelDataTypeDrillImpl.java |   6 +-
 .../exec/planner/types/RelDataTypeHolder.java   |  12 ++-
 .../apache/drill/exec/store/AbstractSchema.java |   5 +
 .../drill/exec/store/SchemaTreeProvider.java|   4 +-
 .../drill/exec/store/ischema/Records.java   |  41 +++-
 .../java/org/apache/drill/PlanningBase.java |   4 +-
 .../exec/planner/logical/DrillOptiqTest.java|   2 +-
 .../exec/sql/TestSqlBracketlessSyntax.java  |   4 +-
 .../store/parquet/TestParquetGroupScan.java |   4 +-
 exec/jdbc-all/pom.xml   |   4 +-
 exec/jdbc/pom.xml   |   5 +-

[15/18] drill git commit: DRILL-3993: Add unit tests for DRILL-4469 & DRILL-5768

2018-01-18 Thread amansinha 
DRILL-3993: Add unit tests for DRILL-4469 & DRILL-5768


Project: http://git-wip-us.apache.org/repos/asf/drill/repo
Commit: http://git-wip-us.apache.org/repos/asf/drill/commit/90b5e639
Tree: http://git-wip-us.apache.org/repos/asf/drill/tree/90b5e639
Diff: http://git-wip-us.apache.org/repos/asf/drill/diff/90b5e639

Branch: refs/heads/master
Commit: 90b5e6392d09db44ddef058b8c8caf88f52e23a2
Parents: d06a7cb
Author: Volodymyr Vysotskyi 
Authored: Fri Jan 5 15:35:45 2018 +0200
Committer: Volodymyr Vysotskyi 
Committed: Tue Jan 16 12:10:13 2018 +0200

--
 .../apache/drill/exec/TestWindowFunctions.java  | 22 ++--
 .../exec/fn/impl/TestAggregateFunctions.java| 15 +
 2 files changed, 35 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/drill/blob/90b5e639/exec/java-exec/src/test/java/org/apache/drill/exec/TestWindowFunctions.java
--
diff --git 
a/exec/java-exec/src/test/java/org/apache/drill/exec/TestWindowFunctions.java 
b/exec/java-exec/src/test/java/org/apache/drill/exec/TestWindowFunctions.java
index 3851228..db72ec2 100644
--- 
a/exec/java-exec/src/test/java/org/apache/drill/exec/TestWindowFunctions.java
+++ 
b/exec/java-exec/src/test/java/org/apache/drill/exec/TestWindowFunctions.java
@@ -1,4 +1,4 @@
-/**
+/*
  * Licensed to the Apache Software Foundation (ASF) under one
  * or more contributor license agreements.  See the NOTICE file
  * distributed with this work for additional information
@@ -20,7 +20,6 @@ package org.apache.drill.exec;
 import org.apache.drill.test.BaseTestQuery;
 import org.apache.drill.categories.UnlikelyTest;
 import org.apache.drill.common.exceptions.UserException;
-import org.apache.drill.common.util.DrillFileUtils;
 import org.apache.drill.exec.proto.UserBitShared;
 import org.apache.drill.exec.work.foreman.SqlUnsupportedException;
 import org.apache.drill.exec.work.foreman.UnsupportedFunctionException;
@@ -935,4 +934,23 @@ public class TestWindowFunctions extends BaseTestQuery {
   assert(ex.getMessage().contains("Expression 'n_nationkey' is not being 
grouped"));
 }
   }
+
+  @Test // DRILL-4469
+  public void testWindowOnSubqueryWithStar() throws Exception {
+String query = "SELECT SUM(n_nationkey) OVER w as s\n" +
+"FROM (SELECT * FROM cp.`tpch/nation.parquet`) subQry\n" +
+"WINDOW w AS (PARTITION BY REGION ORDER BY n_nationkey)\n" +
+"limit 1";
+
+final String[] expectedPlan = {"Project.*\\$0=\\[ITEM\\(\\$1, 
'n_nationkey'\\)\\].*"};
+PlanTestBase.testPlanMatchingPatterns(query, expectedPlan, new String[]{});
+
+testBuilder()
+.sqlQuery(query)
+.ordered()
+.baselineColumns("s")
+.baselineValues(0L)
+.build()
+.run();
+  }
 }

http://git-wip-us.apache.org/repos/asf/drill/blob/90b5e639/exec/java-exec/src/test/java/org/apache/drill/exec/fn/impl/TestAggregateFunctions.java
--
diff --git 
a/exec/java-exec/src/test/java/org/apache/drill/exec/fn/impl/TestAggregateFunctions.java
 
b/exec/java-exec/src/test/java/org/apache/drill/exec/fn/impl/TestAggregateFunctions.java
index ebad2f7..120f1e6 100644
--- 
a/exec/java-exec/src/test/java/org/apache/drill/exec/fn/impl/TestAggregateFunctions.java
+++ 
b/exec/java-exec/src/test/java/org/apache/drill/exec/fn/impl/TestAggregateFunctions.java
@@ -20,6 +20,7 @@ package org.apache.drill.exec.fn.impl;
 import com.google.common.collect.Lists;
 import com.google.common.collect.Maps;
 import org.apache.commons.lang3.tuple.Pair;
+import org.apache.drill.common.exceptions.UserRemoteException;
 import org.apache.drill.test.BaseTestQuery;
 import org.apache.drill.categories.OperatorTest;
 import org.apache.drill.PlanTestBase;
@@ -42,6 +43,9 @@ import java.nio.file.Paths;
 import java.util.List;
 import java.util.Map;
 
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
+
 @Category({SqlFunctionTest.class, OperatorTest.class, PlannerTest.class})
 public class TestAggregateFunctions extends BaseTestQuery {
 
@@ -608,4 +612,15 @@ public class TestAggregateFunctions extends BaseTestQuery {
   .baselineValues(1L, 1L, 1L, 0L, 1L)
   .go();
   }
+
+  @Test // DRILL-5768
+  public void testGroupByWithoutAggregate() throws Exception {
+try {
+  test("select * from cp.`tpch/nation.parquet` group by n_regionkey");
+  fail("Exception was not thrown");
+} catch (UserRemoteException e) {
+  assertTrue("No expected current \"Expression 'tpch/nation.parquet.**' is 
not being grouped\"",
+  e.getMessage().matches(".*Expression 
'tpch/nation\\.parquet\\.\\*\\*' is not being grouped(.*\\n)*"));
+}
+  }
 }

[11/18] drill git commit: DRILL-3993: Changes for CALCITE-2092

2018-01-18 Thread amansinha 
DRILL-3993: Changes for CALCITE-2092


Project: http://git-wip-us.apache.org/repos/asf/drill/repo
Commit: http://git-wip-us.apache.org/repos/asf/drill/commit/65806741
Tree: http://git-wip-us.apache.org/repos/asf/drill/tree/65806741
Diff: http://git-wip-us.apache.org/repos/asf/drill/diff/65806741

Branch: refs/heads/master
Commit: 6580674134339fb31cd42f8003e2494f04cac205
Parents: 663518e
Author: Volodymyr Vysotskyi 
Authored: Tue Dec 26 14:03:54 2017 +0200
Committer: Volodymyr Vysotskyi 
Committed: Tue Jan 16 12:10:13 2018 +0200

--
 .../java/org/apache/drill/exec/planner/sql/SqlConverter.java  | 7 +++
 1 file changed, 7 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/drill/blob/65806741/exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/SqlConverter.java
--
diff --git 
a/exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/SqlConverter.java
 
b/exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/SqlConverter.java
index 563f58e..8ad4df7 100644
--- 
a/exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/SqlConverter.java
+++ 
b/exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/SqlConverter.java
@@ -65,6 +65,7 @@ import org.apache.calcite.sql.validate.SqlValidatorScope;
 import org.apache.calcite.sql.validate.SqlValidatorUtil;
 import org.apache.calcite.sql2rel.RelDecorrelator;
 import org.apache.calcite.sql2rel.SqlToRelConverter;
+import org.apache.calcite.tools.RelBuilderFactory;
 import org.apache.calcite.util.Util;
 import org.apache.commons.collections.ListUtils;
 import org.apache.drill.common.config.DrillConfig;
@@ -76,6 +77,7 @@ import org.apache.drill.exec.ops.QueryContext;
 import org.apache.drill.exec.ops.UdfUtilities;
 import org.apache.drill.exec.planner.cost.DrillCostBase;
 import org.apache.drill.exec.planner.logical.DrillConstExecutor;
+import org.apache.drill.exec.planner.logical.DrillRelFactories;
 import org.apache.drill.exec.planner.physical.DrillDistributionTraitDef;
 import org.apache.drill.exec.planner.physical.PlannerSettings;
 import org.apache.drill.exec.rpc.user.UserSession;
@@ -443,6 +445,11 @@ public class SqlConverter {
 public int getInSubQueryThreshold() {
   return inSubqueryThreshold;
 }
+
+@Override
+public RelBuilderFactory getRelBuilderFactory() {
+  return DrillRelFactories.LOGICAL_BUILDER;
+}
   }
 
   /**

[01/18] drill git commit: DRILL-5089: Dynamically load schema of storage plugin only when needed for every query

2018-01-18 Thread amansinha 
Repository: drill
Updated Branches:
  refs/heads/master ef0fafea2 -> 9e944c97e


DRILL-5089: Dynamically load schema of storage plugin only when needed for 
every query

For each query, loading all storage plugins and loading all workspaces under 
file system plugins is not needed.

This patch use DynamicRootSchema as the root schema for Drill. Which loads 
correspondent storage only when needed.

infoschema to read full schema information and load second level schema 
accordingly.

for workspaces under the same Filesyetm, no need to create FileSystem for each 
workspace.

use fs.access API to check permission which is available after HDFS 2.6 except 
for windows + local file system case.

Add unit tests to test with a broken mock storage: with a storage that will 
throw Exception in regiterSchema method,
all queries even on good storages shall fail without this fix(Drill still load 
all schemas from all storages).

(cherry picked from commit a66d1d7)


Project: http://git-wip-us.apache.org/repos/asf/drill/repo
Commit: http://git-wip-us.apache.org/repos/asf/drill/commit/18a71a38
Tree: http://git-wip-us.apache.org/repos/asf/drill/tree/18a71a38
Diff: http://git-wip-us.apache.org/repos/asf/drill/diff/18a71a38

Branch: refs/heads/master
Commit: 18a71a38f6bd1fd33d21d1c68fc23c5901b0080a
Parents: 3f0e517
Author: chunhui-shi 
Authored: Fri Nov 3 02:06:25 2017 +0200
Committer: Volodymyr Vysotskyi 
Committed: Tue Jan 16 12:10:13 2018 +0200

--
 .../apache/calcite/jdbc/DynamicRootSchema.java  | 129 +++
 .../org/apache/calcite/jdbc/DynamicSchema.java  |  57 
 .../apache/drill/exec/ops/FragmentContext.java  |   9 +-
 .../org/apache/drill/exec/ops/QueryContext.java |  11 +-
 .../drill/exec/planner/sql/SqlConverter.java|  10 +-
 .../drill/exec/store/SchemaTreeProvider.java|  31 -
 .../exec/store/StoragePluginRegistryImpl.java   |   2 +
 .../exec/store/dfs/FileSystemSchemaFactory.java |  23 +++-
 .../exec/store/dfs/WorkspaceSchemaFactory.java  |  87 -
 .../store/ischema/InfoSchemaBatchCreator.java   |   2 +-
 .../exec/store/mock/MockBreakageStorage.java|  47 +++
 .../exec/store/mock/MockStorageEngine.java  |   7 +-
 .../exec/work/metadata/MetadataProvider.java|   2 +-
 .../drill/exec/physical/impl/TestSchema.java|  87 +
 .../drill/test/ClusterFixtureBuilder.java   |   4 +
 .../drill/test/ClusterMockStorageFixture.java   |  51 
 16 files changed, 512 insertions(+), 47 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/drill/blob/18a71a38/exec/java-exec/src/main/java/org/apache/calcite/jdbc/DynamicRootSchema.java
--
diff --git 
a/exec/java-exec/src/main/java/org/apache/calcite/jdbc/DynamicRootSchema.java 
b/exec/java-exec/src/main/java/org/apache/calcite/jdbc/DynamicRootSchema.java
new file mode 100644
index 000..cde46f2
--- /dev/null
+++ 
b/exec/java-exec/src/main/java/org/apache/calcite/jdbc/DynamicRootSchema.java
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.jdbc;
+
+import com.google.common.collect.Lists;
+import org.apache.calcite.DataContext;
+
+import org.apache.calcite.linq4j.tree.Expression;
+import org.apache.calcite.linq4j.tree.Expressions;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.calcite.schema.impl.AbstractSchema;
+import org.apache.calcite.util.BuiltInMethod;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.StoragePlugin;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.exec.store.SubSchemaWrapper;
+
+import java.io.IOException;
+import java.util.List;
+
+/**
+ * This class is to allow us loading schemas from storage plugins later when 
{@link #getSubSchema(String, boolean)}
+ * is called.
+ */
+public class DynamicRootSchema extends DynamicSchema {
+  private static final org.slf4j.Logger logger =

[07/18] drill git commit: DRILL-3993: Changes to support Calcite 1.15.

2018-01-18 Thread amansinha 
DRILL-3993: Changes to support Calcite 1.15.

Fix AssertionError: type mismatch for tests with aggregate functions.
Fix VARIANCE agg function
Remove using deprecated Subtype enum
Fix 'Failure while loading table a in database hbase' error
Fix 'Field ordinal 1 is invalid for  type '(DrillRecordRow[*])'' unit test 
failures


Project: http://git-wip-us.apache.org/repos/asf/drill/repo
Commit: http://git-wip-us.apache.org/repos/asf/drill/commit/d59f0cda
Tree: http://git-wip-us.apache.org/repos/asf/drill/tree/d59f0cda
Diff: http://git-wip-us.apache.org/repos/asf/drill/diff/d59f0cda

Branch: refs/heads/master
Commit: d59f0cda481d98a15d11ecbfb1463db53c954dfb
Parents: 9274cb9
Author: Volodymyr Vysotskyi 
Authored: Fri Dec 8 15:05:03 2017 +0200
Committer: Volodymyr Vysotskyi 
Committed: Tue Jan 16 12:10:13 2018 +0200

--
 .../exec/store/hbase/HBaseSchemaFactory.java| 11 ++-
 .../store/hive/schema/HiveDatabaseSchema.java   | 23 --
 .../openTSDB/schema/OpenTSDBSchemaFactory.java  |  9 +++
 .../src/main/codegen/includes/parserImpls.ftl   | 12 +--
 .../logical/DrillReduceAggregatesRule.java  | 83 
 .../drill/exec/planner/logical/DrillTable.java  | 15 +++-
 .../planner/logical/DrillTranslatableTable.java | 17 +++-
 .../exec/planner/logical/DrillViewTable.java| 13 +++
 .../exec/planner/physical/AggPruleBase.java | 10 ++-
 .../planner/sql/DrillAvgVarianceConvertlet.java |  6 +-
 .../exec/planner/sql/DrillConvertletTable.java  | 16 ++--
 .../drill/exec/planner/sql/SqlConverter.java| 19 -
 .../exec/planner/sql/TypeInferenceUtils.java| 35 -
 .../apache/drill/exec/store/AbstractSchema.java |  8 +-
 .../exec/store/mock/MockStorageEngine.java  | 16 +++-
 .../java/org/apache/drill/TestBugFixes.java |  2 +-
 .../apache/drill/exec/TestWindowFunctions.java  |  4 +-
 pom.xml |  2 +-
 18 files changed, 207 insertions(+), 94 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/drill/blob/d59f0cda/contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/HBaseSchemaFactory.java
--
diff --git 
a/contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/HBaseSchemaFactory.java
 
b/contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/HBaseSchemaFactory.java
index 548b679..810daaa 100644
--- 
a/contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/HBaseSchemaFactory.java
+++ 
b/contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/HBaseSchemaFactory.java
@@ -72,7 +72,16 @@ public class HBaseSchemaFactory implements SchemaFactory {
 @Override
 public Table getTable(String name) {
   HBaseScanSpec scanSpec = new HBaseScanSpec(name);
-  return new DrillHBaseTable(schemaName, plugin, scanSpec);
+  try {
+return new DrillHBaseTable(schemaName, plugin, scanSpec);
+  } catch (Exception e) {
+// Calcite firstly is looking for a table in the default schema, if a 
table was not found,
+// it is looking in root schema.
+// If a table does not exist, a query will fail at validation stage,
+// so the error should not be thrown there.
+logger.warn("Failure while loading table '{}' for database '{}'.", 
name, schemaName, e.getCause());
+return null;
+  }
 }
 
 @Override

http://git-wip-us.apache.org/repos/asf/drill/blob/d59f0cda/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/schema/HiveDatabaseSchema.java
--
diff --git 
a/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/schema/HiveDatabaseSchema.java
 
b/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/schema/HiveDatabaseSchema.java
index 90f30d8..ec1d0c6 100644
--- 
a/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/schema/HiveDatabaseSchema.java
+++ 
b/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/schema/HiveDatabaseSchema.java
@@ -1,4 +1,4 @@
-/**
+/*
  * Licensed to the Apache Software Foundation (ASF) under one
  * or more contributor license agreements.  See the NOTICE file
  * distributed with this work for additional information
@@ -17,14 +17,16 @@
  */
 package org.apache.drill.exec.store.hive.schema;
 
-import com.google.common.collect.ImmutableList;
 import com.google.common.collect.Lists;
 import com.google.common.collect.Sets;
+import org.apache.calcite.config.CalciteConnectionConfig;
 import org.apache.calcite.rel.type.RelDataType;
 import org.apache.calcite.rel.type.RelDataTypeFactory;
 import org.apache.calcite.schema.Schema;
 import org.apache.calcite.schema.Statistic;
 import

[17/18] drill git commit: DRILL-3993: Changes after review:

2018-01-18 Thread amansinha 
DRILL-3993: Changes after review:

Moved checks for OOM into spillIfNeeded method to avoid excessive call of 
chooseAPartitionToFlush method.
Ignored unit tests until DRILL-6018 is fixed.
These failures appears because new Calcite tries to simplify expressions from 
the query when applying ReduceExpressionsRule and fails with NFE.
Similar problem, but with old version of Calcite was described in DRILL-6018, 
therefore these unit tests are marked as ignored until DRILL-6018 is fixed.


Project: http://git-wip-us.apache.org/repos/asf/drill/repo
Commit: http://git-wip-us.apache.org/repos/asf/drill/commit/de3889ac
Tree: http://git-wip-us.apache.org/repos/asf/drill/tree/de3889ac
Diff: http://git-wip-us.apache.org/repos/asf/drill/diff/de3889ac

Branch: refs/heads/master
Commit: de3889ac1e6c1808b1f24b32039dd3c2fff658d7
Parents: 18a71a3
Author: Volodymyr Vysotskyi 
Authored: Tue Jan 16 15:38:17 2018 +0200
Committer: Volodymyr Vysotskyi 
Committed: Thu Jan 18 12:56:01 2018 +0200

--
 .../impl/aggregate/HashAggTemplate.java | 52 ++--
 .../fn/impl/TestMathFunctionsWithNanInf.java|  3 ++
 .../vector/complex/writer/TestJsonNanInf.java   |  3 ++
 3 files changed, 32 insertions(+), 26 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/drill/blob/de3889ac/exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggTemplate.java
--
diff --git 
a/exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggTemplate.java
 
b/exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggTemplate.java
index 4b43b22..2f181fe 100644
--- 
a/exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggTemplate.java
+++ 
b/exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggTemplate.java
@@ -1280,8 +1280,6 @@ public abstract class HashAggTemplate implements 
HashAggregator {
   logger.trace("Reserved memory runs short, trying to {} a partition and 
retry Hash Table put() again.",
 is1stPhase ? "early return" : "spill");
 
-  checkForSpillPossibility(currentPartition);
-
   doSpill(currentPartition); // spill to free some memory
 
   retrySameIndex = true;
@@ -1305,8 +1303,6 @@ public abstract class HashAggTemplate implements 
HashAggregator {
   long memDiff = allocator.getAllocatedMemory() - allocatedBeforeHTput;
   if ( memDiff > 0 ) { logger.warn("Leak: HashTable put() OOM left behind 
{} bytes allocated",memDiff); }
 
-  checkForSpillPossibility(currentPartition);
-
   doSpill(currentPartition); // spill to free some memory
 
   retrySameIndex = true;
@@ -1383,17 +1379,6 @@ public abstract class HashAggTemplate implements 
HashAggregator {
 }
   }
 
-  /**
-   * Checks that spill is possible, otherwise throws {@link 
OutOfMemoryException}.
-   *
-   * @param currentPartition the partition that hit the memory limit
-   */
-  private void checkForSpillPossibility(int currentPartition) {
-if (chooseAPartitionToFlush(currentPartition, true) < 0) {
-  throw new OutOfMemoryException(getOOMErrorMsg("AGGR"));
-}
-  }
-
   private void spillIfNeeded(int currentPartition) { 
spillIfNeeded(currentPartition, false);}
   private void doSpill(int currentPartition) { spillIfNeeded(currentPartition, 
true);}
   /**
@@ -1426,9 +1411,17 @@ public abstract class HashAggTemplate implements 
HashAggregator {
   // Pick a "victim" partition to spill or return
   int victimPartition = chooseAPartitionToFlush(currentPartition, 
forceSpill);
 
-  // In case no partition has more than one batch -- try and "push the 
limits"; maybe next
-  // time the spill could work.
-  if ( victimPartition < 0 ) { return; }
+  // In case no partition has more than one batch and
+  // non-forced spill -- try and "push the limits";
+  // maybe next time the spill could work.
+  if (victimPartition < 0) {
+// In the case of the forced spill, there is not enough memory to 
continue.
+// Throws OOM to avoid the infinite loop.
+if (forceSpill) {
+  throw new OutOfMemoryException(getOOMErrorMsg("AGGR"));
+}
+return;
+  }
 
   if ( is2ndPhase ) {
 long before = allocator.getAllocatedMemory();
@@ -1443,14 +1436,21 @@ public abstract class HashAggTemplate implements 
HashAggregator {
 boolean spillAgain = reserveOutgoingMemory == 0 || 
reserveValueBatchMemory == 0;
 // in some "edge" cases (e.g. testing), spilling one partition may not 
be enough
 if ( spillAgain || allocator.getAllocatedMemory() + maxMemoryNeeded > 
allocator.getLimit() ) {
-int victimPartition2 =

[05/18] drill git commit: REVERTED: DRILL-5089

2018-01-18 Thread amansinha 
REVERTED:  DRILL-5089

Dynamically load schema of storage plugin only when needed for every query


Project: http://git-wip-us.apache.org/repos/asf/drill/repo
Commit: http://git-wip-us.apache.org/repos/asf/drill/commit/450e6709
Tree: http://git-wip-us.apache.org/repos/asf/drill/tree/450e6709
Diff: http://git-wip-us.apache.org/repos/asf/drill/diff/450e6709

Branch: refs/heads/master
Commit: 450e67094eb6e9a6484d7f86c49b51c77a08d7b2
Parents: ef0fafe
Author: Roman Kulyk 
Authored: Thu Nov 30 16:19:12 2017 +
Committer: Volodymyr Vysotskyi 
Committed: Tue Jan 16 12:10:13 2018 +0200

--
 .../apache/drill/exec/ops/FragmentContext.java  |   9 +-
 .../org/apache/drill/exec/ops/QueryContext.java |  11 +-
 .../exec/planner/sql/DynamicRootSchema.java | 142 ---
 .../drill/exec/planner/sql/DynamicSchema.java   |  58 
 .../drill/exec/planner/sql/SqlConverter.java|   9 +-
 .../drill/exec/store/SchemaTreeProvider.java|  31 +---
 .../exec/store/StoragePluginRegistryImpl.java   |   3 -
 .../exec/store/dfs/FileSystemSchemaFactory.java |  23 +--
 .../exec/store/dfs/WorkspaceSchemaFactory.java  |  87 
 .../store/ischema/InfoSchemaBatchCreator.java   |   2 +-
 .../exec/store/mock/MockBreakageStorage.java|  47 --
 .../exec/store/mock/MockStorageEngine.java  |   7 +-
 .../exec/work/metadata/MetadataProvider.java|   2 +-
 .../drill/exec/physical/impl/TestSchema.java|  87 
 .../drill/test/ClusterFixtureBuilder.java   |   4 -
 .../drill/test/ClusterMockStorageFixture.java   |  51 ---
 16 files changed, 46 insertions(+), 527 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/drill/blob/450e6709/exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java
--
diff --git 
a/exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java 
b/exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java
index 210d0d4..736d550 100644
--- 
a/exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java
+++ 
b/exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java
@@ -230,12 +230,7 @@ public class FragmentContext extends BaseFragmentContext 
implements AutoCloseabl
 return context;
   }
 
-  /**
-   * This method is only used to construt InfoSchemaReader, it is for the 
reader to get full schema, so here we
-   * are going to return a fully initialized schema tree.
-   * @return root schema's plus
-   */
-  public SchemaPlus getFullRootSchema() {
+  public SchemaPlus getRootSchema() {
 if (queryContext == null) {
   fail(new UnsupportedOperationException("Schema tree can only be created 
in root fragment. " +
   "This is a non-root fragment."));
@@ -253,7 +248,7 @@ public class FragmentContext extends BaseFragmentContext 
implements AutoCloseabl
 .setIgnoreAuthErrors(isImpersonationEnabled)
 .build();
 
-return queryContext.getFullRootSchema(schemaConfig);
+return queryContext.getRootSchema(schemaConfig);
   }
 
   /**

http://git-wip-us.apache.org/repos/asf/drill/blob/450e6709/exec/java-exec/src/main/java/org/apache/drill/exec/ops/QueryContext.java
--
diff --git 
a/exec/java-exec/src/main/java/org/apache/drill/exec/ops/QueryContext.java 
b/exec/java-exec/src/main/java/org/apache/drill/exec/ops/QueryContext.java
index eb32bc6..8dbddbf 100644
--- a/exec/java-exec/src/main/java/org/apache/drill/exec/ops/QueryContext.java
+++ b/exec/java-exec/src/main/java/org/apache/drill/exec/ops/QueryContext.java
@@ -163,23 +163,14 @@ public class QueryContext implements AutoCloseable, 
OptimizerRulesContext, Schem
   }
 
   /**
-   *  Create and return a SchemaTree with given schemaConfig but some 
schemas (from storage plugins)
-   *  could be initialized later.
+   *  Create and return a SchemaTree with given schemaConfig.
* @param schemaConfig
* @return
*/
   public SchemaPlus getRootSchema(SchemaConfig schemaConfig) {
 return schemaTreeProvider.createRootSchema(schemaConfig);
   }
-  /**
-   *  Create and return a fully initialized SchemaTree with given 
schemaConfig.
-   * @param schemaConfig
-   * @return
-   */
 
-  public SchemaPlus getFullRootSchema(SchemaConfig schemaConfig) {
-return schemaTreeProvider.createFullRootSchema(schemaConfig);
-  }
   /**
* Get the user name of the user who issued the query that is managed by 
this QueryContext.
* @return

http://git-wip-us.apache.org/repos/asf/drill/blob/450e6709/exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DynamicRootSchema.java
--
diff --git

[09/18] drill git commit: DRILL-3993: Move Drill-specific commits 'CALCITE-628' and 'Drill-specific change: Add back AbstractConverter in RelSet.java' from Calcite into DRILL

2018-01-18 Thread amansinha 
DRILL-3993: Move Drill-specific commits 'CALCITE-628' and 'Drill-specific 
change: Add back AbstractConverter in RelSet.java' from Calcite into DRILL


Project: http://git-wip-us.apache.org/repos/asf/drill/repo
Commit: http://git-wip-us.apache.org/repos/asf/drill/commit/688e043b
Tree: http://git-wip-us.apache.org/repos/asf/drill/tree/688e043b
Diff: http://git-wip-us.apache.org/repos/asf/drill/diff/688e043b

Branch: refs/heads/master
Commit: 688e043b4546884d82422b266b419817d645acfc
Parents: d59f0cd
Author: Volodymyr Vysotskyi 
Authored: Wed Dec 13 14:53:47 2017 +0200
Committer: Volodymyr Vysotskyi 
Committed: Tue Jan 16 12:10:13 2018 +0200

--
 .../exec/store/jdbc/DrillJdbcRuleBase.java  |  4 +--
 .../exec/store/jdbc/JdbcStoragePlugin.java  |  2 +-
 .../planner/logical/DrillAggregateRule.java |  2 +-
 .../exec/planner/logical/DrillFilterRule.java   |  2 +-
 .../exec/planner/logical/DrillJoinRule.java |  4 +--
 .../exec/planner/logical/DrillLimitRule.java|  2 +-
 .../exec/planner/logical/DrillProjectRule.java  |  2 +-
 .../exec/planner/logical/DrillSortRule.java |  2 +-
 .../exec/planner/logical/DrillUnionAllRule.java |  2 +-
 .../exec/planner/logical/DrillWindowRule.java   |  2 +-
 .../drill/exec/planner/physical/Prel.java   | 29 
 .../drill/exec/planner/physical/Prule.java  |  2 +-
 .../exec/planner/physical/SortConvertPrule.java |  8 --
 .../planner/physical/SubsetTransformer.java |  2 +-
 14 files changed, 37 insertions(+), 28 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/drill/blob/688e043b/contrib/storage-jdbc/src/main/java/org/apache/drill/exec/store/jdbc/DrillJdbcRuleBase.java
--
diff --git 
a/contrib/storage-jdbc/src/main/java/org/apache/drill/exec/store/jdbc/DrillJdbcRuleBase.java
 
b/contrib/storage-jdbc/src/main/java/org/apache/drill/exec/store/jdbc/DrillJdbcRuleBase.java
index 963e80a..9d29a22 100644
--- 
a/contrib/storage-jdbc/src/main/java/org/apache/drill/exec/store/jdbc/DrillJdbcRuleBase.java
+++ 
b/contrib/storage-jdbc/src/main/java/org/apache/drill/exec/store/jdbc/DrillJdbcRuleBase.java
@@ -65,7 +65,7 @@ abstract class DrillJdbcRuleBase extends ConverterRule {
 public RelNode convert(RelNode rel) {
   LogicalProject project = (LogicalProject) rel;
   return new JdbcRules.JdbcProject(rel.getCluster(), 
rel.getTraitSet().replace(this.out), convert(
-  project.getInput(), 
project.getInput().getTraitSet().replace(this.out)), project.getProjects(),
+  project.getInput(), 
project.getInput().getTraitSet().replace(this.out).simplify()), 
project.getProjects(),
   project.getRowType());
 }
 
@@ -97,7 +97,7 @@ abstract class DrillJdbcRuleBase extends ConverterRule {
   LogicalFilter filter = (LogicalFilter) rel;
 
   return new JdbcRules.JdbcFilter(rel.getCluster(), 
rel.getTraitSet().replace(this.out), convert(filter.getInput(),
-  filter.getInput().getTraitSet().replace(this.out)), 
filter.getCondition());
+  filter.getInput().getTraitSet().replace(this.out).simplify()), 
filter.getCondition());
 }
 
 @Override

http://git-wip-us.apache.org/repos/asf/drill/blob/688e043b/contrib/storage-jdbc/src/main/java/org/apache/drill/exec/store/jdbc/JdbcStoragePlugin.java
--
diff --git 
a/contrib/storage-jdbc/src/main/java/org/apache/drill/exec/store/jdbc/JdbcStoragePlugin.java
 
b/contrib/storage-jdbc/src/main/java/org/apache/drill/exec/store/jdbc/JdbcStoragePlugin.java
index 4b41696..47f0f21 100755
--- 
a/contrib/storage-jdbc/src/main/java/org/apache/drill/exec/store/jdbc/JdbcStoragePlugin.java
+++ 
b/contrib/storage-jdbc/src/main/java/org/apache/drill/exec/store/jdbc/JdbcStoragePlugin.java
@@ -230,7 +230,7 @@ public class JdbcStoragePlugin extends 
AbstractStoragePlugin {
 @Override
 public RelNode convert(RelNode in) {
   return new JdbcDrel(in.getCluster(), 
in.getTraitSet().replace(DrillRel.DRILL_LOGICAL),
-  convert(in, in.getTraitSet().replace(this.getInTrait(;
+  convert(in, in.getTraitSet().replace(this.getInTrait()).simplify()));
 }
 
   }

http://git-wip-us.apache.org/repos/asf/drill/blob/688e043b/exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillAggregateRule.java
--
diff --git 
a/exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillAggregateRule.java
 
b/exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillAggregateRule.java
index 70f2215..2b998b2 100644
--- 
a/exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillAggregateRule.java
+++