[GitHub] drill pull request: DRILL-3688: Drill should honor "skip.header.li...

[arina-ielchiieva]

Github user arina-ielchiieva closed the pull request at:

https://github.com/apache/drill/pull/382


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (DRILL-4469) SUM window query returns incorrect results over integer data

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-4469:
-

 Summary: SUM window query returns incorrect results over integer 
data
 Key: DRILL-4469
 URL: https://issues.apache.org/jira/browse/DRILL-4469
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Flow
Affects Versions: 1.6.0
 Environment: 4 node CentOS cluster
Reporter: Khurram Faraaz
Priority: Critical


SUM window query returns incorrect results as compared to Postgres, with or 
without the frame clause in the window definition. Note that there is a sub 
query involved and data in column c1 is sorted integer data with no nulls.

Drill 1.6.0 commit ID: 6d5f4983

Results from Drill 1.6.0

{noformat}
0: jdbc:drill:schema=dfs.tmp> SELECT SUM(c1) OVER w FROM (select * from 
dfs.tmp.`t_alltype`) subQry WINDOW w AS (PARTITION BY c8 ORDER BY c1 RANGE 
BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING);
+-+
| EXPR$0  |
+-+
| 10585   |
| 10585   |
| 10585   |
| 10585   |
| 10585   |
| 10585   |
...
| 10585  |
| 10585  |
| 10585  |
++
145 rows selected (0.257 seconds)
{noformat}

results from Postgres 9.3

{noformat}
postgres=# SELECT SUM(c1) OVER w FROM (select * from t_alltype) subQry WINDOW w 
AS (PARTITION BY c8 ORDER BY c1 RANGE BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED 
FOLLOWING);
 sum
--
 4499
 4499
 4499
 4499
 4499
 4499
...
 5613
 5613
 5613
  473
  473
  473
  473
  473
(145 rows)
{noformat}

Removing the frame clause from window definition, still results in completely 
different results on Postgres vs Drill

Results from Drill 1.6.0

{noformat}
0: jdbc:drill:schema=dfs.tmp>SELECT SUM(c1) OVER w FROM (select * from 
t_alltype) subQry WINDOW w AS (PARTITION BY c8 ORDER BY c1);
+-+
| EXPR$0  |
+-+
| 10585   |
| 10585   |
| 10585   |
| 10585   |
| 10585   |
| 10585   |
| 10585   |
| 10585   |
| 10585   |
...
| 10585  |
| 10585  |
| 10585  |
| 10585  |
| 10585  |
++
145 rows selected (0.28 seconds)
{noformat}

Results from Postgres

{noformat}
postgres=# SELECT SUM(c1) OVER w FROM (select * from t_alltype) subQry WINDOW w 
AS (PARTITION BY c8 ORDER BY c1);
 sum
--
5
   12
   21
   33
   47
   62
   78
   96
  115
  135
  158
  182
  207
  233
  260
  289
...
4914
 5051
 5189
 5328
 5470
 5613
8
   70
  198
  332
  473
(145 rows)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-4470) TPC-H dataset 404

[Michael Mior (JIRA)]

Michael Mior created DRILL-4470:
---

 Summary: TPC-H dataset 404
 Key: DRILL-4470
 URL: https://issues.apache.org/jira/browse/DRILL-4470
 Project: Apache Drill
  Issue Type: Bug
Reporter: Michael Mior


The URL for the TPC-H sample data is returning 404 which breaks the build.

http://apache-drill.s3.amazonaws.com/files//sf-0.01_tpc-h_parquet_typed.tgz



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Apache drill on Android devices!

[Sandeep Choudhary]

Dear Team,

I am looking for running the Apache Drill on Android (linux + java based)
devices, these devices are 4-8 cores and 2-4 GB RAM + storage speed around
to SSD speed!

Is there any way to compile or run it?

I think it will be great for Apache Drill too as an advantage, there are
many No-SQL provider started supporting this but they are not good enough.

Looking for a positive response.

Best,
Sandeep Choudhary

[GitHub] drill pull request: DRILL-4465: Simplify Calcite parsing & plannin...

[jinfengni]

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/401#discussion_r54907078
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/DefaultSqlHandler.java
 ---
@@ -273,12 +282,90 @@ public RelNode visit(RelNode other) {
 
   }
 
+  /**
+   * Transform RelNode to a new RelNode without changing any traits. Also 
will log the outcome.
+   *
+   * @param plannerType
+   *  The type of Planner to use.
+   * @param phase
+   *  The transformation phase we're running.
+   * @param input
+   *  The origianl RelNode
+   * @return The transformed relnode.
+   */
+  private RelNode transform(PlannerType plannerType, PlannerPhase phase, 
RelNode input) {
+return transform(plannerType, phase, input, input.getTraitSet());
+  }
+
+  /**
+   * Transform RelNode to a new RelNode, targeting the provided set of 
traits. Also will log the outcome.
+   *
+   * @param plannerType
+   *  The type of Planner to use.
+   * @param phase
+   *  The transformation phase we're running.
+   * @param input
+   *  The origianl RelNode
+   * @param targetTraits
+   *  The traits we are targeting for output.
+   * @return The transformed relnode.
+   */
+  protected RelNode transform(PlannerType plannerType, PlannerPhase phase, 
RelNode input, RelTraitSet targetTraits) {
+final Stopwatch watch = Stopwatch.createStarted();
+final RuleSet rules = config.getRules(phase);
+final RelTraitSet toTraits = targetTraits.simplify();
+
+final RelNode output;
+switch (plannerType) {
+case HEP_BOTTOM_UP:
+case HEP: {
+  final HepProgramBuilder hepPgmBldr = new HepProgramBuilder();
+  if (plannerType == PlannerType.HEP_BOTTOM_UP) {
+hepPgmBldr.addMatchOrder(HepMatchOrder.BOTTOM_UP);
+  }
+  for (RelOptRule rule : rules) {
+hepPgmBldr.addRuleInstance(rule);
+  }
+
+  final HepPlanner planner = new HepPlanner(hepPgmBldr.build(), 
context.getPlannerSettings());
+
+  final List list = Lists.newArrayList();
+  list.add(DrillDefaultRelMetadataProvider.INSTANCE);
+  planner.registerMetadataProviders(list);
+  final RelMetadataProvider cachingMetaDataProvider = new 
CachingRelMetadataProvider(
+  ChainedRelMetadataProvider.of(list), planner);
+
+  // Modify RelMetaProvider for every RelNode in the SQL operator Rel 
tree.
+  input.accept(new MetaDataProviderModifier(cachingMetaDataProvider));
+  planner.setRoot(input);
+  if (!input.getTraitSet().equals(targetTraits)) {
--- End diff --

Okay. That makes sense. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4465: Simplify Calcite parsing & plannin...

[jinfengni]

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/401#discussion_r54907758
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/DefaultSqlHandler.java
 ---
@@ -273,12 +282,90 @@ public RelNode visit(RelNode other) {
 
   }
 
+  /**
+   * Transform RelNode to a new RelNode without changing any traits. Also 
will log the outcome.
+   *
+   * @param plannerType
+   *  The type of Planner to use.
+   * @param phase
+   *  The transformation phase we're running.
+   * @param input
+   *  The origianl RelNode
+   * @return The transformed relnode.
+   */
+  private RelNode transform(PlannerType plannerType, PlannerPhase phase, 
RelNode input) {
+return transform(plannerType, phase, input, input.getTraitSet());
+  }
+
+  /**
+   * Transform RelNode to a new RelNode, targeting the provided set of 
traits. Also will log the outcome.
+   *
+   * @param plannerType
+   *  The type of Planner to use.
+   * @param phase
+   *  The transformation phase we're running.
+   * @param input
+   *  The origianl RelNode
+   * @param targetTraits
+   *  The traits we are targeting for output.
+   * @return The transformed relnode.
+   */
+  protected RelNode transform(PlannerType plannerType, PlannerPhase phase, 
RelNode input, RelTraitSet targetTraits) {
+final Stopwatch watch = Stopwatch.createStarted();
+final RuleSet rules = config.getRules(phase);
+final RelTraitSet toTraits = targetTraits.simplify();
+
+final RelNode output;
+switch (plannerType) {
+case HEP_BOTTOM_UP:
+case HEP: {
+  final HepProgramBuilder hepPgmBldr = new HepProgramBuilder();
+  if (plannerType == PlannerType.HEP_BOTTOM_UP) {
+hepPgmBldr.addMatchOrder(HepMatchOrder.BOTTOM_UP);
+  }
+  for (RelOptRule rule : rules) {
+hepPgmBldr.addRuleInstance(rule);
+  }
+
+  final HepPlanner planner = new HepPlanner(hepPgmBldr.build(), 
context.getPlannerSettings());
+
+  final List list = Lists.newArrayList();
+  list.add(DrillDefaultRelMetadataProvider.INSTANCE);
+  planner.registerMetadataProviders(list);
+  final RelMetadataProvider cachingMetaDataProvider = new 
CachingRelMetadataProvider(
+  ChainedRelMetadataProvider.of(list), planner);
+
+  // Modify RelMetaProvider for every RelNode in the SQL operator Rel 
tree.
+  input.accept(new MetaDataProviderModifier(cachingMetaDataProvider));
+  planner.setRoot(input);
+  if (!input.getTraitSet().equals(targetTraits)) {
+planner.changeTraits(input, toTraits);
+  }
+  output = planner.findBestExp();
+  break;
+}
+case VOLCANO:
+default: {
+  // as weird as it seems, the cluster's only planner is the volcano 
planner.
+  final RelOptPlanner planner = input.getCluster().getPlanner();
+  final Program program = Programs.of(rules);
+  output = program.run(planner, input, toTraits);
+
+  break;
+}
+}
+
+log(plannerType.name() + ":" + phase.description, output, logger, 
watch);
--- End diff --

Sorry for the confusion. You are right that there is no impact when debug 
is disabled. The reason for the performance difference is IDE enables debug 
mode, which will cause the unit test to run longer. As long as debug is 
disabled, we would not see difference.  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4465: Simplify Calcite parsing & plannin...

[jinfengni]

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/401#discussion_r54908424
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/SqlHandlerConfig.java
 ---
@@ -18,30 +18,41 @@
 
 package org.apache.drill.exec.planner.sql.handlers;
 
-import org.apache.calcite.tools.Planner;
+import java.util.Collection;
+import java.util.Map.Entry;
+
+import org.apache.calcite.tools.RuleSet;
 import org.apache.drill.exec.ops.QueryContext;
-import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.drill.exec.planner.PlannerPhase;
+import org.apache.drill.exec.planner.sql.DrillSqlParser;
+import org.apache.drill.exec.store.StoragePlugin;
+
+import com.beust.jcommander.internal.Lists;
--- End diff --

Import a wrong "Lists" here? checkstyle complained this import.  



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4465: Simplify Calcite parsing & plannin...

[jinfengni]

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/401#discussion_r54912629
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DrillSqlParser.java
 ---
@@ -0,0 +1,349 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql;
+
+import java.util.Arrays;
+import java.util.List;
+
+import org.apache.calcite.adapter.java.JavaTypeFactory;
+import org.apache.calcite.avatica.util.Casing;
+import org.apache.calcite.avatica.util.Quoting;
+import org.apache.calcite.jdbc.CalciteSchemaImpl;
+import org.apache.calcite.jdbc.JavaTypeFactoryImpl;
+import org.apache.calcite.plan.ConventionTraitDef;
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelOptCostFactory;
+import org.apache.calcite.plan.RelOptTable;
+import org.apache.calcite.plan.volcano.VolcanoPlanner;
+import org.apache.calcite.prepare.CalciteCatalogReader;
+import org.apache.calcite.rel.RelCollationTraitDef;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rel.type.RelDataTypeSystemImpl;
+import org.apache.calcite.rex.RexBuilder;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperatorTable;
+import org.apache.calcite.sql.parser.SqlParseException;
+import org.apache.calcite.sql.parser.SqlParser;
+import org.apache.calcite.sql.parser.SqlParserImplFactory;
+import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.calcite.sql.util.ChainedSqlOperatorTable;
+import org.apache.calcite.sql.validate.SqlConformance;
+import org.apache.calcite.sql.validate.SqlValidatorCatalogReader;
+import org.apache.calcite.sql.validate.SqlValidatorImpl;
+import org.apache.calcite.sql2rel.RelDecorrelator;
+import org.apache.calcite.sql2rel.SqlToRelConverter;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.expr.fn.FunctionImplementationRegistry;
+import org.apache.drill.exec.ops.UdfUtilities;
+import org.apache.drill.exec.planner.cost.DrillCostBase;
+import org.apache.drill.exec.planner.logical.DrillConstExecutor;
+import org.apache.drill.exec.planner.physical.DrillDistributionTraitDef;
+import org.apache.drill.exec.planner.physical.PlannerSettings;
+import 
org.apache.drill.exec.planner.sql.parser.impl.DrillParserWithCompoundIdConverter;
+
+/**
+ * Class responsible for managing parsing, validation and toRel conversion 
for sql statements.
+ */
+public class DrillSqlParser {
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(DrillSqlParser.class);
+
+  private static DrillTypeSystem DRILL_TYPE_SYSTEM = new DrillTypeSystem();
+
+  private final JavaTypeFactory typeFactory;
+  private final SqlParser.Config parserConfig;
+  private final CalciteCatalogReader catalog;
+  private final PlannerSettings settings;
+  private final SchemaPlus rootSchema;
+  private final SchemaPlus defaultSchema;
+  private final SqlOperatorTable opTab;
+  private final RelOptCostFactory costFactory;
+  private final DrillValidator validator;
+  private final boolean isInnerQuery;
+  private final UdfUtilities util;
+  private final FunctionImplementationRegistry functions;
+
+  private String sql;
+  private VolcanoPlanner planner;
+
+
+  public DrillSqlParser(PlannerSettings settings, SchemaPlus defaultSchema,
+  final SqlOperatorTable operatorTable, UdfUtilities util, 
FunctionImplementationRegistry functions) {
+this.settings = settings;
+this.util = util;
+this.functions = functions;
+this.parserConfig = new ParserConfig();
+this.isInnerQuery = false;
+this.typeFactory 

[GitHub] drill pull request: DRILL-4465: Simplify Calcite parsing & plannin...

[jinfengni]

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/401#discussion_r54912350
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DrillSqlParser.java
 ---
@@ -0,0 +1,349 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql;
+
+import java.util.Arrays;
+import java.util.List;
+
+import org.apache.calcite.adapter.java.JavaTypeFactory;
+import org.apache.calcite.avatica.util.Casing;
+import org.apache.calcite.avatica.util.Quoting;
+import org.apache.calcite.jdbc.CalciteSchemaImpl;
+import org.apache.calcite.jdbc.JavaTypeFactoryImpl;
+import org.apache.calcite.plan.ConventionTraitDef;
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelOptCostFactory;
+import org.apache.calcite.plan.RelOptTable;
+import org.apache.calcite.plan.volcano.VolcanoPlanner;
+import org.apache.calcite.prepare.CalciteCatalogReader;
+import org.apache.calcite.rel.RelCollationTraitDef;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rel.type.RelDataTypeSystemImpl;
+import org.apache.calcite.rex.RexBuilder;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperatorTable;
+import org.apache.calcite.sql.parser.SqlParseException;
+import org.apache.calcite.sql.parser.SqlParser;
+import org.apache.calcite.sql.parser.SqlParserImplFactory;
+import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.calcite.sql.util.ChainedSqlOperatorTable;
+import org.apache.calcite.sql.validate.SqlConformance;
+import org.apache.calcite.sql.validate.SqlValidatorCatalogReader;
+import org.apache.calcite.sql.validate.SqlValidatorImpl;
+import org.apache.calcite.sql2rel.RelDecorrelator;
+import org.apache.calcite.sql2rel.SqlToRelConverter;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.expr.fn.FunctionImplementationRegistry;
+import org.apache.drill.exec.ops.UdfUtilities;
+import org.apache.drill.exec.planner.cost.DrillCostBase;
+import org.apache.drill.exec.planner.logical.DrillConstExecutor;
+import org.apache.drill.exec.planner.physical.DrillDistributionTraitDef;
+import org.apache.drill.exec.planner.physical.PlannerSettings;
+import 
org.apache.drill.exec.planner.sql.parser.impl.DrillParserWithCompoundIdConverter;
+
+/**
+ * Class responsible for managing parsing, validation and toRel conversion 
for sql statements.
+ */
+public class DrillSqlParser {
--- End diff --

SqlConverter seems fine to me. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: Drill 4372 review

[hsuanyi]

Github user hsuanyi commented on a diff in the pull request:

https://github.com/apache/drill/pull/397#discussion_r54915765
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillFunctionRegistry.java
 ---
@@ -92,38 +94,110 @@ public DrillFunctionRegistry(ScanResult classpathScan) 
{
   }
 
   public int size(){
-return methods.size();
+return registeredFunctions.size();
   }
 
   /** Returns functions with given name. Function name is case 
insensitive. */
   public List getMethods(String name) {
-return this.methods.get(name.toLowerCase());
+return this.registeredFunctions.get(name.toLowerCase());
+  }
+
+  public Collection getAllMethods() {
+return 
Collections.unmodifiableCollection(registeredFunctions.values());
   }
 
   public void register(DrillOperatorTable operatorTable) {
-SqlOperator op;
-for (Entry> function : 
methods.asMap().entrySet()) {
-  Set argCounts = Sets.newHashSet();
-  String name = function.getKey().toUpperCase();
+for (Entry> function : 
registeredFunctions.asMap().entrySet()) {
+  final ArrayListMultimap, DrillFuncHolder> 
functions = ArrayListMultimap.create();
+  final ArrayListMultimap aggregateFunctions 
= ArrayListMultimap.create();
+  final String name = function.getKey().toUpperCase();
+  boolean isDeterministic = true;
   for (DrillFuncHolder func : function.getValue()) {
-if (argCounts.add(func.getParamCount())) {
-  if (func.isAggregating()) {
-op = new DrillSqlAggOperator(name, func.getParamCount());
-  } else {
-boolean isDeterministic;
-// prevent Drill from folding constant functions with types 
that cannot be materialized
-// into literals
-if 
(DrillConstExecutor.NON_REDUCIBLE_TYPES.contains(func.getReturnType().getMinorType()))
 {
-  isDeterministic = false;
-} else {
-  isDeterministic = func.isDeterministic();
-}
-op = new DrillSqlOperator(name, func.getParamCount(), 
func.getReturnType(), isDeterministic);
-  }
-  operatorTable.add(function.getKey(), op);
+final int paramCount = func.getParamCount();
+if(func.isAggregating()) {
+  aggregateFunctions.put(paramCount, func);
+} else {
+  final Pair argNumerRange = 
getArgNumerRange(name, func);
+  functions.put(argNumerRange, func);
 }
+
+if(!func.isDeterministic()) {
+  isDeterministic = false;
+}
+  }
+  for (Entry, Collection> 
entry : functions.asMap().entrySet()) {
+final DrillSqlOperator drillSqlOperator;
+final Pair range = entry.getKey();
+final int max = range.getRight();
+final int min = range.getLeft();
+drillSqlOperator = new DrillSqlOperator(
+name,
+Lists.newArrayList(entry.getValue()),
+min,
+max,
+isDeterministic);
+operatorTable.add(name, drillSqlOperator);
+  }
+  for (Entry> entry : 
aggregateFunctions.asMap().entrySet()) {
+operatorTable.add(name, new DrillSqlAggOperator(name, 
Lists.newArrayList(entry.getValue()), entry.getKey()));
   }
 }
+
+registerCalcitePlaceHolderFunction(operatorTable);
+  }
+
+  /**
+   * These {@link DrillSqlOperator} merely act as a placeholder so that 
Calcite
+   * allows convert_to(), convert_from(), flatten(), date_part() functions 
in SQL.
+   */
+  private void registerCalcitePlaceHolderFunction(DrillOperatorTable 
operatorTable) {
+final String convert_to = "CONVERT_TO";
+final String convert_from = "CONVERT_FROM";
+final String flatten = "FLATTEN";
+final String date_part = "DATE_PART";
+
+operatorTable.add(convert_to,
+new DrillSqlOperator(convert_to,
+2,
+true));
+operatorTable.add(convert_from,
+new DrillSqlOperator(convert_from,
+2,
+true));
+operatorTable.add(flatten,
+new DrillSqlOperator(flatten,
+1,
+true));
+operatorTable.add(date_part,
+new DrillSqlOperator(date_part,
+2,
+true));
   }
 
+  private Pair getArgNumerRange(final String name, final 
DrillFuncHolder func) {
+switch(name.toUpperCase()) {
+  case "CONCAT":
+return Pair.of(1, Integer.MAX_VALUE);
+
+  // Drill does not have a FunctionTemplate for the lpad/rpad with two 
arguments.
+  // It relies on Dril

[GitHub] drill pull request: Drill 4372 review

[hsuanyi]

Github user hsuanyi commented on a diff in the pull request:

https://github.com/apache/drill/pull/397#discussion_r54915844
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/TypeInferenceUtils.java
 ---
@@ -0,0 +1,571 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql;
+
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+
+import org.apache.calcite.avatica.util.TimeUnit;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.sql.SqlCallBinding;
+import org.apache.calcite.sql.SqlCharStringLiteral;
+import org.apache.calcite.sql.SqlDynamicParam;
+import org.apache.calcite.sql.SqlLiteral;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperatorBinding;
+import org.apache.calcite.sql.type.SqlReturnTypeInference;
+import org.apache.calcite.sql.type.SqlTypeName;
+
+import org.apache.drill.common.expression.ExpressionPosition;
+import org.apache.drill.common.expression.FunctionCall;
+import org.apache.drill.common.expression.LogicalExpression;
+import org.apache.drill.common.expression.MajorTypeInLogicalExpression;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.expr.TypeHelper;
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+import org.apache.drill.exec.planner.logical.DrillConstExecutor;
+import org.apache.drill.exec.resolver.FunctionResolver;
+import org.apache.drill.exec.resolver.FunctionResolverFactory;
+
+import java.util.List;
+
+public class TypeInferenceUtils {
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(TypeInferenceUtils.class);
+
+  public static final TypeProtos.MajorType UNKNOWN_TYPE = 
TypeProtos.MajorType.getDefaultInstance();
+  private static ImmutableMap 
DRILL_TO_CALCITE_TYPE_MAPPING =
+  ImmutableMap. builder()
+  .put(TypeProtos.MinorType.INT, SqlTypeName.INTEGER)
+  .put(TypeProtos.MinorType.BIGINT, SqlTypeName.BIGINT)
+  .put(TypeProtos.MinorType.FLOAT4, SqlTypeName.FLOAT)
+  .put(TypeProtos.MinorType.FLOAT8, SqlTypeName.DOUBLE)
+  .put(TypeProtos.MinorType.VARCHAR, SqlTypeName.VARCHAR)
+  .put(TypeProtos.MinorType.BIT, SqlTypeName.BOOLEAN)
+  .put(TypeProtos.MinorType.DATE, SqlTypeName.DATE)
+  .put(TypeProtos.MinorType.DECIMAL9, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.DECIMAL18, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.DECIMAL28SPARSE, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.DECIMAL38SPARSE, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.TIME, SqlTypeName.TIME)
+  .put(TypeProtos.MinorType.TIMESTAMP, SqlTypeName.TIMESTAMP)
+  .put(TypeProtos.MinorType.VARBINARY, SqlTypeName.VARBINARY)
+  .put(TypeProtos.MinorType.INTERVALYEAR, 
SqlTypeName.INTERVAL_YEAR_MONTH)
+  .put(TypeProtos.MinorType.INTERVALDAY, 
SqlTypeName.INTERVAL_DAY_TIME)
+  .put(TypeProtos.MinorType.MAP, SqlTypeName.MAP)
+  .put(TypeProtos.MinorType.LIST, SqlTypeName.ARRAY)
+  .put(TypeProtos.MinorType.LATE, SqlTypeName.ANY)
+  // These are defined in the Drill type system but have been 
turned off for now
+  // .put(TypeProtos.MinorType.TINYINT, SqlTypeName.TINYINT)
+  // .put(TypeProtos.MinorType.SMALLINT, SqlTypeName.SMALLINT)
+  // Calcite types currently not supported by Drill, nor defined 
in the Drill type list:
+  //  - CHAR, SYMBOL, MULTISET, DISTINCT, STRUCTURED, ROW, 
OTHER, CURSOR, COLUMN_LIST
+  .build();
+
+  private static ImmutableMap 
CALCITE_TO_DRILL_MAPPING =
+  ImmutableMap. builder()

[GitHub] drill pull request: Drill 4372 review

[hsuanyi]

Github user hsuanyi commented on a diff in the pull request:

https://github.com/apache/drill/pull/397#discussion_r54916066
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/TypeInferenceUtils.java
 ---
@@ -0,0 +1,571 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql;
+
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+
+import org.apache.calcite.avatica.util.TimeUnit;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.sql.SqlCallBinding;
+import org.apache.calcite.sql.SqlCharStringLiteral;
+import org.apache.calcite.sql.SqlDynamicParam;
+import org.apache.calcite.sql.SqlLiteral;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperatorBinding;
+import org.apache.calcite.sql.type.SqlReturnTypeInference;
+import org.apache.calcite.sql.type.SqlTypeName;
+
+import org.apache.drill.common.expression.ExpressionPosition;
+import org.apache.drill.common.expression.FunctionCall;
+import org.apache.drill.common.expression.LogicalExpression;
+import org.apache.drill.common.expression.MajorTypeInLogicalExpression;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.expr.TypeHelper;
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+import org.apache.drill.exec.planner.logical.DrillConstExecutor;
+import org.apache.drill.exec.resolver.FunctionResolver;
+import org.apache.drill.exec.resolver.FunctionResolverFactory;
+
+import java.util.List;
+
+public class TypeInferenceUtils {
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(TypeInferenceUtils.class);
+
+  public static final TypeProtos.MajorType UNKNOWN_TYPE = 
TypeProtos.MajorType.getDefaultInstance();
+  private static ImmutableMap 
DRILL_TO_CALCITE_TYPE_MAPPING =
+  ImmutableMap. builder()
+  .put(TypeProtos.MinorType.INT, SqlTypeName.INTEGER)
+  .put(TypeProtos.MinorType.BIGINT, SqlTypeName.BIGINT)
+  .put(TypeProtos.MinorType.FLOAT4, SqlTypeName.FLOAT)
+  .put(TypeProtos.MinorType.FLOAT8, SqlTypeName.DOUBLE)
+  .put(TypeProtos.MinorType.VARCHAR, SqlTypeName.VARCHAR)
+  .put(TypeProtos.MinorType.BIT, SqlTypeName.BOOLEAN)
+  .put(TypeProtos.MinorType.DATE, SqlTypeName.DATE)
+  .put(TypeProtos.MinorType.DECIMAL9, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.DECIMAL18, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.DECIMAL28SPARSE, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.DECIMAL38SPARSE, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.TIME, SqlTypeName.TIME)
+  .put(TypeProtos.MinorType.TIMESTAMP, SqlTypeName.TIMESTAMP)
+  .put(TypeProtos.MinorType.VARBINARY, SqlTypeName.VARBINARY)
+  .put(TypeProtos.MinorType.INTERVALYEAR, 
SqlTypeName.INTERVAL_YEAR_MONTH)
+  .put(TypeProtos.MinorType.INTERVALDAY, 
SqlTypeName.INTERVAL_DAY_TIME)
+  .put(TypeProtos.MinorType.MAP, SqlTypeName.MAP)
+  .put(TypeProtos.MinorType.LIST, SqlTypeName.ARRAY)
+  .put(TypeProtos.MinorType.LATE, SqlTypeName.ANY)
+  // These are defined in the Drill type system but have been 
turned off for now
+  // .put(TypeProtos.MinorType.TINYINT, SqlTypeName.TINYINT)
+  // .put(TypeProtos.MinorType.SMALLINT, SqlTypeName.SMALLINT)
+  // Calcite types currently not supported by Drill, nor defined 
in the Drill type list:
+  //  - CHAR, SYMBOL, MULTISET, DISTINCT, STRUCTURED, ROW, 
OTHER, CURSOR, COLUMN_LIST
+  .build();
+
+  private static ImmutableMap 
CALCITE_TO_DRILL_MAPPING =
+  ImmutableMap. builder()

[GitHub] drill pull request: Drill 4372 review

[hsuanyi]

Github user hsuanyi commented on a diff in the pull request:

https://github.com/apache/drill/pull/397#discussion_r54916158
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/TypeInferenceUtils.java
 ---
@@ -0,0 +1,571 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql;
+
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+
+import org.apache.calcite.avatica.util.TimeUnit;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.sql.SqlCallBinding;
+import org.apache.calcite.sql.SqlCharStringLiteral;
+import org.apache.calcite.sql.SqlDynamicParam;
+import org.apache.calcite.sql.SqlLiteral;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperatorBinding;
+import org.apache.calcite.sql.type.SqlReturnTypeInference;
+import org.apache.calcite.sql.type.SqlTypeName;
+
+import org.apache.drill.common.expression.ExpressionPosition;
+import org.apache.drill.common.expression.FunctionCall;
+import org.apache.drill.common.expression.LogicalExpression;
+import org.apache.drill.common.expression.MajorTypeInLogicalExpression;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.expr.TypeHelper;
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+import org.apache.drill.exec.planner.logical.DrillConstExecutor;
+import org.apache.drill.exec.resolver.FunctionResolver;
+import org.apache.drill.exec.resolver.FunctionResolverFactory;
+
+import java.util.List;
+
+public class TypeInferenceUtils {
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(TypeInferenceUtils.class);
+
+  public static final TypeProtos.MajorType UNKNOWN_TYPE = 
TypeProtos.MajorType.getDefaultInstance();
+  private static ImmutableMap 
DRILL_TO_CALCITE_TYPE_MAPPING =
+  ImmutableMap. builder()
+  .put(TypeProtos.MinorType.INT, SqlTypeName.INTEGER)
+  .put(TypeProtos.MinorType.BIGINT, SqlTypeName.BIGINT)
+  .put(TypeProtos.MinorType.FLOAT4, SqlTypeName.FLOAT)
+  .put(TypeProtos.MinorType.FLOAT8, SqlTypeName.DOUBLE)
+  .put(TypeProtos.MinorType.VARCHAR, SqlTypeName.VARCHAR)
+  .put(TypeProtos.MinorType.BIT, SqlTypeName.BOOLEAN)
+  .put(TypeProtos.MinorType.DATE, SqlTypeName.DATE)
+  .put(TypeProtos.MinorType.DECIMAL9, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.DECIMAL18, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.DECIMAL28SPARSE, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.DECIMAL38SPARSE, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.TIME, SqlTypeName.TIME)
+  .put(TypeProtos.MinorType.TIMESTAMP, SqlTypeName.TIMESTAMP)
+  .put(TypeProtos.MinorType.VARBINARY, SqlTypeName.VARBINARY)
+  .put(TypeProtos.MinorType.INTERVALYEAR, 
SqlTypeName.INTERVAL_YEAR_MONTH)
+  .put(TypeProtos.MinorType.INTERVALDAY, 
SqlTypeName.INTERVAL_DAY_TIME)
+  .put(TypeProtos.MinorType.MAP, SqlTypeName.MAP)
+  .put(TypeProtos.MinorType.LIST, SqlTypeName.ARRAY)
+  .put(TypeProtos.MinorType.LATE, SqlTypeName.ANY)
+  // These are defined in the Drill type system but have been 
turned off for now
+  // .put(TypeProtos.MinorType.TINYINT, SqlTypeName.TINYINT)
+  // .put(TypeProtos.MinorType.SMALLINT, SqlTypeName.SMALLINT)
+  // Calcite types currently not supported by Drill, nor defined 
in the Drill type list:
+  //  - CHAR, SYMBOL, MULTISET, DISTINCT, STRUCTURED, ROW, 
OTHER, CURSOR, COLUMN_LIST
+  .build();
+
+  private static ImmutableMap 
CALCITE_TO_DRILL_MAPPING =
+  ImmutableMap. builder()

[GitHub] drill pull request: Drill 4372 review

[hsuanyi]

Github user hsuanyi commented on a diff in the pull request:

https://github.com/apache/drill/pull/397#discussion_r54916226
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/TypeInferenceUtils.java
 ---
@@ -0,0 +1,571 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql;
+
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+
+import org.apache.calcite.avatica.util.TimeUnit;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.sql.SqlCallBinding;
+import org.apache.calcite.sql.SqlCharStringLiteral;
+import org.apache.calcite.sql.SqlDynamicParam;
+import org.apache.calcite.sql.SqlLiteral;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperatorBinding;
+import org.apache.calcite.sql.type.SqlReturnTypeInference;
+import org.apache.calcite.sql.type.SqlTypeName;
+
+import org.apache.drill.common.expression.ExpressionPosition;
+import org.apache.drill.common.expression.FunctionCall;
+import org.apache.drill.common.expression.LogicalExpression;
+import org.apache.drill.common.expression.MajorTypeInLogicalExpression;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.expr.TypeHelper;
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+import org.apache.drill.exec.planner.logical.DrillConstExecutor;
+import org.apache.drill.exec.resolver.FunctionResolver;
+import org.apache.drill.exec.resolver.FunctionResolverFactory;
+
+import java.util.List;
+
+public class TypeInferenceUtils {
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(TypeInferenceUtils.class);
+
+  public static final TypeProtos.MajorType UNKNOWN_TYPE = 
TypeProtos.MajorType.getDefaultInstance();
+  private static ImmutableMap 
DRILL_TO_CALCITE_TYPE_MAPPING =
+  ImmutableMap. builder()
+  .put(TypeProtos.MinorType.INT, SqlTypeName.INTEGER)
+  .put(TypeProtos.MinorType.BIGINT, SqlTypeName.BIGINT)
+  .put(TypeProtos.MinorType.FLOAT4, SqlTypeName.FLOAT)
+  .put(TypeProtos.MinorType.FLOAT8, SqlTypeName.DOUBLE)
+  .put(TypeProtos.MinorType.VARCHAR, SqlTypeName.VARCHAR)
+  .put(TypeProtos.MinorType.BIT, SqlTypeName.BOOLEAN)
+  .put(TypeProtos.MinorType.DATE, SqlTypeName.DATE)
+  .put(TypeProtos.MinorType.DECIMAL9, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.DECIMAL18, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.DECIMAL28SPARSE, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.DECIMAL38SPARSE, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.TIME, SqlTypeName.TIME)
+  .put(TypeProtos.MinorType.TIMESTAMP, SqlTypeName.TIMESTAMP)
+  .put(TypeProtos.MinorType.VARBINARY, SqlTypeName.VARBINARY)
+  .put(TypeProtos.MinorType.INTERVALYEAR, 
SqlTypeName.INTERVAL_YEAR_MONTH)
+  .put(TypeProtos.MinorType.INTERVALDAY, 
SqlTypeName.INTERVAL_DAY_TIME)
+  .put(TypeProtos.MinorType.MAP, SqlTypeName.MAP)
+  .put(TypeProtos.MinorType.LIST, SqlTypeName.ARRAY)
+  .put(TypeProtos.MinorType.LATE, SqlTypeName.ANY)
+  // These are defined in the Drill type system but have been 
turned off for now
+  // .put(TypeProtos.MinorType.TINYINT, SqlTypeName.TINYINT)
+  // .put(TypeProtos.MinorType.SMALLINT, SqlTypeName.SMALLINT)
+  // Calcite types currently not supported by Drill, nor defined 
in the Drill type list:
+  //  - CHAR, SYMBOL, MULTISET, DISTINCT, STRUCTURED, ROW, 
OTHER, CURSOR, COLUMN_LIST
+  .build();
+
+  private static ImmutableMap 
CALCITE_TO_DRILL_MAPPING =
+  ImmutableMap. builder()

[GitHub] drill pull request: Drill 4372 review

[hsuanyi]

Github user hsuanyi commented on a diff in the pull request:

https://github.com/apache/drill/pull/397#discussion_r54916249
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DrillSqlOperator.java
 ---
@@ -18,69 +18,43 @@
 
 package org.apache.drill.exec.planner.sql;
 
-import com.google.common.base.Preconditions;
-import org.apache.drill.common.types.TypeProtos.MajorType;
-import org.apache.drill.common.types.TypeProtos.MinorType;
-import org.apache.calcite.rel.type.RelDataType;
-import org.apache.calcite.rel.type.RelDataTypeFactory;
-import org.apache.calcite.sql.SqlCall;
+import java.util.ArrayList;
+import java.util.List;
+
 import org.apache.calcite.sql.SqlFunction;
 import org.apache.calcite.sql.SqlFunctionCategory;
 import org.apache.calcite.sql.SqlIdentifier;
-import org.apache.calcite.sql.SqlOperatorBinding;
 import org.apache.calcite.sql.parser.SqlParserPos;
-import org.apache.calcite.sql.type.SqlTypeName;
-import org.apache.calcite.sql.validate.SqlValidator;
-import org.apache.calcite.sql.validate.SqlValidatorScope;
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
 
 public class DrillSqlOperator extends SqlFunction {
-  static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(DrillSqlOperator.class);
-
-  private static final MajorType NONE = MajorType.getDefaultInstance();
-  private final MajorType returnType;
+  // static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(DrillSqlOperator.class);
   private final boolean isDeterministic;
+  private final List functions;
 
   public DrillSqlOperator(String name, int argCount, boolean 
isDeterministic) {
-this(name, argCount, MajorType.getDefaultInstance(), isDeterministic);
+this(name, new ArrayList(), argCount, argCount, 
isDeterministic);
--- End diff --

Will file a jira


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4465: Simplify Calcite parsing & plannin...

[jacques-n]

Github user jacques-n commented on the pull request:

https://github.com/apache/drill/pull/401#issuecomment-191890837
  
@jinfengni: I've addressed your review comments. Let me know any additional 
feedback. 

thanks!



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: Drill 4372 review

[sudheeshkatkam]

Github user sudheeshkatkam commented on a diff in the pull request:

https://github.com/apache/drill/pull/397#discussion_r54922569
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/StringFunctions.java
 ---
@@ -770,6 +770,65 @@ public void eval() {
 } // end of eval
   }
 
+  /*
+   * Fill up the string to length 'length' by prepending the character ' ' 
in the beginning of 'text'.
+   * If the string is already longer than length, then it is truncated (on 
the right).
+   */
+  @FunctionTemplate(name = "lpad", scope = FunctionScope.SIMPLE, nulls = 
NullHandling.NULL_IF_NULL)
+  public static class LpadTwoArg implements DrillSimpleFunc {
--- End diff --

Please add unit test for these functions.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: Drill 4372 review

[sudheeshkatkam]

Github user sudheeshkatkam commented on a diff in the pull request:

https://github.com/apache/drill/pull/397#discussion_r54922555
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillFunctionRegistry.java
 ---
@@ -92,38 +112,58 @@ public DrillFunctionRegistry(ScanResult classpathScan) 
{
   }
 
   public int size(){
-return methods.size();
+return registeredFunctions.size();
   }
 
   /** Returns functions with given name. Function name is case 
insensitive. */
   public List getMethods(String name) {
-return this.methods.get(name.toLowerCase());
+return this.registeredFunctions.get(name.toLowerCase());
+  }
+
+  public Collection getAllMethods() {
+return 
Collections.unmodifiableCollection(registeredFunctions.values());
   }
 
   public void register(DrillOperatorTable operatorTable) {
-SqlOperator op;
-for (Entry> function : 
methods.asMap().entrySet()) {
-  Set argCounts = Sets.newHashSet();
-  String name = function.getKey().toUpperCase();
+for (Entry> function : 
registeredFunctions.asMap().entrySet()) {
+  final ArrayListMultimap, DrillFuncHolder> 
functions = ArrayListMultimap.create();
+  final ArrayListMultimap aggregateFunctions 
= ArrayListMultimap.create();
+  final String name = function.getKey().toUpperCase();
+  boolean isDeterministic = true;
   for (DrillFuncHolder func : function.getValue()) {
-if (argCounts.add(func.getParamCount())) {
-  if (func.isAggregating()) {
-op = new DrillSqlAggOperator(name, func.getParamCount());
+final int paramCount = func.getParamCount();
+if(func.isAggregating()) {
+  aggregateFunctions.put(paramCount, func);
+} else {
+  final Pair argNumerRange;
--- End diff --

argNumberRange


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: Drill 4372 review

[sudheeshkatkam]

Github user sudheeshkatkam commented on a diff in the pull request:

https://github.com/apache/drill/pull/397#discussion_r54922552
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillFunctionRegistry.java
 ---
@@ -92,38 +112,58 @@ public DrillFunctionRegistry(ScanResult classpathScan) 
{
   }
 
   public int size(){
-return methods.size();
+return registeredFunctions.size();
   }
 
   /** Returns functions with given name. Function name is case 
insensitive. */
   public List getMethods(String name) {
-return this.methods.get(name.toLowerCase());
+return this.registeredFunctions.get(name.toLowerCase());
+  }
+
+  public Collection getAllMethods() {
--- End diff --

Remove unused method (and FunctionImplementationRegistry#getAllMethods).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: Drill 4372 review

[sudheeshkatkam]

Github user sudheeshkatkam commented on a diff in the pull request:

https://github.com/apache/drill/pull/397#discussion_r54922538
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillFunctionRegistry.java
 ---
@@ -17,40 +17,60 @@
  */
 package org.apache.drill.exec.expr.fn;
 
-import java.util.Arrays;
 import java.util.Collection;
+import java.util.Collections;
 import java.util.HashMap;
-import java.util.HashSet;
 import java.util.List;
+import java.util.Map;
 import java.util.Map.Entry;
-import java.util.Set;
 
-import org.apache.calcite.sql.SqlOperator;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import org.apache.commons.lang3.tuple.Pair;
 import 
org.apache.drill.common.scanner.persistence.AnnotatedClassDescriptor;
 import org.apache.drill.common.scanner.persistence.ScanResult;
-import org.apache.drill.exec.expr.DrillFunc;
 import org.apache.drill.exec.planner.logical.DrillConstExecutor;
 import org.apache.drill.exec.planner.sql.DrillOperatorTable;
 import org.apache.drill.exec.planner.sql.DrillSqlAggOperator;
 import org.apache.drill.exec.planner.sql.DrillSqlOperator;
 
 import com.google.common.collect.ArrayListMultimap;
-import com.google.common.collect.Sets;
 
+/**
+ * Registry of Drill functions.
+ */
 public class DrillFunctionRegistry {
-  static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(DrillFunctionRegistry.class);
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(DrillFunctionRegistry.class);
 
-  private ArrayListMultimap methods = 
ArrayListMultimap.create();
+  // key: function name (lowercase) value: list of functions with that name
+  private final ArrayListMultimap 
registeredFunctions = ArrayListMultimap.create();
 
-  /* Hash map to prevent registering functions with exactly matching 
signatures
-   * key: Function Name + Input's Major Type
-   * Value: Class name where function is implemented
-   */
-  private HashMap functionSignatureMap = new HashMap<>();
+  private static final Map> 
drillFuncToRange = Maps.newHashMap();
--- End diff --

ImmutableMap?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: Drill 4372 review

[sudheeshkatkam]

Github user sudheeshkatkam commented on a diff in the pull request:

https://github.com/apache/drill/pull/397#discussion_r54922761
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillReduceAggregatesRule.java
 ---
@@ -100,8 +108,13 @@ public void onMatch(RelOptRuleCall ruleCall) {
*/
   private boolean containsAvgStddevVarCall(List 
aggCallList) {
 for (AggregateCall call : aggCallList) {
-  if (call.getAggregation() instanceof SqlAvgAggFunction
-  || call.getAggregation() instanceof SqlSumAggFunction) {
+  SqlAggFunction sqlAggFunction = call.getAggregation();
+  if(sqlAggFunction instanceof DrillCalciteSqlWrapper) {
--- End diff --

There are a bunch of unwrap calls like this. Is there a way to avoid this? 
Otherwise, devs need to aware of this unwrapping for *all* Calcite (SQL 
standard) functions.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: Drill 4372 review

[sudheeshkatkam]

Github user sudheeshkatkam commented on a diff in the pull request:

https://github.com/apache/drill/pull/397#discussion_r54922784
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/Checker.java ---
@@ -17,18 +17,51 @@
  */
 package org.apache.drill.exec.planner.sql;
 
+import com.google.common.collect.Maps;
 import org.apache.calcite.sql.SqlCallBinding;
 import org.apache.calcite.sql.SqlOperandCountRange;
 import org.apache.calcite.sql.SqlOperator;
+import org.apache.calcite.sql.type.SqlOperandCountRanges;
 import org.apache.calcite.sql.type.SqlOperandTypeChecker;
+import org.apache.commons.lang3.tuple.Pair;
+
+import java.util.Map;
 
 class Checker implements SqlOperandTypeChecker {
   private SqlOperandCountRange range;
 
-  public Checker(int size) {
+  public static Checker ANY_CHECKER = new Checker();
--- End diff --

final


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: Drill 4372 review

[sudheeshkatkam]

Github user sudheeshkatkam commented on a diff in the pull request:

https://github.com/apache/drill/pull/397#discussion_r54922840
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DrillCalciteSqlWrapper.java
 ---
@@ -0,0 +1,33 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql;
+
+import org.apache.calcite.sql.SqlOperator;
+/**
+ * This interface is meant for the users of the wrappers, {@link 
DrillCalciteSqlOperatorWrapper},
+ * {@link DrillCalciteSqlFunctionWrapper} and {@link 
DrillCalciteSqlAggFunctionWrapper}, to access the wrapped Calcite
+ * {@link SqlOperator} without knowing exactly which wrapper it is.
+ */
+public interface DrillCalciteSqlWrapper {
--- End diff --

There is redundant code in DrillCalciteSqlAggFunctionWrapper, 
DrillCalciteSqlFunctionWrapper and DrillCalciteSqlOperatorWrapper.

Can these extend from a base implementation which has common methods?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: Drill 4372 review

[sudheeshkatkam]

Github user sudheeshkatkam commented on a diff in the pull request:

https://github.com/apache/drill/pull/397#discussion_r54922890
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/TypeInferenceUtils.java
 ---
@@ -0,0 +1,568 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql;
+
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+
+import com.google.common.collect.Maps;
+import org.apache.calcite.avatica.util.TimeUnit;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.sql.SqlCallBinding;
+import org.apache.calcite.sql.SqlCharStringLiteral;
+import org.apache.calcite.sql.SqlDynamicParam;
+import org.apache.calcite.sql.SqlLiteral;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperatorBinding;
+import org.apache.calcite.sql.type.SqlReturnTypeInference;
+import org.apache.calcite.sql.type.SqlTypeName;
+
+import org.apache.drill.common.expression.ExpressionPosition;
+import org.apache.drill.common.expression.FunctionCall;
+import org.apache.drill.common.expression.LogicalExpression;
+import org.apache.drill.common.expression.MajorTypeInLogicalExpression;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.expr.TypeHelper;
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+import org.apache.drill.exec.planner.logical.DrillConstExecutor;
+import org.apache.drill.exec.resolver.FunctionResolver;
+import org.apache.drill.exec.resolver.FunctionResolverFactory;
+import org.apache.drill.exec.resolver.TypeCastRules;
+
+import java.util.List;
+import java.util.Map;
+
+public class TypeInferenceUtils {
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(TypeInferenceUtils.class);
+
+  public static final TypeProtos.MajorType UNKNOWN_TYPE = 
TypeProtos.MajorType.getDefaultInstance();
+  private static ImmutableMap 
DRILL_TO_CALCITE_TYPE_MAPPING =
+  ImmutableMap. builder()
+  .put(TypeProtos.MinorType.INT, SqlTypeName.INTEGER)
+  .put(TypeProtos.MinorType.BIGINT, SqlTypeName.BIGINT)
+  .put(TypeProtos.MinorType.FLOAT4, SqlTypeName.FLOAT)
+  .put(TypeProtos.MinorType.FLOAT8, SqlTypeName.DOUBLE)
+  .put(TypeProtos.MinorType.VARCHAR, SqlTypeName.VARCHAR)
+  .put(TypeProtos.MinorType.BIT, SqlTypeName.BOOLEAN)
+  .put(TypeProtos.MinorType.DATE, SqlTypeName.DATE)
+  .put(TypeProtos.MinorType.DECIMAL9, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.DECIMAL18, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.DECIMAL28SPARSE, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.DECIMAL38SPARSE, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.TIME, SqlTypeName.TIME)
+  .put(TypeProtos.MinorType.TIMESTAMP, SqlTypeName.TIMESTAMP)
+  .put(TypeProtos.MinorType.VARBINARY, SqlTypeName.VARBINARY)
+  .put(TypeProtos.MinorType.INTERVALYEAR, 
SqlTypeName.INTERVAL_YEAR_MONTH)
+  .put(TypeProtos.MinorType.INTERVALDAY, 
SqlTypeName.INTERVAL_DAY_TIME)
+  .put(TypeProtos.MinorType.MAP, SqlTypeName.MAP)
+  .put(TypeProtos.MinorType.LIST, SqlTypeName.ARRAY)
+  .put(TypeProtos.MinorType.LATE, SqlTypeName.ANY)
--- End diff --

I recollect there were more types here, albeit commented out. Can you add 
them back (including comments)?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
w

[GitHub] drill pull request: Drill 4372 review

[sudheeshkatkam]

Github user sudheeshkatkam commented on a diff in the pull request:

https://github.com/apache/drill/pull/397#discussion_r54923241
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/TypeInferenceUtils.java
 ---
@@ -0,0 +1,568 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql;
+
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+
+import com.google.common.collect.Maps;
+import org.apache.calcite.avatica.util.TimeUnit;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.sql.SqlCallBinding;
+import org.apache.calcite.sql.SqlCharStringLiteral;
+import org.apache.calcite.sql.SqlDynamicParam;
+import org.apache.calcite.sql.SqlLiteral;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperatorBinding;
+import org.apache.calcite.sql.type.SqlReturnTypeInference;
+import org.apache.calcite.sql.type.SqlTypeName;
+
+import org.apache.drill.common.expression.ExpressionPosition;
+import org.apache.drill.common.expression.FunctionCall;
+import org.apache.drill.common.expression.LogicalExpression;
+import org.apache.drill.common.expression.MajorTypeInLogicalExpression;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.expr.TypeHelper;
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+import org.apache.drill.exec.planner.logical.DrillConstExecutor;
+import org.apache.drill.exec.resolver.FunctionResolver;
+import org.apache.drill.exec.resolver.FunctionResolverFactory;
+import org.apache.drill.exec.resolver.TypeCastRules;
+
+import java.util.List;
+import java.util.Map;
+
+public class TypeInferenceUtils {
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(TypeInferenceUtils.class);
+
+  public static final TypeProtos.MajorType UNKNOWN_TYPE = 
TypeProtos.MajorType.getDefaultInstance();
+  private static ImmutableMap 
DRILL_TO_CALCITE_TYPE_MAPPING =
+  ImmutableMap. builder()
+  .put(TypeProtos.MinorType.INT, SqlTypeName.INTEGER)
+  .put(TypeProtos.MinorType.BIGINT, SqlTypeName.BIGINT)
+  .put(TypeProtos.MinorType.FLOAT4, SqlTypeName.FLOAT)
+  .put(TypeProtos.MinorType.FLOAT8, SqlTypeName.DOUBLE)
+  .put(TypeProtos.MinorType.VARCHAR, SqlTypeName.VARCHAR)
+  .put(TypeProtos.MinorType.BIT, SqlTypeName.BOOLEAN)
+  .put(TypeProtos.MinorType.DATE, SqlTypeName.DATE)
+  .put(TypeProtos.MinorType.DECIMAL9, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.DECIMAL18, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.DECIMAL28SPARSE, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.DECIMAL38SPARSE, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.TIME, SqlTypeName.TIME)
+  .put(TypeProtos.MinorType.TIMESTAMP, SqlTypeName.TIMESTAMP)
+  .put(TypeProtos.MinorType.VARBINARY, SqlTypeName.VARBINARY)
+  .put(TypeProtos.MinorType.INTERVALYEAR, 
SqlTypeName.INTERVAL_YEAR_MONTH)
+  .put(TypeProtos.MinorType.INTERVALDAY, 
SqlTypeName.INTERVAL_DAY_TIME)
+  .put(TypeProtos.MinorType.MAP, SqlTypeName.MAP)
+  .put(TypeProtos.MinorType.LIST, SqlTypeName.ARRAY)
+  .put(TypeProtos.MinorType.LATE, SqlTypeName.ANY)
+  .build();
+
+  private static ImmutableMap 
CALCITE_TO_DRILL_MAPPING =
+  ImmutableMap. builder()
+  .put(SqlTypeName.INTEGER, TypeProtos.MinorType.INT)
+  .put(SqlTypeName.BIGINT, TypeProtos.MinorType.BIGINT)
+  .put(SqlTypeName.FLOAT, TypeProtos.MinorType.FLOAT4)
+  .put(SqlTypeName.DOUBLE, TypeProtos.MinorType.FLOAT8)
+  .put(SqlTypeName.VARCHAR, T

[GitHub] drill pull request: Drill 4372 review

[sudheeshkatkam]

Github user sudheeshkatkam commented on a diff in the pull request:

https://github.com/apache/drill/pull/397#discussion_r54923222
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/TypeInferenceUtils.java
 ---
@@ -0,0 +1,568 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql;
+
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+
+import com.google.common.collect.Maps;
+import org.apache.calcite.avatica.util.TimeUnit;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.sql.SqlCallBinding;
+import org.apache.calcite.sql.SqlCharStringLiteral;
+import org.apache.calcite.sql.SqlDynamicParam;
+import org.apache.calcite.sql.SqlLiteral;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperatorBinding;
+import org.apache.calcite.sql.type.SqlReturnTypeInference;
+import org.apache.calcite.sql.type.SqlTypeName;
+
+import org.apache.drill.common.expression.ExpressionPosition;
+import org.apache.drill.common.expression.FunctionCall;
+import org.apache.drill.common.expression.LogicalExpression;
+import org.apache.drill.common.expression.MajorTypeInLogicalExpression;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.expr.TypeHelper;
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+import org.apache.drill.exec.planner.logical.DrillConstExecutor;
+import org.apache.drill.exec.resolver.FunctionResolver;
+import org.apache.drill.exec.resolver.FunctionResolverFactory;
+import org.apache.drill.exec.resolver.TypeCastRules;
+
+import java.util.List;
+import java.util.Map;
+
+public class TypeInferenceUtils {
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(TypeInferenceUtils.class);
+
+  public static final TypeProtos.MajorType UNKNOWN_TYPE = 
TypeProtos.MajorType.getDefaultInstance();
+  private static ImmutableMap 
DRILL_TO_CALCITE_TYPE_MAPPING =
--- End diff --

final


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Apache drill on Android devices!

[Jason Altekruse]

No one has tried to do this yet. The first question I would investigate is
if Android supports java direct memory, which Drill makes extensive use of,
but is not considered a standard feature in all implementations of the JVM.
A quick glance at the docs seems to indicate that it is supported (the
allocateDirect() method is what we are interested in here) [1]. I am not
aware of all of the differences between Android and standard Java so there
may be other major hurdles. It looks like Android now has Java 7 support,
which is the version that we currently develop Drill against.

You can certainly give it a shot, but I doubt it will just work. Drill has
a lot of dependencies, which I believe you would also have to re-compile.
That could end up being quite a task itself. If you decide to try it feel
free to ask questions here and we'll try to help out the best we can.

[1] - http://developer.android.com/reference/java/nio/ByteBuffer.html

- Jason

On Thu, Mar 3, 2016 at 6:16 AM, Sandeep Choudhary <
schoudh...@gofalconsmart.com> wrote:

> Dear Team,
>
> I am looking for running the Apache Drill on Android (linux + java based)
> devices, these devices are 4-8 cores and 2-4 GB RAM + storage speed around
> to SSD speed!
>
> Is there any way to compile or run it?
>
> I think it will be great for Apache Drill too as an advantage, there are
> many No-SQL provider started supporting this but they are not good enough.
>
> Looking for a positive response.
>
> Best,
> Sandeep Choudhary
>

[GitHub] drill pull request: Drill 4372 review

[sudheeshkatkam]

Github user sudheeshkatkam commented on a diff in the pull request:

https://github.com/apache/drill/pull/397#discussion_r54923299
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/TypeInferenceUtils.java
 ---
@@ -0,0 +1,568 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql;
+
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+
+import com.google.common.collect.Maps;
+import org.apache.calcite.avatica.util.TimeUnit;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.sql.SqlCallBinding;
+import org.apache.calcite.sql.SqlCharStringLiteral;
+import org.apache.calcite.sql.SqlDynamicParam;
+import org.apache.calcite.sql.SqlLiteral;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperatorBinding;
+import org.apache.calcite.sql.type.SqlReturnTypeInference;
+import org.apache.calcite.sql.type.SqlTypeName;
+
+import org.apache.drill.common.expression.ExpressionPosition;
+import org.apache.drill.common.expression.FunctionCall;
+import org.apache.drill.common.expression.LogicalExpression;
+import org.apache.drill.common.expression.MajorTypeInLogicalExpression;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.expr.TypeHelper;
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+import org.apache.drill.exec.planner.logical.DrillConstExecutor;
+import org.apache.drill.exec.resolver.FunctionResolver;
+import org.apache.drill.exec.resolver.FunctionResolverFactory;
+import org.apache.drill.exec.resolver.TypeCastRules;
+
+import java.util.List;
+import java.util.Map;
+
+public class TypeInferenceUtils {
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(TypeInferenceUtils.class);
+
+  public static final TypeProtos.MajorType UNKNOWN_TYPE = 
TypeProtos.MajorType.getDefaultInstance();
+  private static ImmutableMap 
DRILL_TO_CALCITE_TYPE_MAPPING =
+  ImmutableMap. builder()
+  .put(TypeProtos.MinorType.INT, SqlTypeName.INTEGER)
+  .put(TypeProtos.MinorType.BIGINT, SqlTypeName.BIGINT)
+  .put(TypeProtos.MinorType.FLOAT4, SqlTypeName.FLOAT)
+  .put(TypeProtos.MinorType.FLOAT8, SqlTypeName.DOUBLE)
+  .put(TypeProtos.MinorType.VARCHAR, SqlTypeName.VARCHAR)
+  .put(TypeProtos.MinorType.BIT, SqlTypeName.BOOLEAN)
+  .put(TypeProtos.MinorType.DATE, SqlTypeName.DATE)
+  .put(TypeProtos.MinorType.DECIMAL9, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.DECIMAL18, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.DECIMAL28SPARSE, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.DECIMAL38SPARSE, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.TIME, SqlTypeName.TIME)
+  .put(TypeProtos.MinorType.TIMESTAMP, SqlTypeName.TIMESTAMP)
+  .put(TypeProtos.MinorType.VARBINARY, SqlTypeName.VARBINARY)
+  .put(TypeProtos.MinorType.INTERVALYEAR, 
SqlTypeName.INTERVAL_YEAR_MONTH)
+  .put(TypeProtos.MinorType.INTERVALDAY, 
SqlTypeName.INTERVAL_DAY_TIME)
+  .put(TypeProtos.MinorType.MAP, SqlTypeName.MAP)
+  .put(TypeProtos.MinorType.LIST, SqlTypeName.ARRAY)
+  .put(TypeProtos.MinorType.LATE, SqlTypeName.ANY)
+  .build();
+
+  private static ImmutableMap 
CALCITE_TO_DRILL_MAPPING =
+  ImmutableMap. builder()
+  .put(SqlTypeName.INTEGER, TypeProtos.MinorType.INT)
+  .put(SqlTypeName.BIGINT, TypeProtos.MinorType.BIGINT)
+  .put(SqlTypeName.FLOAT, TypeProtos.MinorType.FLOAT4)
+  .put(SqlTypeName.DOUBLE, TypeProtos.MinorType.FLOAT8)
+  .put(SqlTypeName.VARCHAR, T

[GitHub] drill pull request: Drill 4372 review

[sudheeshkatkam]

Github user sudheeshkatkam commented on a diff in the pull request:

https://github.com/apache/drill/pull/397#discussion_r54923253
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/TypeInferenceUtils.java
 ---
@@ -0,0 +1,568 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql;
+
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+
+import com.google.common.collect.Maps;
+import org.apache.calcite.avatica.util.TimeUnit;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.sql.SqlCallBinding;
+import org.apache.calcite.sql.SqlCharStringLiteral;
+import org.apache.calcite.sql.SqlDynamicParam;
+import org.apache.calcite.sql.SqlLiteral;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperatorBinding;
+import org.apache.calcite.sql.type.SqlReturnTypeInference;
+import org.apache.calcite.sql.type.SqlTypeName;
+
+import org.apache.drill.common.expression.ExpressionPosition;
+import org.apache.drill.common.expression.FunctionCall;
+import org.apache.drill.common.expression.LogicalExpression;
+import org.apache.drill.common.expression.MajorTypeInLogicalExpression;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.expr.TypeHelper;
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+import org.apache.drill.exec.planner.logical.DrillConstExecutor;
+import org.apache.drill.exec.resolver.FunctionResolver;
+import org.apache.drill.exec.resolver.FunctionResolverFactory;
+import org.apache.drill.exec.resolver.TypeCastRules;
+
+import java.util.List;
+import java.util.Map;
+
+public class TypeInferenceUtils {
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(TypeInferenceUtils.class);
+
+  public static final TypeProtos.MajorType UNKNOWN_TYPE = 
TypeProtos.MajorType.getDefaultInstance();
+  private static ImmutableMap 
DRILL_TO_CALCITE_TYPE_MAPPING =
+  ImmutableMap. builder()
+  .put(TypeProtos.MinorType.INT, SqlTypeName.INTEGER)
+  .put(TypeProtos.MinorType.BIGINT, SqlTypeName.BIGINT)
+  .put(TypeProtos.MinorType.FLOAT4, SqlTypeName.FLOAT)
+  .put(TypeProtos.MinorType.FLOAT8, SqlTypeName.DOUBLE)
+  .put(TypeProtos.MinorType.VARCHAR, SqlTypeName.VARCHAR)
+  .put(TypeProtos.MinorType.BIT, SqlTypeName.BOOLEAN)
+  .put(TypeProtos.MinorType.DATE, SqlTypeName.DATE)
+  .put(TypeProtos.MinorType.DECIMAL9, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.DECIMAL18, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.DECIMAL28SPARSE, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.DECIMAL38SPARSE, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.TIME, SqlTypeName.TIME)
+  .put(TypeProtos.MinorType.TIMESTAMP, SqlTypeName.TIMESTAMP)
+  .put(TypeProtos.MinorType.VARBINARY, SqlTypeName.VARBINARY)
+  .put(TypeProtos.MinorType.INTERVALYEAR, 
SqlTypeName.INTERVAL_YEAR_MONTH)
+  .put(TypeProtos.MinorType.INTERVALDAY, 
SqlTypeName.INTERVAL_DAY_TIME)
+  .put(TypeProtos.MinorType.MAP, SqlTypeName.MAP)
+  .put(TypeProtos.MinorType.LIST, SqlTypeName.ARRAY)
+  .put(TypeProtos.MinorType.LATE, SqlTypeName.ANY)
+  .build();
+
+  private static ImmutableMap 
CALCITE_TO_DRILL_MAPPING =
+  ImmutableMap. builder()
+  .put(SqlTypeName.INTEGER, TypeProtos.MinorType.INT)
+  .put(SqlTypeName.BIGINT, TypeProtos.MinorType.BIGINT)
+  .put(SqlTypeName.FLOAT, TypeProtos.MinorType.FLOAT4)
+  .put(SqlTypeName.DOUBLE, TypeProtos.MinorType.FLOAT8)
+  .put(SqlTypeName.VARCHAR, T

[GitHub] drill pull request: Drill 4372 review

[sudheeshkatkam]

Github user sudheeshkatkam commented on a diff in the pull request:

https://github.com/apache/drill/pull/397#discussion_r54923238
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/TypeInferenceUtils.java
 ---
@@ -0,0 +1,568 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql;
+
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+
+import com.google.common.collect.Maps;
+import org.apache.calcite.avatica.util.TimeUnit;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.sql.SqlCallBinding;
+import org.apache.calcite.sql.SqlCharStringLiteral;
+import org.apache.calcite.sql.SqlDynamicParam;
+import org.apache.calcite.sql.SqlLiteral;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperatorBinding;
+import org.apache.calcite.sql.type.SqlReturnTypeInference;
+import org.apache.calcite.sql.type.SqlTypeName;
+
+import org.apache.drill.common.expression.ExpressionPosition;
+import org.apache.drill.common.expression.FunctionCall;
+import org.apache.drill.common.expression.LogicalExpression;
+import org.apache.drill.common.expression.MajorTypeInLogicalExpression;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.expr.TypeHelper;
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+import org.apache.drill.exec.planner.logical.DrillConstExecutor;
+import org.apache.drill.exec.resolver.FunctionResolver;
+import org.apache.drill.exec.resolver.FunctionResolverFactory;
+import org.apache.drill.exec.resolver.TypeCastRules;
+
+import java.util.List;
+import java.util.Map;
+
+public class TypeInferenceUtils {
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(TypeInferenceUtils.class);
+
+  public static final TypeProtos.MajorType UNKNOWN_TYPE = 
TypeProtos.MajorType.getDefaultInstance();
+  private static ImmutableMap 
DRILL_TO_CALCITE_TYPE_MAPPING =
+  ImmutableMap. builder()
+  .put(TypeProtos.MinorType.INT, SqlTypeName.INTEGER)
+  .put(TypeProtos.MinorType.BIGINT, SqlTypeName.BIGINT)
+  .put(TypeProtos.MinorType.FLOAT4, SqlTypeName.FLOAT)
+  .put(TypeProtos.MinorType.FLOAT8, SqlTypeName.DOUBLE)
+  .put(TypeProtos.MinorType.VARCHAR, SqlTypeName.VARCHAR)
+  .put(TypeProtos.MinorType.BIT, SqlTypeName.BOOLEAN)
+  .put(TypeProtos.MinorType.DATE, SqlTypeName.DATE)
+  .put(TypeProtos.MinorType.DECIMAL9, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.DECIMAL18, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.DECIMAL28SPARSE, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.DECIMAL38SPARSE, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.TIME, SqlTypeName.TIME)
+  .put(TypeProtos.MinorType.TIMESTAMP, SqlTypeName.TIMESTAMP)
+  .put(TypeProtos.MinorType.VARBINARY, SqlTypeName.VARBINARY)
+  .put(TypeProtos.MinorType.INTERVALYEAR, 
SqlTypeName.INTERVAL_YEAR_MONTH)
+  .put(TypeProtos.MinorType.INTERVALDAY, 
SqlTypeName.INTERVAL_DAY_TIME)
+  .put(TypeProtos.MinorType.MAP, SqlTypeName.MAP)
+  .put(TypeProtos.MinorType.LIST, SqlTypeName.ARRAY)
+  .put(TypeProtos.MinorType.LATE, SqlTypeName.ANY)
+  .build();
+
+  private static ImmutableMap 
CALCITE_TO_DRILL_MAPPING =
--- End diff --

final


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INF

[jira] [Created] (DRILL-4471) Add unit test for the Drill Web UI

[Jason Altekruse (JIRA)]

Jason Altekruse created DRILL-4471:
--

 Summary: Add unit test for the Drill Web UI
 Key: DRILL-4471
 URL: https://issues.apache.org/jira/browse/DRILL-4471
 Project: Apache Drill
  Issue Type: Test
Reporter: Jason Altekruse
Assignee: Jason Altekruse


While the Web UI isn't being very actively developed, a few times changes to 
the Drill build or internal parts of the server have broken parts of the Web UI.

As the web UI is a primary interface for viewing cluster information, 
cancelling queries, configuring storage and other tasks, we really should add 
automated tests for it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] drill pull request: Drill 4372 review

[sudheeshkatkam]

Github user sudheeshkatkam commented on a diff in the pull request:

https://github.com/apache/drill/pull/397#discussion_r54924830
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/TypeInferenceUtils.java
 ---
@@ -0,0 +1,568 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql;
+
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+
+import com.google.common.collect.Maps;
+import org.apache.calcite.avatica.util.TimeUnit;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.sql.SqlCallBinding;
+import org.apache.calcite.sql.SqlCharStringLiteral;
+import org.apache.calcite.sql.SqlDynamicParam;
+import org.apache.calcite.sql.SqlLiteral;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperatorBinding;
+import org.apache.calcite.sql.type.SqlReturnTypeInference;
+import org.apache.calcite.sql.type.SqlTypeName;
+
+import org.apache.drill.common.expression.ExpressionPosition;
+import org.apache.drill.common.expression.FunctionCall;
+import org.apache.drill.common.expression.LogicalExpression;
+import org.apache.drill.common.expression.MajorTypeInLogicalExpression;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.expr.TypeHelper;
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+import org.apache.drill.exec.planner.logical.DrillConstExecutor;
+import org.apache.drill.exec.resolver.FunctionResolver;
+import org.apache.drill.exec.resolver.FunctionResolverFactory;
+import org.apache.drill.exec.resolver.TypeCastRules;
+
+import java.util.List;
+import java.util.Map;
+
+public class TypeInferenceUtils {
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(TypeInferenceUtils.class);
+
+  public static final TypeProtos.MajorType UNKNOWN_TYPE = 
TypeProtos.MajorType.getDefaultInstance();
+  private static ImmutableMap 
DRILL_TO_CALCITE_TYPE_MAPPING =
+  ImmutableMap. builder()
+  .put(TypeProtos.MinorType.INT, SqlTypeName.INTEGER)
+  .put(TypeProtos.MinorType.BIGINT, SqlTypeName.BIGINT)
+  .put(TypeProtos.MinorType.FLOAT4, SqlTypeName.FLOAT)
+  .put(TypeProtos.MinorType.FLOAT8, SqlTypeName.DOUBLE)
+  .put(TypeProtos.MinorType.VARCHAR, SqlTypeName.VARCHAR)
+  .put(TypeProtos.MinorType.BIT, SqlTypeName.BOOLEAN)
+  .put(TypeProtos.MinorType.DATE, SqlTypeName.DATE)
+  .put(TypeProtos.MinorType.DECIMAL9, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.DECIMAL18, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.DECIMAL28SPARSE, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.DECIMAL38SPARSE, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.TIME, SqlTypeName.TIME)
+  .put(TypeProtos.MinorType.TIMESTAMP, SqlTypeName.TIMESTAMP)
+  .put(TypeProtos.MinorType.VARBINARY, SqlTypeName.VARBINARY)
+  .put(TypeProtos.MinorType.INTERVALYEAR, 
SqlTypeName.INTERVAL_YEAR_MONTH)
+  .put(TypeProtos.MinorType.INTERVALDAY, 
SqlTypeName.INTERVAL_DAY_TIME)
+  .put(TypeProtos.MinorType.MAP, SqlTypeName.MAP)
+  .put(TypeProtos.MinorType.LIST, SqlTypeName.ARRAY)
+  .put(TypeProtos.MinorType.LATE, SqlTypeName.ANY)
+  .build();
+
+  private static ImmutableMap 
CALCITE_TO_DRILL_MAPPING =
+  ImmutableMap. builder()
+  .put(SqlTypeName.INTEGER, TypeProtos.MinorType.INT)
+  .put(SqlTypeName.BIGINT, TypeProtos.MinorType.BIGINT)
+  .put(SqlTypeName.FLOAT, TypeProtos.MinorType.FLOAT4)
+  .put(SqlTypeName.DOUBLE, TypeProtos.MinorType.FLOAT8)
+  .put(SqlTypeName.VARCHAR, T

[GitHub] drill pull request: Drill 4372 review

[sudheeshkatkam]

Github user sudheeshkatkam commented on a diff in the pull request:

https://github.com/apache/drill/pull/397#discussion_r54925324
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/TypeInferenceUtils.java
 ---
@@ -0,0 +1,568 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql;
+
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+
+import com.google.common.collect.Maps;
+import org.apache.calcite.avatica.util.TimeUnit;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.sql.SqlCallBinding;
+import org.apache.calcite.sql.SqlCharStringLiteral;
+import org.apache.calcite.sql.SqlDynamicParam;
+import org.apache.calcite.sql.SqlLiteral;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperatorBinding;
+import org.apache.calcite.sql.type.SqlReturnTypeInference;
+import org.apache.calcite.sql.type.SqlTypeName;
+
+import org.apache.drill.common.expression.ExpressionPosition;
+import org.apache.drill.common.expression.FunctionCall;
+import org.apache.drill.common.expression.LogicalExpression;
+import org.apache.drill.common.expression.MajorTypeInLogicalExpression;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.expr.TypeHelper;
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+import org.apache.drill.exec.planner.logical.DrillConstExecutor;
+import org.apache.drill.exec.resolver.FunctionResolver;
+import org.apache.drill.exec.resolver.FunctionResolverFactory;
+import org.apache.drill.exec.resolver.TypeCastRules;
+
+import java.util.List;
+import java.util.Map;
+
+public class TypeInferenceUtils {
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(TypeInferenceUtils.class);
+
+  public static final TypeProtos.MajorType UNKNOWN_TYPE = 
TypeProtos.MajorType.getDefaultInstance();
+  private static ImmutableMap 
DRILL_TO_CALCITE_TYPE_MAPPING =
+  ImmutableMap. builder()
+  .put(TypeProtos.MinorType.INT, SqlTypeName.INTEGER)
+  .put(TypeProtos.MinorType.BIGINT, SqlTypeName.BIGINT)
+  .put(TypeProtos.MinorType.FLOAT4, SqlTypeName.FLOAT)
+  .put(TypeProtos.MinorType.FLOAT8, SqlTypeName.DOUBLE)
+  .put(TypeProtos.MinorType.VARCHAR, SqlTypeName.VARCHAR)
+  .put(TypeProtos.MinorType.BIT, SqlTypeName.BOOLEAN)
+  .put(TypeProtos.MinorType.DATE, SqlTypeName.DATE)
+  .put(TypeProtos.MinorType.DECIMAL9, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.DECIMAL18, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.DECIMAL28SPARSE, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.DECIMAL38SPARSE, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.TIME, SqlTypeName.TIME)
+  .put(TypeProtos.MinorType.TIMESTAMP, SqlTypeName.TIMESTAMP)
+  .put(TypeProtos.MinorType.VARBINARY, SqlTypeName.VARBINARY)
+  .put(TypeProtos.MinorType.INTERVALYEAR, 
SqlTypeName.INTERVAL_YEAR_MONTH)
+  .put(TypeProtos.MinorType.INTERVALDAY, 
SqlTypeName.INTERVAL_DAY_TIME)
+  .put(TypeProtos.MinorType.MAP, SqlTypeName.MAP)
+  .put(TypeProtos.MinorType.LIST, SqlTypeName.ARRAY)
+  .put(TypeProtos.MinorType.LATE, SqlTypeName.ANY)
+  .build();
+
+  private static ImmutableMap 
CALCITE_TO_DRILL_MAPPING =
+  ImmutableMap. builder()
+  .put(SqlTypeName.INTEGER, TypeProtos.MinorType.INT)
+  .put(SqlTypeName.BIGINT, TypeProtos.MinorType.BIGINT)
+  .put(SqlTypeName.FLOAT, TypeProtos.MinorType.FLOAT4)
+  .put(SqlTypeName.DOUBLE, TypeProtos.MinorType.FLOAT8)
+  .put(SqlTypeName.VARCHAR, T

[GitHub] drill pull request: Drill 4372 review

[sudheeshkatkam]

Github user sudheeshkatkam commented on a diff in the pull request:

https://github.com/apache/drill/pull/397#discussion_r54926134
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillFunctionRegistry.java
 ---
@@ -17,40 +17,60 @@
  */
 package org.apache.drill.exec.expr.fn;
 
-import java.util.Arrays;
 import java.util.Collection;
+import java.util.Collections;
 import java.util.HashMap;
-import java.util.HashSet;
 import java.util.List;
+import java.util.Map;
 import java.util.Map.Entry;
-import java.util.Set;
 
-import org.apache.calcite.sql.SqlOperator;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import org.apache.commons.lang3.tuple.Pair;
 import 
org.apache.drill.common.scanner.persistence.AnnotatedClassDescriptor;
 import org.apache.drill.common.scanner.persistence.ScanResult;
-import org.apache.drill.exec.expr.DrillFunc;
 import org.apache.drill.exec.planner.logical.DrillConstExecutor;
 import org.apache.drill.exec.planner.sql.DrillOperatorTable;
 import org.apache.drill.exec.planner.sql.DrillSqlAggOperator;
 import org.apache.drill.exec.planner.sql.DrillSqlOperator;
 
 import com.google.common.collect.ArrayListMultimap;
-import com.google.common.collect.Sets;
 
+/**
+ * Registry of Drill functions.
+ */
 public class DrillFunctionRegistry {
-  static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(DrillFunctionRegistry.class);
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(DrillFunctionRegistry.class);
 
-  private ArrayListMultimap methods = 
ArrayListMultimap.create();
+  // key: function name (lowercase) value: list of functions with that name
+  private final ArrayListMultimap 
registeredFunctions = ArrayListMultimap.create();
 
-  /* Hash map to prevent registering functions with exactly matching 
signatures
-   * key: Function Name + Input's Major Type
-   * Value: Class name where function is implemented
-   */
-  private HashMap functionSignatureMap = new HashMap<>();
+  private static final Map> 
drillFuncToRange = Maps.newHashMap();
+  static {
+// CONCAT is allowed to take [1, infinity) number of arguments.
+// Currently, this flexibility is offered by DrillOptiq to rewrite it 
as
+// a nested structure
+drillFuncToRange.put("CONCAT", Pair.of(1, Integer.MAX_VALUE));
+
+// When LENGTH is given two arguments, this function relies on 
DrillOptiq to rewrite it as
+// another function based on the second argument (encodingType)
+drillFuncToRange.put("LENGTH", Pair.of(1, 2));
+
+// Dummy functions
--- End diff --

Are these rewritten too?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4281: Support authorized users to delega...

[sudheeshkatkam]

Github user sudheeshkatkam commented on a diff in the pull request:

https://github.com/apache/drill/pull/400#discussion_r54926507
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java ---
@@ -281,4 +283,12 @@
*/
   String ADMIN_USER_GROUPS_KEY = "security.admin.user_groups";
   StringValidator ADMIN_USER_GROUPS_VALIDATOR = new 
AdminOptionValidator(ADMIN_USER_GROUPS_KEY, "");
+
+  /**
+   * Option whose value is the group delegation definitions.
--- End diff --

need to correct


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: Drill 4372 review

[sudheeshkatkam]

Github user sudheeshkatkam commented on the pull request:

https://github.com/apache/drill/pull/397#issuecomment-191911233
  
Also, commit says I am the author ;)

[caused due to squashing]


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Apache drill on Android devices!

[Hanifi GUNES]

Interesting. Wondering if this is a fun project or one has some serious
intentions to build a Drill cluster of droids. Either way, good luck with
Dalvik and unsafe. NoSql on Android has certain advantages and use cases
but wondering how Drill as a query engine would come handy. Thoughts?


-Hanifi

2016-03-03 10:18 GMT-08:00 Jason Altekruse :

> No one has tried to do this yet. The first question I would investigate is
> if Android supports java direct memory, which Drill makes extensive use of,
> but is not considered a standard feature in all implementations of the JVM.
> A quick glance at the docs seems to indicate that it is supported (the
> allocateDirect() method is what we are interested in here) [1]. I am not
> aware of all of the differences between Android and standard Java so there
> may be other major hurdles. It looks like Android now has Java 7 support,
> which is the version that we currently develop Drill against.
>
> You can certainly give it a shot, but I doubt it will just work. Drill has
> a lot of dependencies, which I believe you would also have to re-compile.
> That could end up being quite a task itself. If you decide to try it feel
> free to ask questions here and we'll try to help out the best we can.
>
> [1] - http://developer.android.com/reference/java/nio/ByteBuffer.html
>
> - Jason
>
> On Thu, Mar 3, 2016 at 6:16 AM, Sandeep Choudhary <
> schoudh...@gofalconsmart.com> wrote:
>
> > Dear Team,
> >
> > I am looking for running the Apache Drill on Android (linux + java based)
> > devices, these devices are 4-8 cores and 2-4 GB RAM + storage speed
> around
> > to SSD speed!
> >
> > Is there any way to compile or run it?
> >
> > I think it will be great for Apache Drill too as an advantage, there are
> > many No-SQL provider started supporting this but they are not good
> enough.
> >
> > Looking for a positive response.
> >
> > Best,
> > Sandeep Choudhary
> >
>

[GitHub] drill pull request: DRILL-4281: Support authorized users to delega...

[jacques-n]

Github user jacques-n commented on the pull request:

https://github.com/apache/drill/pull/400#issuecomment-191963668
  
Are delegate and delegator common used terms for these things? Frankly, I 
would have to look these up to confirm which is which and could see making a 
mistake reversing them. Any way we can make them something with clearer 
directionality? (If everybody else thinks that this distinction and 
directionality is super clear, nevermind.)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-3623: For limit 0 queries, use a shorter...

[sudheeshkatkam]

GitHub user sudheeshkatkam opened a pull request:

https://github.com/apache/drill/pull/405

DRILL-3623: For limit 0 queries, use a shorter path when result column 
types are known

Moving from #193 to here.

+ There is a pull request open for first commit (DRILL-4372: #397).
+ Second commit has a "nice to have" check: ensuring planning and execution 
types patch.
+ My changes are in the third commit (e4cfdfa). Please review this.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sudheeshkatkam/drill DRILL-3623-pr

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/405.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #405


commit c553365e39947ba6c95d645cc971cf4d696ee758
Author: Sudheesh Katkam 
Date:   2015-12-22T04:38:59Z

DRILL-4372: Expose the functions return type to Drill

- Drill-Calite version update:
This commit needs to have Calcite's patch (CALCITE-1062) to plugin 
customized SqlOperator.

- FunctionTemplate
Add FunctionArgumentNumber annotation. This annotation element tells if the 
number of argument(s) is fixed or arbitrary (e.g., String concatenation 
function).

Due to this modification, there are some minor changes in DrillFuncHolder, 
DrillFunctionRegistry and FunctionAttributes.

- Checker
Add a new Checker (which Calcite uses to validate the legitimacy of the 
number of argument(s) for a function) to allow functions with arbitrary 
arguments to pass Caclite's validation

- Type conversion between Drill and Calcite
DrillConstExector is given a static method getDrillTypeFromCalcite() to 
convert Calcite types to Drill's.

- Extract function's return type inference
Unlike other functions, Extract function's return type can be determined 
solely based on the first argument. A logic is added in to allow this inference 
to happen

- DrillCalcite wrapper:
From the aspects of return type inference and argument type checks, 
Calcite's mechanism is very different from Drill's. In addition, currently, 
there is no straightforward way for Drill to plug-in customized mechanisms to 
Calcite. Thus, wrappers are provided to serve the objective.

Except for the mechanisms of type inference and argument type checks, these 
wrappers just forward any method calls to the wrapped SqlOpertor, SqlFuncion or 
SqlAggFunction to respond.

A interface DrillCalciteSqlWrapper is also added for the callers of the 
three wrappers to get the wrapped objects easier.

Due to these wrappers, UnsupportedOperatorsVisitor is modified in a minor 
manner.

- Calcite's SqlOpertor, SqlFuncion or SqlAggFunction are wrapped in 
DrillOperatorTable
Instead of returning Caclite's native SqlOpertor, SqlFuncion or 
SqlAggFunction, return the wrapped ones to ensure customized behaviors can be 
adopted.

- Type inference mechanism
This mechanism is used across all SqlOpertor, SqlFuncion or SqlAggFunction. 
Thus, it is factored out as its own method in TypeInferenceUtils

- Upgrade Drill-Calcite

Bump version number to 1.4.0-drill-test-r16

- Implement two argument version of lpad, rpad

- Implement one argument version of ltrim, rtrim, btrim

commit c3f0649e3ebb45d54e747f099d6699150bfa9869
Author: Hsuan-Yi Chu 
Date:   2016-02-03T05:17:50Z

DRILL-4372: Part 2: Optionally ensure planning and execution types match

commit e4cfdfa9b0562d52ac07f6d80860a82fa8baba40
Author: Sudheesh Katkam 
Date:   2016-03-03T21:25:39Z

DRILL-3623: For limit 0 queries, use a shorter path when result column 
types are known




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-3623: Use shorter query path for LIMIT 0...

[sudheeshkatkam]

Github user sudheeshkatkam closed the pull request at:

https://github.com/apache/drill/pull/193


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-3623: Use shorter query path for LIMIT 0...

[sudheeshkatkam]

Github user sudheeshkatkam commented on the pull request:

https://github.com/apache/drill/pull/193#issuecomment-191973058
  
Moving to #405.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: Drill 4372 review

[hsuanyi]

Github user hsuanyi commented on a diff in the pull request:

https://github.com/apache/drill/pull/397#discussion_r54952793
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillFunctionRegistry.java
 ---
@@ -92,38 +112,58 @@ public DrillFunctionRegistry(ScanResult classpathScan) 
{
   }
 
   public int size(){
-return methods.size();
+return registeredFunctions.size();
   }
 
   /** Returns functions with given name. Function name is case 
insensitive. */
   public List getMethods(String name) {
-return this.methods.get(name.toLowerCase());
+return this.registeredFunctions.get(name.toLowerCase());
+  }
+
+  public Collection getAllMethods() {
+return 
Collections.unmodifiableCollection(registeredFunctions.values());
   }
 
   public void register(DrillOperatorTable operatorTable) {
-SqlOperator op;
-for (Entry> function : 
methods.asMap().entrySet()) {
-  Set argCounts = Sets.newHashSet();
-  String name = function.getKey().toUpperCase();
+for (Entry> function : 
registeredFunctions.asMap().entrySet()) {
+  final ArrayListMultimap, DrillFuncHolder> 
functions = ArrayListMultimap.create();
+  final ArrayListMultimap aggregateFunctions 
= ArrayListMultimap.create();
+  final String name = function.getKey().toUpperCase();
+  boolean isDeterministic = true;
   for (DrillFuncHolder func : function.getValue()) {
-if (argCounts.add(func.getParamCount())) {
-  if (func.isAggregating()) {
-op = new DrillSqlAggOperator(name, func.getParamCount());
+final int paramCount = func.getParamCount();
+if(func.isAggregating()) {
+  aggregateFunctions.put(paramCount, func);
+} else {
+  final Pair argNumerRange;
--- End diff --

addressed


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: Drill 4372 review

[hsuanyi]

Github user hsuanyi commented on a diff in the pull request:

https://github.com/apache/drill/pull/397#discussion_r54952798
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillFunctionRegistry.java
 ---
@@ -17,40 +17,60 @@
  */
 package org.apache.drill.exec.expr.fn;
 
-import java.util.Arrays;
 import java.util.Collection;
+import java.util.Collections;
 import java.util.HashMap;
-import java.util.HashSet;
 import java.util.List;
+import java.util.Map;
 import java.util.Map.Entry;
-import java.util.Set;
 
-import org.apache.calcite.sql.SqlOperator;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import org.apache.commons.lang3.tuple.Pair;
 import 
org.apache.drill.common.scanner.persistence.AnnotatedClassDescriptor;
 import org.apache.drill.common.scanner.persistence.ScanResult;
-import org.apache.drill.exec.expr.DrillFunc;
 import org.apache.drill.exec.planner.logical.DrillConstExecutor;
 import org.apache.drill.exec.planner.sql.DrillOperatorTable;
 import org.apache.drill.exec.planner.sql.DrillSqlAggOperator;
 import org.apache.drill.exec.planner.sql.DrillSqlOperator;
 
 import com.google.common.collect.ArrayListMultimap;
-import com.google.common.collect.Sets;
 
+/**
+ * Registry of Drill functions.
+ */
 public class DrillFunctionRegistry {
-  static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(DrillFunctionRegistry.class);
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(DrillFunctionRegistry.class);
 
-  private ArrayListMultimap methods = 
ArrayListMultimap.create();
+  // key: function name (lowercase) value: list of functions with that name
+  private final ArrayListMultimap 
registeredFunctions = ArrayListMultimap.create();
 
-  /* Hash map to prevent registering functions with exactly matching 
signatures
-   * key: Function Name + Input's Major Type
-   * Value: Class name where function is implemented
-   */
-  private HashMap functionSignatureMap = new HashMap<>();
+  private static final Map> 
drillFuncToRange = Maps.newHashMap();
--- End diff --

addressed


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: Drill 4372 review

[hsuanyi]

Github user hsuanyi commented on a diff in the pull request:

https://github.com/apache/drill/pull/397#discussion_r54952796
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillFunctionRegistry.java
 ---
@@ -92,38 +112,58 @@ public DrillFunctionRegistry(ScanResult classpathScan) 
{
   }
 
   public int size(){
-return methods.size();
+return registeredFunctions.size();
   }
 
   /** Returns functions with given name. Function name is case 
insensitive. */
   public List getMethods(String name) {
-return this.methods.get(name.toLowerCase());
+return this.registeredFunctions.get(name.toLowerCase());
+  }
+
+  public Collection getAllMethods() {
--- End diff --

addressed


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: Drill 4372 review

[hsuanyi]

Github user hsuanyi commented on a diff in the pull request:

https://github.com/apache/drill/pull/397#discussion_r54954482
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/StringFunctions.java
 ---
@@ -770,6 +770,65 @@ public void eval() {
 } // end of eval
   }
 
+  /*
+   * Fill up the string to length 'length' by prepending the character ' ' 
in the beginning of 'text'.
+   * If the string is already longer than length, then it is truncated (on 
the right).
+   */
+  @FunctionTemplate(name = "lpad", scope = FunctionScope.SIMPLE, nulls = 
NullHandling.NULL_IF_NULL)
+  public static class LpadTwoArg implements DrillSimpleFunc {
--- End diff --

I already have some in TestFunctionsWithTypeExpoQueries
I added more to TestStringFunctions


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: Drill 4372 review

[hsuanyi]

Github user hsuanyi commented on a diff in the pull request:

https://github.com/apache/drill/pull/397#discussion_r54954517
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/Checker.java ---
@@ -17,18 +17,51 @@
  */
 package org.apache.drill.exec.planner.sql;
 
+import com.google.common.collect.Maps;
 import org.apache.calcite.sql.SqlCallBinding;
 import org.apache.calcite.sql.SqlOperandCountRange;
 import org.apache.calcite.sql.SqlOperator;
+import org.apache.calcite.sql.type.SqlOperandCountRanges;
 import org.apache.calcite.sql.type.SqlOperandTypeChecker;
+import org.apache.commons.lang3.tuple.Pair;
+
+import java.util.Map;
 
 class Checker implements SqlOperandTypeChecker {
   private SqlOperandCountRange range;
 
-  public Checker(int size) {
+  public static Checker ANY_CHECKER = new Checker();
--- End diff --

addressed


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: Drill 4372 review

[hsuanyi]

Github user hsuanyi commented on a diff in the pull request:

https://github.com/apache/drill/pull/397#discussion_r54955290
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/TypeInferenceUtils.java
 ---
@@ -0,0 +1,568 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql;
+
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+
+import com.google.common.collect.Maps;
+import org.apache.calcite.avatica.util.TimeUnit;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.sql.SqlCallBinding;
+import org.apache.calcite.sql.SqlCharStringLiteral;
+import org.apache.calcite.sql.SqlDynamicParam;
+import org.apache.calcite.sql.SqlLiteral;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperatorBinding;
+import org.apache.calcite.sql.type.SqlReturnTypeInference;
+import org.apache.calcite.sql.type.SqlTypeName;
+
+import org.apache.drill.common.expression.ExpressionPosition;
+import org.apache.drill.common.expression.FunctionCall;
+import org.apache.drill.common.expression.LogicalExpression;
+import org.apache.drill.common.expression.MajorTypeInLogicalExpression;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.expr.TypeHelper;
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+import org.apache.drill.exec.planner.logical.DrillConstExecutor;
+import org.apache.drill.exec.resolver.FunctionResolver;
+import org.apache.drill.exec.resolver.FunctionResolverFactory;
+import org.apache.drill.exec.resolver.TypeCastRules;
+
+import java.util.List;
+import java.util.Map;
+
+public class TypeInferenceUtils {
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(TypeInferenceUtils.class);
+
+  public static final TypeProtos.MajorType UNKNOWN_TYPE = 
TypeProtos.MajorType.getDefaultInstance();
+  private static ImmutableMap 
DRILL_TO_CALCITE_TYPE_MAPPING =
--- End diff --

addressed


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: Drill 4372 review

[hsuanyi]

Github user hsuanyi commented on a diff in the pull request:

https://github.com/apache/drill/pull/397#discussion_r54955232
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DrillCalciteSqlWrapper.java
 ---
@@ -0,0 +1,33 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql;
+
+import org.apache.calcite.sql.SqlOperator;
+/**
+ * This interface is meant for the users of the wrappers, {@link 
DrillCalciteSqlOperatorWrapper},
+ * {@link DrillCalciteSqlFunctionWrapper} and {@link 
DrillCalciteSqlAggFunctionWrapper}, to access the wrapped Calcite
+ * {@link SqlOperator} without knowing exactly which wrapper it is.
+ */
+public interface DrillCalciteSqlWrapper {
--- End diff --

This is unfortunately not possible. Take DrillCalciteSqlAggFunctionWrapper 
as an example:

In order to let Calcite believe this is a SqlAggFunction, we have to claim 
DrillCalciteSqlAggFunctionWrapper is "extended" from SqlAggFunction. Not being 
able to do multi-inherentence in JAVA prevents code sharing from another base 
implementation


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: Drill 4372 review

[hsuanyi]

Github user hsuanyi commented on a diff in the pull request:

https://github.com/apache/drill/pull/397#discussion_r54955354
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/TypeInferenceUtils.java
 ---
@@ -0,0 +1,568 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql;
+
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+
+import com.google.common.collect.Maps;
+import org.apache.calcite.avatica.util.TimeUnit;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.sql.SqlCallBinding;
+import org.apache.calcite.sql.SqlCharStringLiteral;
+import org.apache.calcite.sql.SqlDynamicParam;
+import org.apache.calcite.sql.SqlLiteral;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperatorBinding;
+import org.apache.calcite.sql.type.SqlReturnTypeInference;
+import org.apache.calcite.sql.type.SqlTypeName;
+
+import org.apache.drill.common.expression.ExpressionPosition;
+import org.apache.drill.common.expression.FunctionCall;
+import org.apache.drill.common.expression.LogicalExpression;
+import org.apache.drill.common.expression.MajorTypeInLogicalExpression;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.expr.TypeHelper;
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+import org.apache.drill.exec.planner.logical.DrillConstExecutor;
+import org.apache.drill.exec.resolver.FunctionResolver;
+import org.apache.drill.exec.resolver.FunctionResolverFactory;
+import org.apache.drill.exec.resolver.TypeCastRules;
+
+import java.util.List;
+import java.util.Map;
+
+public class TypeInferenceUtils {
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(TypeInferenceUtils.class);
+
+  public static final TypeProtos.MajorType UNKNOWN_TYPE = 
TypeProtos.MajorType.getDefaultInstance();
+  private static ImmutableMap 
DRILL_TO_CALCITE_TYPE_MAPPING =
+  ImmutableMap. builder()
+  .put(TypeProtos.MinorType.INT, SqlTypeName.INTEGER)
+  .put(TypeProtos.MinorType.BIGINT, SqlTypeName.BIGINT)
+  .put(TypeProtos.MinorType.FLOAT4, SqlTypeName.FLOAT)
+  .put(TypeProtos.MinorType.FLOAT8, SqlTypeName.DOUBLE)
+  .put(TypeProtos.MinorType.VARCHAR, SqlTypeName.VARCHAR)
+  .put(TypeProtos.MinorType.BIT, SqlTypeName.BOOLEAN)
+  .put(TypeProtos.MinorType.DATE, SqlTypeName.DATE)
+  .put(TypeProtos.MinorType.DECIMAL9, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.DECIMAL18, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.DECIMAL28SPARSE, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.DECIMAL38SPARSE, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.TIME, SqlTypeName.TIME)
+  .put(TypeProtos.MinorType.TIMESTAMP, SqlTypeName.TIMESTAMP)
+  .put(TypeProtos.MinorType.VARBINARY, SqlTypeName.VARBINARY)
+  .put(TypeProtos.MinorType.INTERVALYEAR, 
SqlTypeName.INTERVAL_YEAR_MONTH)
+  .put(TypeProtos.MinorType.INTERVALDAY, 
SqlTypeName.INTERVAL_DAY_TIME)
+  .put(TypeProtos.MinorType.MAP, SqlTypeName.MAP)
+  .put(TypeProtos.MinorType.LIST, SqlTypeName.ARRAY)
+  .put(TypeProtos.MinorType.LATE, SqlTypeName.ANY)
+  .build();
+
+  private static ImmutableMap 
CALCITE_TO_DRILL_MAPPING =
--- End diff --

addressed


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.

[GitHub] drill pull request: Drill 4372 review

[hsuanyi]

Github user hsuanyi commented on a diff in the pull request:

https://github.com/apache/drill/pull/397#discussion_r54956169
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillReduceAggregatesRule.java
 ---
@@ -100,8 +108,13 @@ public void onMatch(RelOptRuleCall ruleCall) {
*/
   private boolean containsAvgStddevVarCall(List 
aggCallList) {
 for (AggregateCall call : aggCallList) {
-  if (call.getAggregation() instanceof SqlAvgAggFunction
-  || call.getAggregation() instanceof SqlSumAggFunction) {
+  SqlAggFunction sqlAggFunction = call.getAggregation();
+  if(sqlAggFunction instanceof DrillCalciteSqlWrapper) {
--- End diff --

Yeah,... I see the point. But this is necessary to still have wrapper here. 

For example, the type of avg(integer_type_col) in Calcite is defined as 
Integer. Thus, say, if the wrapper is removed somehow before this rule is 
fired, Calcite will complain the type it sees here (Integer) does not match 
with that (Double) we inferred before.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: Drill 4372 review

[hsuanyi]

Github user hsuanyi commented on a diff in the pull request:

https://github.com/apache/drill/pull/397#discussion_r54956465
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillReduceAggregatesRule.java
 ---
@@ -100,8 +108,13 @@ public void onMatch(RelOptRuleCall ruleCall) {
*/
   private boolean containsAvgStddevVarCall(List 
aggCallList) {
 for (AggregateCall call : aggCallList) {
-  if (call.getAggregation() instanceof SqlAvgAggFunction
-  || call.getAggregation() instanceof SqlSumAggFunction) {
+  SqlAggFunction sqlAggFunction = call.getAggregation();
+  if(sqlAggFunction instanceof DrillCalciteSqlWrapper) {
--- End diff --

The exception you might see is like this one:

Error: SYSTEM ERROR: AssertionError: Internal error: Conversion to 
relational algebra failed to preserve datatypes:
validated type:
RecordType(DOUBLE NOT NULL EXPR$0) NOT NULL
converted type:
RecordType(INTEGER NOT NULL EXPR$0) NOT NULL


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: Drill 4372 review

[hsuanyi]

Github user hsuanyi commented on a diff in the pull request:

https://github.com/apache/drill/pull/397#discussion_r54957475
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/TypeInferenceUtils.java
 ---
@@ -0,0 +1,568 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql;
+
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+
+import com.google.common.collect.Maps;
+import org.apache.calcite.avatica.util.TimeUnit;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.sql.SqlCallBinding;
+import org.apache.calcite.sql.SqlCharStringLiteral;
+import org.apache.calcite.sql.SqlDynamicParam;
+import org.apache.calcite.sql.SqlLiteral;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperatorBinding;
+import org.apache.calcite.sql.type.SqlReturnTypeInference;
+import org.apache.calcite.sql.type.SqlTypeName;
+
+import org.apache.drill.common.expression.ExpressionPosition;
+import org.apache.drill.common.expression.FunctionCall;
+import org.apache.drill.common.expression.LogicalExpression;
+import org.apache.drill.common.expression.MajorTypeInLogicalExpression;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.expr.TypeHelper;
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+import org.apache.drill.exec.planner.logical.DrillConstExecutor;
+import org.apache.drill.exec.resolver.FunctionResolver;
+import org.apache.drill.exec.resolver.FunctionResolverFactory;
+import org.apache.drill.exec.resolver.TypeCastRules;
+
+import java.util.List;
+import java.util.Map;
+
+public class TypeInferenceUtils {
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(TypeInferenceUtils.class);
+
+  public static final TypeProtos.MajorType UNKNOWN_TYPE = 
TypeProtos.MajorType.getDefaultInstance();
+  private static ImmutableMap 
DRILL_TO_CALCITE_TYPE_MAPPING =
+  ImmutableMap. builder()
+  .put(TypeProtos.MinorType.INT, SqlTypeName.INTEGER)
+  .put(TypeProtos.MinorType.BIGINT, SqlTypeName.BIGINT)
+  .put(TypeProtos.MinorType.FLOAT4, SqlTypeName.FLOAT)
+  .put(TypeProtos.MinorType.FLOAT8, SqlTypeName.DOUBLE)
+  .put(TypeProtos.MinorType.VARCHAR, SqlTypeName.VARCHAR)
+  .put(TypeProtos.MinorType.BIT, SqlTypeName.BOOLEAN)
+  .put(TypeProtos.MinorType.DATE, SqlTypeName.DATE)
+  .put(TypeProtos.MinorType.DECIMAL9, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.DECIMAL18, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.DECIMAL28SPARSE, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.DECIMAL38SPARSE, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.TIME, SqlTypeName.TIME)
+  .put(TypeProtos.MinorType.TIMESTAMP, SqlTypeName.TIMESTAMP)
+  .put(TypeProtos.MinorType.VARBINARY, SqlTypeName.VARBINARY)
+  .put(TypeProtos.MinorType.INTERVALYEAR, 
SqlTypeName.INTERVAL_YEAR_MONTH)
+  .put(TypeProtos.MinorType.INTERVALDAY, 
SqlTypeName.INTERVAL_DAY_TIME)
+  .put(TypeProtos.MinorType.MAP, SqlTypeName.MAP)
+  .put(TypeProtos.MinorType.LIST, SqlTypeName.ARRAY)
+  .put(TypeProtos.MinorType.LATE, SqlTypeName.ANY)
+  .build();
+
+  private static ImmutableMap 
CALCITE_TO_DRILL_MAPPING =
+  ImmutableMap. builder()
+  .put(SqlTypeName.INTEGER, TypeProtos.MinorType.INT)
+  .put(SqlTypeName.BIGINT, TypeProtos.MinorType.BIGINT)
+  .put(SqlTypeName.FLOAT, TypeProtos.MinorType.FLOAT4)
+  .put(SqlTypeName.DOUBLE, TypeProtos.MinorType.FLOAT8)
+  .put(SqlTypeName.VARCHAR, TypeProt

[GitHub] drill pull request: Drill 4372 review

[hsuanyi]

Github user hsuanyi commented on a diff in the pull request:

https://github.com/apache/drill/pull/397#discussion_r54963651
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/TypeInferenceUtils.java
 ---
@@ -0,0 +1,568 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql;
+
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+
+import com.google.common.collect.Maps;
+import org.apache.calcite.avatica.util.TimeUnit;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.sql.SqlCallBinding;
+import org.apache.calcite.sql.SqlCharStringLiteral;
+import org.apache.calcite.sql.SqlDynamicParam;
+import org.apache.calcite.sql.SqlLiteral;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperatorBinding;
+import org.apache.calcite.sql.type.SqlReturnTypeInference;
+import org.apache.calcite.sql.type.SqlTypeName;
+
+import org.apache.drill.common.expression.ExpressionPosition;
+import org.apache.drill.common.expression.FunctionCall;
+import org.apache.drill.common.expression.LogicalExpression;
+import org.apache.drill.common.expression.MajorTypeInLogicalExpression;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.expr.TypeHelper;
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+import org.apache.drill.exec.planner.logical.DrillConstExecutor;
+import org.apache.drill.exec.resolver.FunctionResolver;
+import org.apache.drill.exec.resolver.FunctionResolverFactory;
+import org.apache.drill.exec.resolver.TypeCastRules;
+
+import java.util.List;
+import java.util.Map;
+
+public class TypeInferenceUtils {
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(TypeInferenceUtils.class);
+
+  public static final TypeProtos.MajorType UNKNOWN_TYPE = 
TypeProtos.MajorType.getDefaultInstance();
+  private static ImmutableMap 
DRILL_TO_CALCITE_TYPE_MAPPING =
+  ImmutableMap. builder()
+  .put(TypeProtos.MinorType.INT, SqlTypeName.INTEGER)
+  .put(TypeProtos.MinorType.BIGINT, SqlTypeName.BIGINT)
+  .put(TypeProtos.MinorType.FLOAT4, SqlTypeName.FLOAT)
+  .put(TypeProtos.MinorType.FLOAT8, SqlTypeName.DOUBLE)
+  .put(TypeProtos.MinorType.VARCHAR, SqlTypeName.VARCHAR)
+  .put(TypeProtos.MinorType.BIT, SqlTypeName.BOOLEAN)
+  .put(TypeProtos.MinorType.DATE, SqlTypeName.DATE)
+  .put(TypeProtos.MinorType.DECIMAL9, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.DECIMAL18, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.DECIMAL28SPARSE, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.DECIMAL38SPARSE, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.TIME, SqlTypeName.TIME)
+  .put(TypeProtos.MinorType.TIMESTAMP, SqlTypeName.TIMESTAMP)
+  .put(TypeProtos.MinorType.VARBINARY, SqlTypeName.VARBINARY)
+  .put(TypeProtos.MinorType.INTERVALYEAR, 
SqlTypeName.INTERVAL_YEAR_MONTH)
+  .put(TypeProtos.MinorType.INTERVALDAY, 
SqlTypeName.INTERVAL_DAY_TIME)
+  .put(TypeProtos.MinorType.MAP, SqlTypeName.MAP)
+  .put(TypeProtos.MinorType.LIST, SqlTypeName.ARRAY)
+  .put(TypeProtos.MinorType.LATE, SqlTypeName.ANY)
+  .build();
+
+  private static ImmutableMap 
CALCITE_TO_DRILL_MAPPING =
+  ImmutableMap. builder()
+  .put(SqlTypeName.INTEGER, TypeProtos.MinorType.INT)
+  .put(SqlTypeName.BIGINT, TypeProtos.MinorType.BIGINT)
+  .put(SqlTypeName.FLOAT, TypeProtos.MinorType.FLOAT4)
+  .put(SqlTypeName.DOUBLE, TypeProtos.MinorType.FLOAT8)
+  .put(SqlTypeName.VARCHAR, TypeProt

[GitHub] drill pull request: Drill 4372 review

[hsuanyi]

Github user hsuanyi commented on a diff in the pull request:

https://github.com/apache/drill/pull/397#discussion_r54964785
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/TypeInferenceUtils.java
 ---
@@ -0,0 +1,568 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql;
+
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+
+import com.google.common.collect.Maps;
+import org.apache.calcite.avatica.util.TimeUnit;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.sql.SqlCallBinding;
+import org.apache.calcite.sql.SqlCharStringLiteral;
+import org.apache.calcite.sql.SqlDynamicParam;
+import org.apache.calcite.sql.SqlLiteral;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperatorBinding;
+import org.apache.calcite.sql.type.SqlReturnTypeInference;
+import org.apache.calcite.sql.type.SqlTypeName;
+
+import org.apache.drill.common.expression.ExpressionPosition;
+import org.apache.drill.common.expression.FunctionCall;
+import org.apache.drill.common.expression.LogicalExpression;
+import org.apache.drill.common.expression.MajorTypeInLogicalExpression;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.expr.TypeHelper;
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+import org.apache.drill.exec.planner.logical.DrillConstExecutor;
+import org.apache.drill.exec.resolver.FunctionResolver;
+import org.apache.drill.exec.resolver.FunctionResolverFactory;
+import org.apache.drill.exec.resolver.TypeCastRules;
+
+import java.util.List;
+import java.util.Map;
+
+public class TypeInferenceUtils {
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(TypeInferenceUtils.class);
+
+  public static final TypeProtos.MajorType UNKNOWN_TYPE = 
TypeProtos.MajorType.getDefaultInstance();
+  private static ImmutableMap 
DRILL_TO_CALCITE_TYPE_MAPPING =
+  ImmutableMap. builder()
+  .put(TypeProtos.MinorType.INT, SqlTypeName.INTEGER)
+  .put(TypeProtos.MinorType.BIGINT, SqlTypeName.BIGINT)
+  .put(TypeProtos.MinorType.FLOAT4, SqlTypeName.FLOAT)
+  .put(TypeProtos.MinorType.FLOAT8, SqlTypeName.DOUBLE)
+  .put(TypeProtos.MinorType.VARCHAR, SqlTypeName.VARCHAR)
+  .put(TypeProtos.MinorType.BIT, SqlTypeName.BOOLEAN)
+  .put(TypeProtos.MinorType.DATE, SqlTypeName.DATE)
+  .put(TypeProtos.MinorType.DECIMAL9, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.DECIMAL18, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.DECIMAL28SPARSE, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.DECIMAL38SPARSE, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.TIME, SqlTypeName.TIME)
+  .put(TypeProtos.MinorType.TIMESTAMP, SqlTypeName.TIMESTAMP)
+  .put(TypeProtos.MinorType.VARBINARY, SqlTypeName.VARBINARY)
+  .put(TypeProtos.MinorType.INTERVALYEAR, 
SqlTypeName.INTERVAL_YEAR_MONTH)
+  .put(TypeProtos.MinorType.INTERVALDAY, 
SqlTypeName.INTERVAL_DAY_TIME)
+  .put(TypeProtos.MinorType.MAP, SqlTypeName.MAP)
+  .put(TypeProtos.MinorType.LIST, SqlTypeName.ARRAY)
+  .put(TypeProtos.MinorType.LATE, SqlTypeName.ANY)
+  .build();
+
+  private static ImmutableMap 
CALCITE_TO_DRILL_MAPPING =
+  ImmutableMap. builder()
+  .put(SqlTypeName.INTEGER, TypeProtos.MinorType.INT)
+  .put(SqlTypeName.BIGINT, TypeProtos.MinorType.BIGINT)
+  .put(SqlTypeName.FLOAT, TypeProtos.MinorType.FLOAT4)
+  .put(SqlTypeName.DOUBLE, TypeProtos.MinorType.FLOAT8)
+  .put(SqlTypeName.VARCHAR, TypeProt

[GitHub] drill pull request: Drill 4372 review

[hsuanyi]

Github user hsuanyi commented on a diff in the pull request:

https://github.com/apache/drill/pull/397#discussion_r54964989
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/TypeInferenceUtils.java
 ---
@@ -0,0 +1,568 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql;
+
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+
+import com.google.common.collect.Maps;
+import org.apache.calcite.avatica.util.TimeUnit;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.sql.SqlCallBinding;
+import org.apache.calcite.sql.SqlCharStringLiteral;
+import org.apache.calcite.sql.SqlDynamicParam;
+import org.apache.calcite.sql.SqlLiteral;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperatorBinding;
+import org.apache.calcite.sql.type.SqlReturnTypeInference;
+import org.apache.calcite.sql.type.SqlTypeName;
+
+import org.apache.drill.common.expression.ExpressionPosition;
+import org.apache.drill.common.expression.FunctionCall;
+import org.apache.drill.common.expression.LogicalExpression;
+import org.apache.drill.common.expression.MajorTypeInLogicalExpression;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.expr.TypeHelper;
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+import org.apache.drill.exec.planner.logical.DrillConstExecutor;
+import org.apache.drill.exec.resolver.FunctionResolver;
+import org.apache.drill.exec.resolver.FunctionResolverFactory;
+import org.apache.drill.exec.resolver.TypeCastRules;
+
+import java.util.List;
+import java.util.Map;
+
+public class TypeInferenceUtils {
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(TypeInferenceUtils.class);
+
+  public static final TypeProtos.MajorType UNKNOWN_TYPE = 
TypeProtos.MajorType.getDefaultInstance();
+  private static ImmutableMap 
DRILL_TO_CALCITE_TYPE_MAPPING =
+  ImmutableMap. builder()
+  .put(TypeProtos.MinorType.INT, SqlTypeName.INTEGER)
+  .put(TypeProtos.MinorType.BIGINT, SqlTypeName.BIGINT)
+  .put(TypeProtos.MinorType.FLOAT4, SqlTypeName.FLOAT)
+  .put(TypeProtos.MinorType.FLOAT8, SqlTypeName.DOUBLE)
+  .put(TypeProtos.MinorType.VARCHAR, SqlTypeName.VARCHAR)
+  .put(TypeProtos.MinorType.BIT, SqlTypeName.BOOLEAN)
+  .put(TypeProtos.MinorType.DATE, SqlTypeName.DATE)
+  .put(TypeProtos.MinorType.DECIMAL9, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.DECIMAL18, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.DECIMAL28SPARSE, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.DECIMAL38SPARSE, SqlTypeName.DECIMAL)
+  .put(TypeProtos.MinorType.TIME, SqlTypeName.TIME)
+  .put(TypeProtos.MinorType.TIMESTAMP, SqlTypeName.TIMESTAMP)
+  .put(TypeProtos.MinorType.VARBINARY, SqlTypeName.VARBINARY)
+  .put(TypeProtos.MinorType.INTERVALYEAR, 
SqlTypeName.INTERVAL_YEAR_MONTH)
+  .put(TypeProtos.MinorType.INTERVALDAY, 
SqlTypeName.INTERVAL_DAY_TIME)
+  .put(TypeProtos.MinorType.MAP, SqlTypeName.MAP)
+  .put(TypeProtos.MinorType.LIST, SqlTypeName.ARRAY)
+  .put(TypeProtos.MinorType.LATE, SqlTypeName.ANY)
+  .build();
+
+  private static ImmutableMap 
CALCITE_TO_DRILL_MAPPING =
+  ImmutableMap. builder()
+  .put(SqlTypeName.INTEGER, TypeProtos.MinorType.INT)
+  .put(SqlTypeName.BIGINT, TypeProtos.MinorType.BIGINT)
+  .put(SqlTypeName.FLOAT, TypeProtos.MinorType.FLOAT4)
+  .put(SqlTypeName.DOUBLE, TypeProtos.MinorType.FLOAT8)
+  .put(SqlTypeName.VARCHAR, TypeProt

[GitHub] drill pull request: Drill 4372 review

[hsuanyi]

Github user hsuanyi commented on a diff in the pull request:

https://github.com/apache/drill/pull/397#discussion_r54965064
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillFunctionRegistry.java
 ---
@@ -17,40 +17,60 @@
  */
 package org.apache.drill.exec.expr.fn;
 
-import java.util.Arrays;
 import java.util.Collection;
+import java.util.Collections;
 import java.util.HashMap;
-import java.util.HashSet;
 import java.util.List;
+import java.util.Map;
 import java.util.Map.Entry;
-import java.util.Set;
 
-import org.apache.calcite.sql.SqlOperator;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import org.apache.commons.lang3.tuple.Pair;
 import 
org.apache.drill.common.scanner.persistence.AnnotatedClassDescriptor;
 import org.apache.drill.common.scanner.persistence.ScanResult;
-import org.apache.drill.exec.expr.DrillFunc;
 import org.apache.drill.exec.planner.logical.DrillConstExecutor;
 import org.apache.drill.exec.planner.sql.DrillOperatorTable;
 import org.apache.drill.exec.planner.sql.DrillSqlAggOperator;
 import org.apache.drill.exec.planner.sql.DrillSqlOperator;
 
 import com.google.common.collect.ArrayListMultimap;
-import com.google.common.collect.Sets;
 
+/**
+ * Registry of Drill functions.
+ */
 public class DrillFunctionRegistry {
-  static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(DrillFunctionRegistry.class);
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(DrillFunctionRegistry.class);
 
-  private ArrayListMultimap methods = 
ArrayListMultimap.create();
+  // key: function name (lowercase) value: list of functions with that name
+  private final ArrayListMultimap 
registeredFunctions = ArrayListMultimap.create();
 
-  /* Hash map to prevent registering functions with exactly matching 
signatures
-   * key: Function Name + Input's Major Type
-   * Value: Class name where function is implemented
-   */
-  private HashMap functionSignatureMap = new HashMap<>();
+  private static final Map> 
drillFuncToRange = Maps.newHashMap();
+  static {
+// CONCAT is allowed to take [1, infinity) number of arguments.
+// Currently, this flexibility is offered by DrillOptiq to rewrite it 
as
+// a nested structure
+drillFuncToRange.put("CONCAT", Pair.of(1, Integer.MAX_VALUE));
+
+// When LENGTH is given two arguments, this function relies on 
DrillOptiq to rewrite it as
+// another function based on the second argument (encodingType)
+drillFuncToRange.put("LENGTH", Pair.of(1, 2));
+
+// Dummy functions
--- End diff --

Concat is due to the flexibility of variable # of arguments
LENGTH is.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Time for the 1.6 Release

[Jason Altekruse]

I have updated the PR for the parquet date corruption issue that didn't
make it into 1.5.

https://github.com/apache/drill/pull/341
https://issues.apache.org/jira/browse/DRILL-4203

If this can get reviewed, I think it would be good to get into the release.
Any takers?

On Wed, Mar 2, 2016 at 11:07 PM, Parth Chandra  wrote:

> I've summarized the list of JIRs below.
> The first set of pull requests is under review (or have some reviewer
> assigned).
> The second set contains pull requests that need review. We need committers
> to review these. Please volunteer or these will not be able to make it into
> the release.
> The third set is Jira's that do not have a patch and/or should not be
> included because they require deeper scrutiny.
> I'm hoping we can finalize the list of PRs that can be reviewed by Friday
> morning and possibly *finalize the list of issues to be included by Friday
> end of day* so please take some time to review the PRs.
> Also note that the QA team has offered to do sanity testing once we decide
> on the final commit to be included, before the release candidate is rolled
> out, which helps with the release candidate moving forward smoothly.
>
> Here's the list -
>
> *Committed for 1.6 -*
> DRILL-4281/pr 400 (Drill should support inbound impersonation)
> DRILL-4372/pr 377(?) (Drill Operators and Functions should correctly expose
> their types within Calcite.) - Waiting for Aman to review.
> DRILL-4313/pr 396  (Improved client randomization. Update JIRA with
> warnings about using the feature ) Sudheesh to review.
> DRILL-3488/pr 388 (Java 1.8 support) Hanifi to review
> DRILL-4437 (and others)/pr 394 (Operator unit test framework). Parth to
> review
> DRILL-4384 - Query profile is missing important information on WebUi -
> Marked as resolved. Patch not applied?
>
> *Need review -*
> DRILL-4465/pr 401 (Simplify Calcite parsing & planning integration)
> DRILL-4375/pr 402 (Fix the maven release profile)
> DRILL-4452/pr 395 (Update Avatica Driver to latest Calcite)
> DRILL-4332/pr 389 (Make vector comparison order stable in test framework)
> DRILL-4449/pr 389 (Wrong results when metadata cache is used..)
> DRILL-4416/pr 385 (quote path separator)
> DRILL-4411/pr 381 (hash join over-memory condition)
> DRILL-4410/pr 380 (listvector should initiatlize bits...)
> DRILL-4387/pr 379 (GroupScan should not use star column)
> DRILL-4383/pr 375 (Allow custom configs for S3, Kerberos, etc)
> DRILL-4184/pr 372 (support variable length decimal fields in parquet)
> DRILL-4069/pr 352 Enable RPC thread offload by default
> DRILL-4120 - dir0 does not work when the directory structure contains Avro
> files - Partial patch available.
>
> *Not included (yet) - *
> DRILL-3149 - No patch available
> DRILL-4441 - IN operator does not work with Avro reader - No patch
> available
> DRILL-3745/pr 399 - Hive char support - New feature - Needs QA - Not
> included in 1.6
> DRILL-3623 - Limit 0 should avoid execution when querying a known schema.
> (Need to add limitations of current impl). Intrusive change; should be
> included at beginning of release cycle.
>
> *Others -*
> DRILL-2517   - Already resolved.
> DRILL-3688/pr 382 (skip.header.line.count in hive). - Already merged. PR
> needs to be closed.
>
>
>
> On Wed, Mar 2, 2016 at 3:11 PM, Vicky Markman 
> wrote:
>
> > You are welcome, Jacques.
> >
> > Vick*y *:)
> >
> > On Wed, Mar 2, 2016 at 3:06 PM, Jacques Nadeau 
> wrote:
> >
> > > I just realized that we didn't merge the broken profile patch (thanks
> > > Vicki). We should get it merged as well.
> > >
> > > DRILL-4384
> > >
> > > --
> > > Jacques Nadeau
> > > CTO and Co-Founder, Dremio
> > >
> > > On Wed, Mar 2, 2016 at 10:46 AM, Jason Altekruse <
> > altekruseja...@gmail.com
> > > >
> > > wrote:
> > >
> > > > I should have merged this sooner but we will need this patch that I
> had
> > > > applied to the 1.5 release branch. The change is small and fixes a
> > build
> > > > problem that only appears when running the maven release profile.
> > > >
> > > > https://github.com/apache/drill/pull/402
> > > >
> > > > On Wed, Mar 2, 2016 at 9:28 AM, Jinfeng Ni 
> > > wrote:
> > > >
> > > > > Hi John,
> > > > >
> > > > > I think patch for DRILL-2517 has been merged to the apache master
> > > > > branch. Have you tried your query on the latest master branch?
> > > > >
> > > > > In DRILL-2517, I posted some performance number for 117k small
> > parquet
> > > > > files. The patch did show improvement.
> > > > >
> > > > > Before DRILL-3996 is resolved, for now if your query relies on
> filter
> > > > > pushdown logic to push partitioning filter first, then the patch
> for
> > > > > DRILL-2517 will not help.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Mar 2, 2016 at 4:23 AM, John Omernik 
> > wrote:
> > > > > > I'd like to request drill-2517 be added as a bandaid for the
> > planning
> > > > > > issues when there are lots of directories of parquet files.
>  This
> > > > issue
> > > > > is
> > > > > > really h

[GitHub] drill pull request: DRILL-4465: Simplify Calcite parsing & plannin...

[jinfengni]

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/401#discussion_r54970576
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/SqlConverter.java
 ---
@@ -0,0 +1,364 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql;
+
+import java.util.Arrays;
+import java.util.List;
+
+import org.apache.calcite.adapter.java.JavaTypeFactory;
+import org.apache.calcite.avatica.util.Casing;
+import org.apache.calcite.avatica.util.Quoting;
+import org.apache.calcite.jdbc.CalciteSchemaImpl;
+import org.apache.calcite.jdbc.JavaTypeFactoryImpl;
+import org.apache.calcite.plan.ConventionTraitDef;
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelOptCostFactory;
+import org.apache.calcite.plan.RelOptTable;
+import org.apache.calcite.plan.volcano.VolcanoPlanner;
+import org.apache.calcite.prepare.CalciteCatalogReader;
+import org.apache.calcite.rel.RelCollationTraitDef;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rel.type.RelDataTypeSystemImpl;
+import org.apache.calcite.rex.RexBuilder;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperatorTable;
+import org.apache.calcite.sql.parser.SqlParseException;
+import org.apache.calcite.sql.parser.SqlParser;
+import org.apache.calcite.sql.parser.SqlParserImplFactory;
+import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.calcite.sql.util.ChainedSqlOperatorTable;
+import org.apache.calcite.sql.validate.SqlConformance;
+import org.apache.calcite.sql.validate.SqlValidatorCatalogReader;
+import org.apache.calcite.sql.validate.SqlValidatorImpl;
+import org.apache.calcite.sql2rel.RelDecorrelator;
+import org.apache.calcite.sql2rel.SqlToRelConverter;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.expr.fn.FunctionImplementationRegistry;
+import org.apache.drill.exec.ops.UdfUtilities;
+import org.apache.drill.exec.planner.cost.DrillCostBase;
+import org.apache.drill.exec.planner.logical.DrillConstExecutor;
+import org.apache.drill.exec.planner.physical.DrillDistributionTraitDef;
+import org.apache.drill.exec.planner.physical.PlannerSettings;
+import 
org.apache.drill.exec.planner.sql.parser.impl.DrillParserWithCompoundIdConverter;
+
+import com.google.common.base.Joiner;
+
+/**
+ * Class responsible for managing parsing, validation and toRel conversion 
for sql statements.
+ */
+public class SqlConverter {
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(SqlConverter.class);
+
+  private static DrillTypeSystem DRILL_TYPE_SYSTEM = new DrillTypeSystem();
+
+  private final JavaTypeFactory typeFactory;
+  private final SqlParser.Config parserConfig;
+  private final CalciteCatalogReader catalog;
+  private final PlannerSettings settings;
+  private final SchemaPlus rootSchema;
+  private final SchemaPlus defaultSchema;
+  private final SqlOperatorTable opTab;
+  private final RelOptCostFactory costFactory;
+  private final DrillValidator validator;
+  private final boolean isInnerQuery;
+  private final UdfUtilities util;
+  private final FunctionImplementationRegistry functions;
+
+  private String sql;
+  private VolcanoPlanner planner;
+
+
+  public SqlConverter(PlannerSettings settings, SchemaPlus defaultSchema,
+  final SqlOperatorTable operatorTable, UdfUtilities util, 
FunctionImplementationRegistry functions) {
+this.settings = settings;
+this.util = util;
+this.functions = functions;
+this.parserConfig = new ParserConfig();
+this.isInner

[GitHub] drill pull request: DRILL-4465: Simplify Calcite parsing & plannin...

[jinfengni]

Github user jinfengni commented on the pull request:

https://github.com/apache/drill/pull/401#issuecomment-192034285
  
Overall, looks good to me.  The patch simplifies Calcite integration, make 
the code cleaner, and make it easier to extend in Drill. 

+1 




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4281: Support authorized users to delega...

[sudheeshkatkam]

Github user sudheeshkatkam commented on the pull request:

https://github.com/apache/drill/pull/400#issuecomment-192036063
  
I don't think they are common. How about "principals" and 
"can_delegate_for"? I am not strongly against "can_impersonate", but I want to 
avoid confusion with user impersonation.

Does everything else look good?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4281: Support authorized users to delega...

[jacques-n]

Github user jacques-n commented on a diff in the pull request:

https://github.com/apache/drill/pull/400#discussion_r54976762
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java ---
@@ -88,6 +89,7 @@
   String USER_AUTHENTICATION_ENABLED = 
"drill.exec.security.user.auth.enabled";
   String USER_AUTHENTICATOR_IMPL = "drill.exec.security.user.auth.impl";
   String PAM_AUTHENTICATOR_PROFILES = 
"drill.exec.security.user.auth.pam_profiles";
+  String USER_DELEGATION_ENABLED = "drill.exec.delegation.enabled";
--- End diff --

Isn't an empty delegation block enough? Any reason to have a second kill 
switch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4281: Support authorized users to delega...

[jacques-n]

Github user jacques-n commented on a diff in the pull request:

https://github.com/apache/drill/pull/400#discussion_r54976899
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/rpc/user/UserSession.java ---
@@ -116,14 +137,38 @@ public OptionManager getOptions() {
 return sessionOptions;
   }
 
-  public DrillUser getUser() {
-return user;
-  }
-
   public UserCredentials getCredentials() {
 return credentials;
   }
 
+  /**
+   * Replace current user credentials with the given user's credentials, 
if authorized.
+   *
+   * @param delegatorName delegator name
+   * @throws DrillRuntimeException if credentials cannot be replaced
+   */
+  public void replaceUserCredentials(final String delegatorName) {
+assert enableDelegation;
--- End diff --

No need for assert, preconditions makes sure we get Exception instead of 
typically uncaptured Error subclass and this isn't perf sensitive.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4281: Support authorized users to delega...

[jacques-n]

Github user jacques-n commented on a diff in the pull request:

https://github.com/apache/drill/pull/400#discussion_r54977126
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/util/UserDelegationUtil.java 
---
@@ -0,0 +1,147 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.util;
+
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.collect.Sets;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.server.options.OptionValue;
+import org.apache.drill.exec.server.options.TypeValidators;
+import org.apache.hadoop.security.UserGroupInformation;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Set;
+
+/**
+ * Utilities for user delegation purpose.
+ */
+public class UserDelegationUtil {
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(UserDelegationUtil.class);
+
+  private static final String STAR = "*";
+
+  private static final ObjectMapper delegationDefinitionsMapper = new 
ObjectMapper();
+
+  static {
+
delegationDefinitionsMapper.configure(JsonGenerator.Feature.QUOTE_FIELD_NAMES, 
false);
+
delegationDefinitionsMapper.configure(JsonParser.Feature.ALLOW_UNQUOTED_FIELD_NAMES,
 true);
+  }
+
+  private static class DelegationDefinition {
+public UserGroupDefinition delegates = new UserGroupDefinition();
+public UserGroupDefinition delegators = new UserGroupDefinition();
+  }
+
+  private static class UserGroupDefinition {
+public Set users = Sets.newHashSet();
+public Set groups = Sets.newHashSet();
+  }
+
+  /**
+   * Deserialize delegation definitions string to a list of delegation 
definition objects.
+   *
+   * @param delegationDefinitions delegation definitions as a sting
+   * @return delegation definitions as a list of objects
+   * @throws IOException
+   */
+  public static List 
deserializeDelegationDefinitions(final String delegationDefinitions)
+  throws IOException {
+return delegationDefinitionsMapper.readValue(delegationDefinitions,
+new TypeReference>() {});
+  }
+
+  /**
+   * Validator for delegation definitions.
+   */
+  public static class DelegationDefinitionsValidator extends 
TypeValidators.AdminOptionValidator {
+
+public DelegationDefinitionsValidator(String name, String def) {
+  super(name, def);
+}
+
+@Override
+public void validate(OptionValue v) {
+  super.validate(v);
+
+  final List definitions;
+  try {
+definitions = deserializeDelegationDefinitions(v.string_val);
+  } catch (final IOException e) {
+throw UserException.validationError()
+.message("Invalid delegation definition.\nDetails: %s", 
e.getMessage())
+.build(logger);
+  }
+
+  for (final DelegationDefinition definition : definitions) {
+if (definition.delegates.users.contains(STAR) ||
+definition.delegates.groups.contains(STAR)) {
+  throw UserException.validationError()
+  .message("No wildcard delegates allowed.")
+  .build(logger);
+}
+  }
+}
+  }
+
+  /**
+   * Check if the given delegate is authorized to delegate for the 
delegator based on the delegation definitions.
+   *
+   * @param delegateName  delegate name
+   * @param delegatorName delegator name
+   * @param delegationDefinitions valid delegation definitions
+   * @return true iff delegate is authorized to delegate for the delegator
+   */
+  public static boolean hasDelegationPrivileges(final String delegateName, 

[GitHub] drill pull request: DRILL-4281: Support authorized users to delega...

[jacques-n]

Github user jacques-n commented on a diff in the pull request:

https://github.com/apache/drill/pull/400#discussion_r54977178
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/util/UserDelegationUtil.java 
---
@@ -0,0 +1,147 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.util;
+
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.collect.Sets;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.server.options.OptionValue;
+import org.apache.drill.exec.server.options.TypeValidators;
+import org.apache.hadoop.security.UserGroupInformation;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Set;
+
+/**
+ * Utilities for user delegation purpose.
+ */
+public class UserDelegationUtil {
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(UserDelegationUtil.class);
+
+  private static final String STAR = "*";
+
+  private static final ObjectMapper delegationDefinitionsMapper = new 
ObjectMapper();
+
+  static {
+
delegationDefinitionsMapper.configure(JsonGenerator.Feature.QUOTE_FIELD_NAMES, 
false);
+
delegationDefinitionsMapper.configure(JsonParser.Feature.ALLOW_UNQUOTED_FIELD_NAMES,
 true);
+  }
+
+  private static class DelegationDefinition {
+public UserGroupDefinition delegates = new UserGroupDefinition();
+public UserGroupDefinition delegators = new UserGroupDefinition();
+  }
+
+  private static class UserGroupDefinition {
+public Set users = Sets.newHashSet();
+public Set groups = Sets.newHashSet();
+  }
+
+  /**
+   * Deserialize delegation definitions string to a list of delegation 
definition objects.
+   *
+   * @param delegationDefinitions delegation definitions as a sting
+   * @return delegation definitions as a list of objects
+   * @throws IOException
+   */
+  public static List 
deserializeDelegationDefinitions(final String delegationDefinitions)
+  throws IOException {
+return delegationDefinitionsMapper.readValue(delegationDefinitions,
+new TypeReference>() {});
+  }
+
+  /**
+   * Validator for delegation definitions.
+   */
+  public static class DelegationDefinitionsValidator extends 
TypeValidators.AdminOptionValidator {
+
+public DelegationDefinitionsValidator(String name, String def) {
+  super(name, def);
+}
+
+@Override
+public void validate(OptionValue v) {
+  super.validate(v);
+
+  final List definitions;
+  try {
+definitions = deserializeDelegationDefinitions(v.string_val);
+  } catch (final IOException e) {
+throw UserException.validationError()
+.message("Invalid delegation definition.\nDetails: %s", 
e.getMessage())
+.build(logger);
+  }
+
+  for (final DelegationDefinition definition : definitions) {
+if (definition.delegates.users.contains(STAR) ||
+definition.delegates.groups.contains(STAR)) {
+  throw UserException.validationError()
+  .message("No wildcard delegates allowed.")
+  .build(logger);
+}
+  }
+}
+  }
+
+  /**
+   * Check if the given delegate is authorized to delegate for the 
delegator based on the delegation definitions.
+   *
+   * @param delegateName  delegate name
+   * @param delegatorName delegator name
+   * @param delegationDefinitions valid delegation definitions
+   * @return true iff delegate is authorized to delegate for the delegator
+   */
+  public static boolean hasDelegationPrivileges(final String delegateName, 

[GitHub] drill pull request: DRILL-4281: Support authorized users to delega...

[jacques-n]

Github user jacques-n commented on a diff in the pull request:

https://github.com/apache/drill/pull/400#discussion_r54977306
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/delegation/TestDelegationPrivileges.java
 ---
@@ -0,0 +1,137 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.delegation;
+
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.impersonation.BaseTestImpersonation;
+import org.apache.drill.exec.server.options.OptionValue;
+import org.apache.drill.exec.util.UserDelegationUtil;
+import org.junit.Test;
+
+import static junit.framework.Assert.assertEquals;
+
+public class TestDelegationPrivileges extends BaseTestImpersonation {
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(TestDelegationPrivileges.class);
+
+  // definitions on which the tests are based
+  private static final String DELEGATION_DEFINITIONS = "[" +
+  "{ delegates  : { users  : [\"user0_1\"] }," +
--- End diff --

Might be nice to put this in a file so we can have people refer to an 
example set of settings in the codebase (without having to filter out Java 
escaping).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4281: Support authorized users to delega...

[jacques-n]

Github user jacques-n commented on a diff in the pull request:

https://github.com/apache/drill/pull/400#discussion_r54977484
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/delegation/TestUserDelegation.java
 ---
@@ -0,0 +1,124 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.delegation;
+
+import com.google.common.collect.Maps;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.dotdrill.DotDrillType;
+import org.apache.drill.exec.impersonation.BaseTestImpersonation;
+import org.apache.drill.exec.rpc.user.UserSession;
+import 
org.apache.drill.exec.rpc.user.security.testing.UserAuthenticatorToTestDelegation;
+import org.apache.drill.exec.store.dfs.WorkspaceConfig;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.permission.FsPermission;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import java.util.Map;
+import java.util.Properties;
+
+import static org.junit.Assert.assertEquals;
+
+public class TestUserDelegation extends BaseTestImpersonation {
--- End diff --

Can you also add some negative tests that confirm nice error messages? 
(User tries to delegate to disallowed user, group, etc)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4281: Support authorized users to delega...

[jacques-n]

Github user jacques-n commented on the pull request:

https://github.com/apache/drill/pull/400#issuecomment-192046477
  
Generally looks good. +1 with the few small items above addressed. Updating 
the names to something else would be good. Since this is also impersonation 
(just client impersonation instead of storage plugin impersonation) I'm not 
sure I would shy away from using the term. The main goal for me is clear 
directionality. I think "principals" works well for the first piece. Ideas for 
the second:
"can_execute_as", "can_impersonate", "can_act_as", ?



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4281: Support authorized users to delega...

[vkorukanti]

Github user vkorukanti commented on a diff in the pull request:

https://github.com/apache/drill/pull/400#discussion_r54979630
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/rpc/user/security/testing/UserAuthenticatorToTestDelegation.java
 ---
@@ -0,0 +1,72 @@
+package org.apache.drill.exec.rpc.user.security.testing;
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.exec.exception.DrillbitStartupException;
+import org.apache.drill.exec.rpc.user.security.UserAuthenticationException;
+import org.apache.drill.exec.rpc.user.security.UserAuthenticator;
+import org.apache.drill.exec.rpc.user.security.UserAuthenticatorTemplate;
+import org.apache.drill.exec.util.ImpersonationUtil;
+
+import java.io.IOException;
+
+import static org.apache.drill.exec.delegation.TestUserDelegation.OWNER;
+import static 
org.apache.drill.exec.delegation.TestUserDelegation.OWNER_PASSWORD;
+import static 
org.apache.drill.exec.delegation.TestUserDelegation.DELEGATOR_NAME;
+import static 
org.apache.drill.exec.delegation.TestUserDelegation.DELEGATOR_PASSWORD;
+import static 
org.apache.drill.exec.delegation.TestUserDelegation.DELEGATE_NAME;
+import static 
org.apache.drill.exec.delegation.TestUserDelegation.DELEGATE_PASSWORD;
+
+/**
+ * Used by {@link org.apache.drill.exec.delegation.TestUserDelegation}.
+ *
+ * Needs to be in this package.
+ */
+@UserAuthenticatorTemplate(type = UserAuthenticatorToTestDelegation.TYPE)
+public class UserAuthenticatorToTestDelegation implements 
UserAuthenticator {
--- End diff --

Can you add the new users to existing test authenticator impl 
UserAuthenticatorTestImpl.class?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4281: Support authorized users to delega...

[vkorukanti]

Github user vkorukanti commented on a diff in the pull request:

https://github.com/apache/drill/pull/400#discussion_r54980440
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/util/UserDelegationUtil.java 
---
@@ -0,0 +1,147 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.util;
+
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.collect.Sets;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.server.options.OptionValue;
+import org.apache.drill.exec.server.options.TypeValidators;
+import org.apache.hadoop.security.UserGroupInformation;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Set;
+
+/**
+ * Utilities for user delegation purpose.
+ */
+public class UserDelegationUtil {
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(UserDelegationUtil.class);
+
+  private static final String STAR = "*";
+
+  private static final ObjectMapper delegationDefinitionsMapper = new 
ObjectMapper();
+
+  static {
+
delegationDefinitionsMapper.configure(JsonGenerator.Feature.QUOTE_FIELD_NAMES, 
false);
+
delegationDefinitionsMapper.configure(JsonParser.Feature.ALLOW_UNQUOTED_FIELD_NAMES,
 true);
+  }
+
+  private static class DelegationDefinition {
+public UserGroupDefinition delegates = new UserGroupDefinition();
+public UserGroupDefinition delegators = new UserGroupDefinition();
+  }
+
+  private static class UserGroupDefinition {
+public Set users = Sets.newHashSet();
+public Set groups = Sets.newHashSet();
+  }
+
+  /**
+   * Deserialize delegation definitions string to a list of delegation 
definition objects.
+   *
+   * @param delegationDefinitions delegation definitions as a sting
+   * @return delegation definitions as a list of objects
+   * @throws IOException
+   */
+  public static List 
deserializeDelegationDefinitions(final String delegationDefinitions)
+  throws IOException {
+return delegationDefinitionsMapper.readValue(delegationDefinitions,
+new TypeReference>() {});
+  }
+
+  /**
+   * Validator for delegation definitions.
+   */
+  public static class DelegationDefinitionsValidator extends 
TypeValidators.AdminOptionValidator {
+
+public DelegationDefinitionsValidator(String name, String def) {
+  super(name, def);
+}
+
+@Override
+public void validate(OptionValue v) {
+  super.validate(v);
+
+  final List definitions;
+  try {
+definitions = deserializeDelegationDefinitions(v.string_val);
+  } catch (final IOException e) {
+throw UserException.validationError()
+.message("Invalid delegation definition.\nDetails: %s", 
e.getMessage())
+.build(logger);
+  }
+
+  for (final DelegationDefinition definition : definitions) {
+if (definition.delegates.users.contains(STAR) ||
+definition.delegates.groups.contains(STAR)) {
+  throw UserException.validationError()
+  .message("No wildcard delegates allowed.")
+  .build(logger);
+}
+  }
+}
+  }
+
+  /**
+   * Check if the given delegate is authorized to delegate for the 
delegator based on the delegation definitions.
+   *
+   * @param delegateName  delegate name
+   * @param delegatorName delegator name
+   * @param delegationDefinitions valid delegation definitions
+   * @return true iff delegate is authorized to delegate for the delegator
+   */
+  public static boolean hasDelegationPrivileges(final String delegateName,

Re: Time for the 1.6 Release

[Parth Chandra]

Here's an updated list with names of reviewers added. If anyone else is
reviewing the open PRs please let me know. Some PRs have owners names that
I will follow up with.
Jason, I've included your JIRA in the list.


Committed for 1.6 -

DRILL-4384 - Query profile is missing important information on WebUi -
Merged
DRILL-3488/pr 388 (Java 1.8 support) - Merged.
DRILL-4410/pr 380 (listvector should initiatlize bits...) - Merged
DRILL-4383/pr 375 (Allow custom configs for S3, Kerberos, etc) - Merged
DRILL-4465/pr 401 (Simplify Calcite parsing & planning integration) -
Waiting to be merged

DRILL-4281/pr 400 (Drill should support inbound impersonation) (Sudheesh to
review)
DRILL-4372/pr 377(?) (Drill Operators and Functions should correctly expose
their types within Calcite.) - Waiting for Aman to review. (Owners: Hsuan,
Jinfeng, Aman, Sudheesh)
DRILL-4313/pr 396  (Improved client randomization. Update JIRA with
warnings about using the feature ) (Sudheesh to review.)
DRILL-4437 (and others)/pr 394 (Operator unit test framework). (Parth to
review)
DRILL-4449/pr 389 (Wrong results when metadata cache is used..) (Aman to
review)
DRILL-4416/pr 385 (quote path separator) (Owner: Hanifi)
DRILL-4069/pr 352 Enable RPC thread offload by default (Owner: Sudheesh)

Need review -
DRILL-4375/pr 402 (Fix the maven release profile)
DRILL-4452/pr 395 (Update Avatica Driver to latest Calcite)
DRILL-4332/pr 389 (Make vector comparison order stable in test framework)
DRILL-4411/pr 381 (hash join over-memory condition)
DRILL-4387/pr 379 (GroupScan should not use star column)
DRILL-4184/pr 372 (support variable length decimal fields in parquet)
DRILL-4120 - dir0 does not work when the directory structure contains Avro
files - Partial patch available.
DRILL-4203/pr 341 (fix dates written into parquet files to conform to
parquet format spec)

Not included (yet) -
DRILL-3149 - No patch available
DRILL-4441 - IN operator does not work with Avro reader - No patch available
DRILL-3745/pr 399 - Hive char support - New feature - Needs QA - Not
included in 1.6
DRILL-3623 - Limit 0 should avoid execution when querying a known schema.
(Need to add limitations of current impl). Intrusive change; should be
included at beginning of release cycle.

Others -
DRILL-2517   - Already resolved.
DRILL-3688/pr 382 (skip.header.line.count in hive). - Already merged. PR
needs to be closed.



Thanks

Parth





On Thu, Mar 3, 2016 at 3:21 PM, Jason Altekruse 
wrote:

> I have updated the PR for the parquet date corruption issue that didn't
> make it into 1.5.
>
> https://github.com/apache/drill/pull/341
> https://issues.apache.org/jira/browse/DRILL-4203
>
> If this can get reviewed, I think it would be good to get into the release.
> Any takers?
>
> On Wed, Mar 2, 2016 at 11:07 PM, Parth Chandra  wrote:
>
> > I've summarized the list of JIRs below.
> > The first set of pull requests is under review (or have some reviewer
> > assigned).
> > The second set contains pull requests that need review. We need
> committers
> > to review these. Please volunteer or these will not be able to make it
> into
> > the release.
> > The third set is Jira's that do not have a patch and/or should not be
> > included because they require deeper scrutiny.
> > I'm hoping we can finalize the list of PRs that can be reviewed by Friday
> > morning and possibly *finalize the list of issues to be included by
> Friday
> > end of day* so please take some time to review the PRs.
> > Also note that the QA team has offered to do sanity testing once we
> decide
> > on the final commit to be included, before the release candidate is
> rolled
> > out, which helps with the release candidate moving forward smoothly.
> >
> > Here's the list -
> >
> > *Committed for 1.6 -*
> > DRILL-4281/pr 400 (Drill should support inbound impersonation)
> > DRILL-4372/pr 377(?) (Drill Operators and Functions should correctly
> expose
> > their types within Calcite.) - Waiting for Aman to review.
> > DRILL-4313/pr 396  (Improved client randomization. Update JIRA with
> > warnings about using the feature ) Sudheesh to review.
> > DRILL-3488/pr 388 (Java 1.8 support) Hanifi to review
> > DRILL-4437 (and others)/pr 394 (Operator unit test framework). Parth to
> > review
> > DRILL-4384 - Query profile is missing important information on WebUi -
> > Marked as resolved. Patch not applied?
> >
> > *Need review -*
> > DRILL-4465/pr 401 (Simplify Calcite parsing & planning integration)
> > DRILL-4375/pr 402 (Fix the maven release profile)
> > DRILL-4452/pr 395 (Update Avatica Driver to latest Calcite)
> > DRILL-4332/pr 389 (Make vector comparison order stable in test framework)
> > DRILL-4449/pr 389 (Wrong results when metadata cache is used..)
> > DRILL-4416/pr 385 (quote path separator)
> > DRILL-4411/pr 381 (hash join over-memory condition)
> > DRILL-4410/pr 380 (listvector should initiatlize bits...)
> > DRILL-4387/pr 379 (GroupScan should not use star column)
> > DRILL-4383/pr 375 (Allow custom configs for S3, K

Re: Time for the 1.6 Release

[Hanifi Gunes]

DRILL-4416 is going to make it to 1.7. The patch causes a leak and I will
look into it once I make some time but 1.6 is too early.

On Thu, Mar 3, 2016 at 6:30 PM, Parth Chandra  wrote:

> Here's an updated list with names of reviewers added. If anyone else is
> reviewing the open PRs please let me know. Some PRs have owners names that
> I will follow up with.
> Jason, I've included your JIRA in the list.
>
>
> Committed for 1.6 -
>
> DRILL-4384 - Query profile is missing important information on WebUi -
> Merged
> DRILL-3488/pr 388 (Java 1.8 support) - Merged.
> DRILL-4410/pr 380 (listvector should initiatlize bits...) - Merged
> DRILL-4383/pr 375 (Allow custom configs for S3, Kerberos, etc) - Merged
> DRILL-4465/pr 401 (Simplify Calcite parsing & planning integration) -
> Waiting to be merged
>
> DRILL-4281/pr 400 (Drill should support inbound impersonation) (Sudheesh to
> review)
> DRILL-4372/pr 377(?) (Drill Operators and Functions should correctly expose
> their types within Calcite.) - Waiting for Aman to review. (Owners: Hsuan,
> Jinfeng, Aman, Sudheesh)
> DRILL-4313/pr 396  (Improved client randomization. Update JIRA with
> warnings about using the feature ) (Sudheesh to review.)
> DRILL-4437 (and others)/pr 394 (Operator unit test framework). (Parth to
> review)
> DRILL-4449/pr 389 (Wrong results when metadata cache is used..) (Aman to
> review)
> DRILL-4416/pr 385 (quote path separator) (Owner: Hanifi)
> DRILL-4069/pr 352 Enable RPC thread offload by default (Owner: Sudheesh)
>
> Need review -
> DRILL-4375/pr 402 (Fix the maven release profile)
> DRILL-4452/pr 395 (Update Avatica Driver to latest Calcite)
> DRILL-4332/pr 389 (Make vector comparison order stable in test framework)
> DRILL-4411/pr 381 (hash join over-memory condition)
> DRILL-4387/pr 379 (GroupScan should not use star column)
> DRILL-4184/pr 372 (support variable length decimal fields in parquet)
> DRILL-4120 - dir0 does not work when the directory structure contains Avro
> files - Partial patch available.
> DRILL-4203/pr 341 (fix dates written into parquet files to conform to
> parquet format spec)
>
> Not included (yet) -
> DRILL-3149 - No patch available
> DRILL-4441 - IN operator does not work with Avro reader - No patch
> available
> DRILL-3745/pr 399 - Hive char support - New feature - Needs QA - Not
> included in 1.6
> DRILL-3623 - Limit 0 should avoid execution when querying a known schema.
> (Need to add limitations of current impl). Intrusive change; should be
> included at beginning of release cycle.
>
> Others -
> DRILL-2517   - Already resolved.
> DRILL-3688/pr 382 (skip.header.line.count in hive). - Already merged. PR
> needs to be closed.
>
>
>
> Thanks
>
> Parth
>
>
>
>
>
> On Thu, Mar 3, 2016 at 3:21 PM, Jason Altekruse 
> wrote:
>
> > I have updated the PR for the parquet date corruption issue that didn't
> > make it into 1.5.
> >
> > https://github.com/apache/drill/pull/341
> > https://issues.apache.org/jira/browse/DRILL-4203
> >
> > If this can get reviewed, I think it would be good to get into the
> release.
> > Any takers?
> >
> > On Wed, Mar 2, 2016 at 11:07 PM, Parth Chandra 
> wrote:
> >
> > > I've summarized the list of JIRs below.
> > > The first set of pull requests is under review (or have some reviewer
> > > assigned).
> > > The second set contains pull requests that need review. We need
> > committers
> > > to review these. Please volunteer or these will not be able to make it
> > into
> > > the release.
> > > The third set is Jira's that do not have a patch and/or should not be
> > > included because they require deeper scrutiny.
> > > I'm hoping we can finalize the list of PRs that can be reviewed by
> Friday
> > > morning and possibly *finalize the list of issues to be included by
> > Friday
> > > end of day* so please take some time to review the PRs.
> > > Also note that the QA team has offered to do sanity testing once we
> > decide
> > > on the final commit to be included, before the release candidate is
> > rolled
> > > out, which helps with the release candidate moving forward smoothly.
> > >
> > > Here's the list -
> > >
> > > *Committed for 1.6 -*
> > > DRILL-4281/pr 400 (Drill should support inbound impersonation)
> > > DRILL-4372/pr 377(?) (Drill Operators and Functions should correctly
> > expose
> > > their types within Calcite.) - Waiting for Aman to review.
> > > DRILL-4313/pr 396  (Improved client randomization. Update JIRA with
> > > warnings about using the feature ) Sudheesh to review.
> > > DRILL-3488/pr 388 (Java 1.8 support) Hanifi to review
> > > DRILL-4437 (and others)/pr 394 (Operator unit test framework). Parth to
> > > review
> > > DRILL-4384 - Query profile is missing important information on WebUi -
> > > Marked as resolved. Patch not applied?
> > >
> > > *Need review -*
> > > DRILL-4465/pr 401 (Simplify Calcite parsing & planning integration)
> > > DRILL-4375/pr 402 (Fix the maven release profile)
> > > DRILL-4452/pr 395 (Update Avatica Driver to latest Calcite)
> > > D

[GitHub] drill pull request: DRILL-4416: quote path separator for cross pla...

[hnfgns]

Github user hnfgns commented on the pull request:

https://github.com/apache/drill/pull/385#issuecomment-192074959
  
This patch causes a random leak. I am backing it off for a while.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4281: Support authorized users to delega...

[sudheeshkatkam]

Github user sudheeshkatkam commented on a diff in the pull request:

https://github.com/apache/drill/pull/400#discussion_r54987453
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/util/UserDelegationUtil.java 
---
@@ -0,0 +1,147 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.util;
+
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.collect.Sets;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.server.options.OptionValue;
+import org.apache.drill.exec.server.options.TypeValidators;
+import org.apache.hadoop.security.UserGroupInformation;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Set;
+
+/**
+ * Utilities for user delegation purpose.
+ */
+public class UserDelegationUtil {
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(UserDelegationUtil.class);
+
+  private static final String STAR = "*";
+
+  private static final ObjectMapper delegationDefinitionsMapper = new 
ObjectMapper();
+
+  static {
+
delegationDefinitionsMapper.configure(JsonGenerator.Feature.QUOTE_FIELD_NAMES, 
false);
+
delegationDefinitionsMapper.configure(JsonParser.Feature.ALLOW_UNQUOTED_FIELD_NAMES,
 true);
+  }
+
+  private static class DelegationDefinition {
+public UserGroupDefinition delegates = new UserGroupDefinition();
+public UserGroupDefinition delegators = new UserGroupDefinition();
+  }
+
+  private static class UserGroupDefinition {
+public Set users = Sets.newHashSet();
+public Set groups = Sets.newHashSet();
+  }
+
+  /**
+   * Deserialize delegation definitions string to a list of delegation 
definition objects.
+   *
+   * @param delegationDefinitions delegation definitions as a sting
+   * @return delegation definitions as a list of objects
+   * @throws IOException
+   */
+  public static List 
deserializeDelegationDefinitions(final String delegationDefinitions)
+  throws IOException {
+return delegationDefinitionsMapper.readValue(delegationDefinitions,
+new TypeReference>() {});
+  }
+
+  /**
+   * Validator for delegation definitions.
+   */
+  public static class DelegationDefinitionsValidator extends 
TypeValidators.AdminOptionValidator {
+
+public DelegationDefinitionsValidator(String name, String def) {
+  super(name, def);
+}
+
+@Override
+public void validate(OptionValue v) {
+  super.validate(v);
+
+  final List definitions;
+  try {
+definitions = deserializeDelegationDefinitions(v.string_val);
+  } catch (final IOException e) {
+throw UserException.validationError()
+.message("Invalid delegation definition.\nDetails: %s", 
e.getMessage())
+.build(logger);
+  }
+
+  for (final DelegationDefinition definition : definitions) {
+if (definition.delegates.users.contains(STAR) ||
+definition.delegates.groups.contains(STAR)) {
+  throw UserException.validationError()
+  .message("No wildcard delegates allowed.")
+  .build(logger);
+}
+  }
+}
+  }
+
+  /**
+   * Check if the given delegate is authorized to delegate for the 
delegator based on the delegation definitions.
+   *
+   * @param delegateName  delegate name
+   * @param delegatorName delegator name
+   * @param delegationDefinitions valid delegation definitions
+   * @return true iff delegate is authorized to delegate for the delegator
+   */
+  public static boolean hasDelegationPrivileges(final String delegateN

Re: Time for the 1.6 Release

[Parth Chandra]

Yes. I noticed the comment after I sent out the email. Removing it from the
list.


On Thu, Mar 3, 2016 at 7:02 PM, Hanifi Gunes  wrote:

> DRILL-4416 is going to make it to 1.7. The patch causes a leak and I will
> look into it once I make some time but 1.6 is too early.
>
> On Thu, Mar 3, 2016 at 6:30 PM, Parth Chandra  wrote:
>
> > Here's an updated list with names of reviewers added. If anyone else is
> > reviewing the open PRs please let me know. Some PRs have owners names
> that
> > I will follow up with.
> > Jason, I've included your JIRA in the list.
> >
> >
> > Committed for 1.6 -
> >
> > DRILL-4384 - Query profile is missing important information on WebUi -
> > Merged
> > DRILL-3488/pr 388 (Java 1.8 support) - Merged.
> > DRILL-4410/pr 380 (listvector should initiatlize bits...) - Merged
> > DRILL-4383/pr 375 (Allow custom configs for S3, Kerberos, etc) - Merged
> > DRILL-4465/pr 401 (Simplify Calcite parsing & planning integration) -
> > Waiting to be merged
> >
> > DRILL-4281/pr 400 (Drill should support inbound impersonation) (Sudheesh
> to
> > review)
> > DRILL-4372/pr 377(?) (Drill Operators and Functions should correctly
> expose
> > their types within Calcite.) - Waiting for Aman to review. (Owners:
> Hsuan,
> > Jinfeng, Aman, Sudheesh)
> > DRILL-4313/pr 396  (Improved client randomization. Update JIRA with
> > warnings about using the feature ) (Sudheesh to review.)
> > DRILL-4437 (and others)/pr 394 (Operator unit test framework). (Parth to
> > review)
> > DRILL-4449/pr 389 (Wrong results when metadata cache is used..) (Aman to
> > review)
> > DRILL-4416/pr 385 (quote path separator) (Owner: Hanifi)
> > DRILL-4069/pr 352 Enable RPC thread offload by default (Owner: Sudheesh)
> >
> > Need review -
> > DRILL-4375/pr 402 (Fix the maven release profile)
> > DRILL-4452/pr 395 (Update Avatica Driver to latest Calcite)
> > DRILL-4332/pr 389 (Make vector comparison order stable in test framework)
> > DRILL-4411/pr 381 (hash join over-memory condition)
> > DRILL-4387/pr 379 (GroupScan should not use star column)
> > DRILL-4184/pr 372 (support variable length decimal fields in parquet)
> > DRILL-4120 - dir0 does not work when the directory structure contains
> Avro
> > files - Partial patch available.
> > DRILL-4203/pr 341 (fix dates written into parquet files to conform to
> > parquet format spec)
> >
> > Not included (yet) -
> > DRILL-3149 - No patch available
> > DRILL-4441 - IN operator does not work with Avro reader - No patch
> > available
> > DRILL-3745/pr 399 - Hive char support - New feature - Needs QA - Not
> > included in 1.6
> > DRILL-3623 - Limit 0 should avoid execution when querying a known schema.
> > (Need to add limitations of current impl). Intrusive change; should be
> > included at beginning of release cycle.
> >
> > Others -
> > DRILL-2517   - Already resolved.
> > DRILL-3688/pr 382 (skip.header.line.count in hive). - Already merged. PR
> > needs to be closed.
> >
> >
> >
> > Thanks
> >
> > Parth
> >
> >
> >
> >
> >
> > On Thu, Mar 3, 2016 at 3:21 PM, Jason Altekruse <
> altekruseja...@gmail.com>
> > wrote:
> >
> > > I have updated the PR for the parquet date corruption issue that didn't
> > > make it into 1.5.
> > >
> > > https://github.com/apache/drill/pull/341
> > > https://issues.apache.org/jira/browse/DRILL-4203
> > >
> > > If this can get reviewed, I think it would be good to get into the
> > release.
> > > Any takers?
> > >
> > > On Wed, Mar 2, 2016 at 11:07 PM, Parth Chandra 
> > wrote:
> > >
> > > > I've summarized the list of JIRs below.
> > > > The first set of pull requests is under review (or have some reviewer
> > > > assigned).
> > > > The second set contains pull requests that need review. We need
> > > committers
> > > > to review these. Please volunteer or these will not be able to make
> it
> > > into
> > > > the release.
> > > > The third set is Jira's that do not have a patch and/or should not be
> > > > included because they require deeper scrutiny.
> > > > I'm hoping we can finalize the list of PRs that can be reviewed by
> > Friday
> > > > morning and possibly *finalize the list of issues to be included by
> > > Friday
> > > > end of day* so please take some time to review the PRs.
> > > > Also note that the QA team has offered to do sanity testing once we
> > > decide
> > > > on the final commit to be included, before the release candidate is
> > > rolled
> > > > out, which helps with the release candidate moving forward smoothly.
> > > >
> > > > Here's the list -
> > > >
> > > > *Committed for 1.6 -*
> > > > DRILL-4281/pr 400 (Drill should support inbound impersonation)
> > > > DRILL-4372/pr 377(?) (Drill Operators and Functions should correctly
> > > expose
> > > > their types within Calcite.) - Waiting for Aman to review.
> > > > DRILL-4313/pr 396  (Improved client randomization. Update JIRA with
> > > > warnings about using the feature ) Sudheesh to review.
> > > > DRILL-3488/pr 388 (Java 1.8 support) Hanifi to review
> > > > DRILL-4437 (and other

Re: Time for the 1.6 Release

[Zelaine Fong]

DRILL-4281/pr 400 (Drill should support inbound impersonation) (Sudheesh to
review)

Sudheesh is the fixer of DRILL-4281, so I don't think he can be the
reviewer :).

-- Zelaine

On Thu, Mar 3, 2016 at 6:30 PM, Parth Chandra  wrote:

> Here's an updated list with names of reviewers added. If anyone else is
> reviewing the open PRs please let me know. Some PRs have owners names that
> I will follow up with.
> Jason, I've included your JIRA in the list.
>
>
> Committed for 1.6 -
>
> DRILL-4384 - Query profile is missing important information on WebUi -
> Merged
> DRILL-3488/pr 388 (Java 1.8 support) - Merged.
> DRILL-4410/pr 380 (listvector should initiatlize bits...) - Merged
> DRILL-4383/pr 375 (Allow custom configs for S3, Kerberos, etc) - Merged
> DRILL-4465/pr 401 (Simplify Calcite parsing & planning integration) -
> Waiting to be merged
>
> DRILL-4281/pr 400 (Drill should support inbound impersonation) (Sudheesh to
> review)
> DRILL-4372/pr 377(?) (Drill Operators and Functions should correctly expose
> their types within Calcite.) - Waiting for Aman to review. (Owners: Hsuan,
> Jinfeng, Aman, Sudheesh)
> DRILL-4313/pr 396  (Improved client randomization. Update JIRA with
> warnings about using the feature ) (Sudheesh to review.)
> DRILL-4437 (and others)/pr 394 (Operator unit test framework). (Parth to
> review)
> DRILL-4449/pr 389 (Wrong results when metadata cache is used..) (Aman to
> review)
> DRILL-4416/pr 385 (quote path separator) (Owner: Hanifi)
> DRILL-4069/pr 352 Enable RPC thread offload by default (Owner: Sudheesh)
>
> Need review -
> DRILL-4375/pr 402 (Fix the maven release profile)
> DRILL-4452/pr 395 (Update Avatica Driver to latest Calcite)
> DRILL-4332/pr 389 (Make vector comparison order stable in test framework)
> DRILL-4411/pr 381 (hash join over-memory condition)
> DRILL-4387/pr 379 (GroupScan should not use star column)
> DRILL-4184/pr 372 (support variable length decimal fields in parquet)
> DRILL-4120 - dir0 does not work when the directory structure contains Avro
> files - Partial patch available.
> DRILL-4203/pr 341 (fix dates written into parquet files to conform to
> parquet format spec)
>
> Not included (yet) -
> DRILL-3149 - No patch available
> DRILL-4441 - IN operator does not work with Avro reader - No patch
> available
> DRILL-3745/pr 399 - Hive char support - New feature - Needs QA - Not
> included in 1.6
> DRILL-3623 - Limit 0 should avoid execution when querying a known schema.
> (Need to add limitations of current impl). Intrusive change; should be
> included at beginning of release cycle.
>
> Others -
> DRILL-2517   - Already resolved.
> DRILL-3688/pr 382 (skip.header.line.count in hive). - Already merged. PR
> needs to be closed.
>
>
>
> Thanks
>
> Parth
>
>
>
>
>
> On Thu, Mar 3, 2016 at 3:21 PM, Jason Altekruse 
> wrote:
>
> > I have updated the PR for the parquet date corruption issue that didn't
> > make it into 1.5.
> >
> > https://github.com/apache/drill/pull/341
> > https://issues.apache.org/jira/browse/DRILL-4203
> >
> > If this can get reviewed, I think it would be good to get into the
> release.
> > Any takers?
> >
> > On Wed, Mar 2, 2016 at 11:07 PM, Parth Chandra 
> wrote:
> >
> > > I've summarized the list of JIRs below.
> > > The first set of pull requests is under review (or have some reviewer
> > > assigned).
> > > The second set contains pull requests that need review. We need
> > committers
> > > to review these. Please volunteer or these will not be able to make it
> > into
> > > the release.
> > > The third set is Jira's that do not have a patch and/or should not be
> > > included because they require deeper scrutiny.
> > > I'm hoping we can finalize the list of PRs that can be reviewed by
> Friday
> > > morning and possibly *finalize the list of issues to be included by
> > Friday
> > > end of day* so please take some time to review the PRs.
> > > Also note that the QA team has offered to do sanity testing once we
> > decide
> > > on the final commit to be included, before the release candidate is
> > rolled
> > > out, which helps with the release candidate moving forward smoothly.
> > >
> > > Here's the list -
> > >
> > > *Committed for 1.6 -*
> > > DRILL-4281/pr 400 (Drill should support inbound impersonation)
> > > DRILL-4372/pr 377(?) (Drill Operators and Functions should correctly
> > expose
> > > their types within Calcite.) - Waiting for Aman to review.
> > > DRILL-4313/pr 396  (Improved client randomization. Update JIRA with
> > > warnings about using the feature ) Sudheesh to review.
> > > DRILL-3488/pr 388 (Java 1.8 support) Hanifi to review
> > > DRILL-4437 (and others)/pr 394 (Operator unit test framework). Parth to
> > > review
> > > DRILL-4384 - Query profile is missing important information on WebUi -
> > > Marked as resolved. Patch not applied?
> > >
> > > *Need review -*
> > > DRILL-4465/pr 401 (Simplify Calcite parsing & planning integration)
> > > DRILL-4375/pr 402 (Fix the maven release profile)
> > > DRILL-4452/pr 395 (U

[GitHub] drill pull request: DRILL-4281: Support authorized users to delega...

[jacques-n]

Github user jacques-n commented on a diff in the pull request:

https://github.com/apache/drill/pull/400#discussion_r54990639
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/util/UserDelegationUtil.java 
---
@@ -0,0 +1,147 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.util;
+
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.collect.Sets;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.server.options.OptionValue;
+import org.apache.drill.exec.server.options.TypeValidators;
+import org.apache.hadoop.security.UserGroupInformation;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Set;
+
+/**
+ * Utilities for user delegation purpose.
+ */
+public class UserDelegationUtil {
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(UserDelegationUtil.class);
+
+  private static final String STAR = "*";
+
+  private static final ObjectMapper delegationDefinitionsMapper = new 
ObjectMapper();
+
+  static {
+
delegationDefinitionsMapper.configure(JsonGenerator.Feature.QUOTE_FIELD_NAMES, 
false);
+
delegationDefinitionsMapper.configure(JsonParser.Feature.ALLOW_UNQUOTED_FIELD_NAMES,
 true);
+  }
+
+  private static class DelegationDefinition {
+public UserGroupDefinition delegates = new UserGroupDefinition();
+public UserGroupDefinition delegators = new UserGroupDefinition();
+  }
+
+  private static class UserGroupDefinition {
+public Set users = Sets.newHashSet();
+public Set groups = Sets.newHashSet();
+  }
+
+  /**
+   * Deserialize delegation definitions string to a list of delegation 
definition objects.
+   *
+   * @param delegationDefinitions delegation definitions as a sting
+   * @return delegation definitions as a list of objects
+   * @throws IOException
+   */
+  public static List 
deserializeDelegationDefinitions(final String delegationDefinitions)
+  throws IOException {
+return delegationDefinitionsMapper.readValue(delegationDefinitions,
+new TypeReference>() {});
+  }
+
+  /**
+   * Validator for delegation definitions.
+   */
+  public static class DelegationDefinitionsValidator extends 
TypeValidators.AdminOptionValidator {
+
+public DelegationDefinitionsValidator(String name, String def) {
+  super(name, def);
+}
+
+@Override
+public void validate(OptionValue v) {
+  super.validate(v);
+
+  final List definitions;
+  try {
+definitions = deserializeDelegationDefinitions(v.string_val);
+  } catch (final IOException e) {
+throw UserException.validationError()
+.message("Invalid delegation definition.\nDetails: %s", 
e.getMessage())
+.build(logger);
+  }
+
+  for (final DelegationDefinition definition : definitions) {
+if (definition.delegates.users.contains(STAR) ||
+definition.delegates.groups.contains(STAR)) {
+  throw UserException.validationError()
+  .message("No wildcard delegates allowed.")
+  .build(logger);
+}
+  }
+}
+  }
+
+  /**
+   * Check if the given delegate is authorized to delegate for the 
delegator based on the delegation definitions.
+   *
+   * @param delegateName  delegate name
+   * @param delegatorName delegator name
+   * @param delegationDefinitions valid delegation definitions
+   * @return true iff delegate is authorized to delegate for the delegator
+   */
+  public static boolean hasDelegationPrivileges(final String delegateName, 

Re: Time for the 1.6 Release

[Jacques Nadeau]

I've been reviewing 4281. It's really close to ready.

--
Jacques Nadeau
CTO and Co-Founder, Dremio

On Thu, Mar 3, 2016 at 9:08 PM, Zelaine Fong  wrote:

> DRILL-4281/pr 400 (Drill should support inbound impersonation) (Sudheesh to
> review)
>
> Sudheesh is the fixer of DRILL-4281, so I don't think he can be the
> reviewer :).
>
> -- Zelaine
>
> On Thu, Mar 3, 2016 at 6:30 PM, Parth Chandra  wrote:
>
> > Here's an updated list with names of reviewers added. If anyone else is
> > reviewing the open PRs please let me know. Some PRs have owners names
> that
> > I will follow up with.
> > Jason, I've included your JIRA in the list.
> >
> >
> > Committed for 1.6 -
> >
> > DRILL-4384 - Query profile is missing important information on WebUi -
> > Merged
> > DRILL-3488/pr 388 (Java 1.8 support) - Merged.
> > DRILL-4410/pr 380 (listvector should initiatlize bits...) - Merged
> > DRILL-4383/pr 375 (Allow custom configs for S3, Kerberos, etc) - Merged
> > DRILL-4465/pr 401 (Simplify Calcite parsing & planning integration) -
> > Waiting to be merged
> >
> > DRILL-4281/pr 400 (Drill should support inbound impersonation) (Sudheesh
> to
> > review)
> > DRILL-4372/pr 377(?) (Drill Operators and Functions should correctly
> expose
> > their types within Calcite.) - Waiting for Aman to review. (Owners:
> Hsuan,
> > Jinfeng, Aman, Sudheesh)
> > DRILL-4313/pr 396  (Improved client randomization. Update JIRA with
> > warnings about using the feature ) (Sudheesh to review.)
> > DRILL-4437 (and others)/pr 394 (Operator unit test framework). (Parth to
> > review)
> > DRILL-4449/pr 389 (Wrong results when metadata cache is used..) (Aman to
> > review)
> > DRILL-4416/pr 385 (quote path separator) (Owner: Hanifi)
> > DRILL-4069/pr 352 Enable RPC thread offload by default (Owner: Sudheesh)
> >
> > Need review -
> > DRILL-4375/pr 402 (Fix the maven release profile)
> > DRILL-4452/pr 395 (Update Avatica Driver to latest Calcite)
> > DRILL-4332/pr 389 (Make vector comparison order stable in test framework)
> > DRILL-4411/pr 381 (hash join over-memory condition)
> > DRILL-4387/pr 379 (GroupScan should not use star column)
> > DRILL-4184/pr 372 (support variable length decimal fields in parquet)
> > DRILL-4120 - dir0 does not work when the directory structure contains
> Avro
> > files - Partial patch available.
> > DRILL-4203/pr 341 (fix dates written into parquet files to conform to
> > parquet format spec)
> >
> > Not included (yet) -
> > DRILL-3149 - No patch available
> > DRILL-4441 - IN operator does not work with Avro reader - No patch
> > available
> > DRILL-3745/pr 399 - Hive char support - New feature - Needs QA - Not
> > included in 1.6
> > DRILL-3623 - Limit 0 should avoid execution when querying a known schema.
> > (Need to add limitations of current impl). Intrusive change; should be
> > included at beginning of release cycle.
> >
> > Others -
> > DRILL-2517   - Already resolved.
> > DRILL-3688/pr 382 (skip.header.line.count in hive). - Already merged. PR
> > needs to be closed.
> >
> >
> >
> > Thanks
> >
> > Parth
> >
> >
> >
> >
> >
> > On Thu, Mar 3, 2016 at 3:21 PM, Jason Altekruse <
> altekruseja...@gmail.com>
> > wrote:
> >
> > > I have updated the PR for the parquet date corruption issue that didn't
> > > make it into 1.5.
> > >
> > > https://github.com/apache/drill/pull/341
> > > https://issues.apache.org/jira/browse/DRILL-4203
> > >
> > > If this can get reviewed, I think it would be good to get into the
> > release.
> > > Any takers?
> > >
> > > On Wed, Mar 2, 2016 at 11:07 PM, Parth Chandra 
> > wrote:
> > >
> > > > I've summarized the list of JIRs below.
> > > > The first set of pull requests is under review (or have some reviewer
> > > > assigned).
> > > > The second set contains pull requests that need review. We need
> > > committers
> > > > to review these. Please volunteer or these will not be able to make
> it
> > > into
> > > > the release.
> > > > The third set is Jira's that do not have a patch and/or should not be
> > > > included because they require deeper scrutiny.
> > > > I'm hoping we can finalize the list of PRs that can be reviewed by
> > Friday
> > > > morning and possibly *finalize the list of issues to be included by
> > > Friday
> > > > end of day* so please take some time to review the PRs.
> > > > Also note that the QA team has offered to do sanity testing once we
> > > decide
> > > > on the final commit to be included, before the release candidate is
> > > rolled
> > > > out, which helps with the release candidate moving forward smoothly.
> > > >
> > > > Here's the list -
> > > >
> > > > *Committed for 1.6 -*
> > > > DRILL-4281/pr 400 (Drill should support inbound impersonation)
> > > > DRILL-4372/pr 377(?) (Drill Operators and Functions should correctly
> > > expose
> > > > their types within Calcite.) - Waiting for Aman to review.
> > > > DRILL-4313/pr 396  (Improved client randomization. Update JIRA with
> > > > warnings about using the feature ) Sudheesh to review.
> > > > DRILL-3488/

[GitHub] drill pull request: DRILL-4437: Operator unit tests

[parthchandra]

Github user parthchandra commented on the pull request:

https://github.com/apache/drill/pull/394#issuecomment-192114749
  
+1. Great to have this framework; nicely done.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Time for the 1.6 Release

[Parth Chandra]

Right. My mistake. Thanks, Jacques, for reviewing.

On Thu, Mar 3, 2016 at 9:08 PM, Zelaine Fong  wrote:

> DRILL-4281/pr 400 (Drill should support inbound impersonation) (Sudheesh to
> review)
>
> Sudheesh is the fixer of DRILL-4281, so I don't think he can be the
> reviewer :).
>
> -- Zelaine
>
> On Thu, Mar 3, 2016 at 6:30 PM, Parth Chandra  wrote:
>
> > Here's an updated list with names of reviewers added. If anyone else is
> > reviewing the open PRs please let me know. Some PRs have owners names
> that
> > I will follow up with.
> > Jason, I've included your JIRA in the list.
> >
> >
> > Committed for 1.6 -
> >
> > DRILL-4384 - Query profile is missing important information on WebUi -
> > Merged
> > DRILL-3488/pr 388 (Java 1.8 support) - Merged.
> > DRILL-4410/pr 380 (listvector should initiatlize bits...) - Merged
> > DRILL-4383/pr 375 (Allow custom configs for S3, Kerberos, etc) - Merged
> > DRILL-4465/pr 401 (Simplify Calcite parsing & planning integration) -
> > Waiting to be merged
> >
> > DRILL-4281/pr 400 (Drill should support inbound impersonation) (Sudheesh
> to
> > review)
> > DRILL-4372/pr 377(?) (Drill Operators and Functions should correctly
> expose
> > their types within Calcite.) - Waiting for Aman to review. (Owners:
> Hsuan,
> > Jinfeng, Aman, Sudheesh)
> > DRILL-4313/pr 396  (Improved client randomization. Update JIRA with
> > warnings about using the feature ) (Sudheesh to review.)
> > DRILL-4437 (and others)/pr 394 (Operator unit test framework). (Parth to
> > review)
> > DRILL-4449/pr 389 (Wrong results when metadata cache is used..) (Aman to
> > review)
> > DRILL-4416/pr 385 (quote path separator) (Owner: Hanifi)
> > DRILL-4069/pr 352 Enable RPC thread offload by default (Owner: Sudheesh)
> >
> > Need review -
> > DRILL-4375/pr 402 (Fix the maven release profile)
> > DRILL-4452/pr 395 (Update Avatica Driver to latest Calcite)
> > DRILL-4332/pr 389 (Make vector comparison order stable in test framework)
> > DRILL-4411/pr 381 (hash join over-memory condition)
> > DRILL-4387/pr 379 (GroupScan should not use star column)
> > DRILL-4184/pr 372 (support variable length decimal fields in parquet)
> > DRILL-4120 - dir0 does not work when the directory structure contains
> Avro
> > files - Partial patch available.
> > DRILL-4203/pr 341 (fix dates written into parquet files to conform to
> > parquet format spec)
> >
> > Not included (yet) -
> > DRILL-3149 - No patch available
> > DRILL-4441 - IN operator does not work with Avro reader - No patch
> > available
> > DRILL-3745/pr 399 - Hive char support - New feature - Needs QA - Not
> > included in 1.6
> > DRILL-3623 - Limit 0 should avoid execution when querying a known schema.
> > (Need to add limitations of current impl). Intrusive change; should be
> > included at beginning of release cycle.
> >
> > Others -
> > DRILL-2517   - Already resolved.
> > DRILL-3688/pr 382 (skip.header.line.count in hive). - Already merged. PR
> > needs to be closed.
> >
> >
> >
> > Thanks
> >
> > Parth
> >
> >
> >
> >
> >
> > On Thu, Mar 3, 2016 at 3:21 PM, Jason Altekruse <
> altekruseja...@gmail.com>
> > wrote:
> >
> > > I have updated the PR for the parquet date corruption issue that didn't
> > > make it into 1.5.
> > >
> > > https://github.com/apache/drill/pull/341
> > > https://issues.apache.org/jira/browse/DRILL-4203
> > >
> > > If this can get reviewed, I think it would be good to get into the
> > release.
> > > Any takers?
> > >
> > > On Wed, Mar 2, 2016 at 11:07 PM, Parth Chandra 
> > wrote:
> > >
> > > > I've summarized the list of JIRs below.
> > > > The first set of pull requests is under review (or have some reviewer
> > > > assigned).
> > > > The second set contains pull requests that need review. We need
> > > committers
> > > > to review these. Please volunteer or these will not be able to make
> it
> > > into
> > > > the release.
> > > > The third set is Jira's that do not have a patch and/or should not be
> > > > included because they require deeper scrutiny.
> > > > I'm hoping we can finalize the list of PRs that can be reviewed by
> > Friday
> > > > morning and possibly *finalize the list of issues to be included by
> > > Friday
> > > > end of day* so please take some time to review the PRs.
> > > > Also note that the QA team has offered to do sanity testing once we
> > > decide
> > > > on the final commit to be included, before the release candidate is
> > > rolled
> > > > out, which helps with the release candidate moving forward smoothly.
> > > >
> > > > Here's the list -
> > > >
> > > > *Committed for 1.6 -*
> > > > DRILL-4281/pr 400 (Drill should support inbound impersonation)
> > > > DRILL-4372/pr 377(?) (Drill Operators and Functions should correctly
> > > expose
> > > > their types within Calcite.) - Waiting for Aman to review.
> > > > DRILL-4313/pr 396  (Improved client randomization. Update JIRA with
> > > > warnings about using the feature ) Sudheesh to review.
> > > > DRILL-3488/pr 388 (Java 1.8 support) Hanifi to review
> > > 

Re: Time for the 1.6 Release

[Parth Chandra]

Updated list  (I'll follow up with the folks named here separately) -

Committed for 1.6 -

DRILL-4384 - Query profile is missing important information on WebUi -
Merged
DRILL-3488/pr 388 (Java 1.8 support) - Merged.
DRILL-4410/pr 380 (listvector should initiatlize bits...) - Merged
DRILL-4383/pr 375 (Allow custom configs for S3, Kerberos, etc) - Merged
DRILL-4465/pr 401 (Simplify Calcite parsing & planning integration) -
Waiting to be merged
DRILL-4437 (and others)/pr 394 (Operator unit test framework). Waiting to
be merged.

DRILL-4281/pr 400 (Drill should support inbound impersonation) (Jacques to
review)
DRILL-4372/pr 377(?) (Drill Operators and Functions should correctly expose
their types within Calcite.) - Waiting for Aman to review. (Owners: Hsuan,
Jinfeng, Aman, Sudheesh)
DRILL-4313/pr 396  (Improved client randomization. Update JIRA with
warnings about using the feature ) (Sudheesh to review.)
DRILL-4449/pr 389 (Wrong results when metadata cache is used..) (Aman to
review)
DRILL-4069/pr 352 Enable RPC thread offload by default (Owner: Sudheesh)

Need review -
DRILL-4375/pr 402 (Fix the maven release profile)
DRILL-4452/pr 395 (Update Avatica Driver to latest Calcite)
DRILL-4332/pr 389 (Make vector comparison order stable in test framework)
DRILL-4411/pr 381 (hash join over-memory condition)
DRILL-4387/pr 379 (GroupScan should not use star column)
DRILL-4184/pr 372 (support variable length decimal fields in parquet)
DRILL-4120 - dir0 does not work when the directory structure contains Avro
files - Partial patch available.
DRILL-4203/pr 341 (fix dates written into parquet files to conform to
parquet format spec)

Not included (yet) -
DRILL-3149 - No patch available
DRILL-4441 - IN operator does not work with Avro reader - No patch available
DRILL-3745/pr 399 - Hive char support - New feature - Needs QA - Not
included in 1.6
DRILL-3623 - Limit 0 should avoid execution when querying a known schema.
(Need to add limitations of current impl). Intrusive change; should be
included at beginning of release cycle.
DRILL-4416/pr 385 (quote path separator) (Owner: Hanifi) - Causes leak.

Others -
DRILL-2517   - Already resolved.
DRILL-3688/pr 382 (skip.header.line.count in hive). - Already merged. PR
needs to be closed.


On Thu, Mar 3, 2016 at 9:44 PM, Parth Chandra  wrote:

> Right. My mistake. Thanks, Jacques, for reviewing.
>
> On Thu, Mar 3, 2016 at 9:08 PM, Zelaine Fong  wrote:
>
>> DRILL-4281/pr 400 (Drill should support inbound impersonation) (Sudheesh
>> to
>> review)
>>
>> Sudheesh is the fixer of DRILL-4281, so I don't think he can be the
>> reviewer :).
>>
>> -- Zelaine
>>
>> On Thu, Mar 3, 2016 at 6:30 PM, Parth Chandra  wrote:
>>
>> > Here's an updated list with names of reviewers added. If anyone else is
>> > reviewing the open PRs please let me know. Some PRs have owners names
>> that
>> > I will follow up with.
>> > Jason, I've included your JIRA in the list.
>> >
>> >
>> > Committed for 1.6 -
>> >
>> > DRILL-4384 - Query profile is missing important information on WebUi -
>> > Merged
>> > DRILL-3488/pr 388 (Java 1.8 support) - Merged.
>> > DRILL-4410/pr 380 (listvector should initiatlize bits...) - Merged
>> > DRILL-4383/pr 375 (Allow custom configs for S3, Kerberos, etc) - Merged
>> > DRILL-4465/pr 401 (Simplify Calcite parsing & planning integration) -
>> > Waiting to be merged
>> >
>> > DRILL-4281/pr 400 (Drill should support inbound impersonation)
>> (Sudheesh to
>> > review)
>> > DRILL-4372/pr 377(?) (Drill Operators and Functions should correctly
>> expose
>> > their types within Calcite.) - Waiting for Aman to review. (Owners:
>> Hsuan,
>> > Jinfeng, Aman, Sudheesh)
>> > DRILL-4313/pr 396  (Improved client randomization. Update JIRA with
>> > warnings about using the feature ) (Sudheesh to review.)
>> > DRILL-4437 (and others)/pr 394 (Operator unit test framework). (Parth to
>> > review)
>> > DRILL-4449/pr 389 (Wrong results when metadata cache is used..) (Aman to
>> > review)
>> > DRILL-4416/pr 385 (quote path separator) (Owner: Hanifi)
>> > DRILL-4069/pr 352 Enable RPC thread offload by default (Owner: Sudheesh)
>> >
>> > Need review -
>> > DRILL-4375/pr 402 (Fix the maven release profile)
>> > DRILL-4452/pr 395 (Update Avatica Driver to latest Calcite)
>> > DRILL-4332/pr 389 (Make vector comparison order stable in test
>> framework)
>> > DRILL-4411/pr 381 (hash join over-memory condition)
>> > DRILL-4387/pr 379 (GroupScan should not use star column)
>> > DRILL-4184/pr 372 (support variable length decimal fields in parquet)
>> > DRILL-4120 - dir0 does not work when the directory structure contains
>> Avro
>> > files - Partial patch available.
>> > DRILL-4203/pr 341 (fix dates written into parquet files to conform to
>> > parquet format spec)
>> >
>> > Not included (yet) -
>> > DRILL-3149 - No patch available
>> > DRILL-4441 - IN operator does not work with Avro reader - No patch
>> > available
>> > DRILL-3745/pr 399 - Hive char support - New feature - Needs QA - Not
>> > included in

Re: Time for the 1.6 Release

[Jacques Nadeau]

I think we need to include DRILL-4467
. I think it is a one
line patch and it provides unpredictable plans at a minimum but may also
present invalid result. Still need to think through the second half. I've
seen this plan instability in some of my recent test runs (even without
Java 8) when running extended HBase tests.

--
Jacques Nadeau
CTO and Co-Founder, Dremio

On Thu, Mar 3, 2016 at 10:02 PM, Parth Chandra  wrote:

> Updated list  (I'll follow up with the folks named here separately) -
>
> Committed for 1.6 -
>
> DRILL-4384 - Query profile is missing important information on WebUi -
> Merged
> DRILL-3488/pr 388 (Java 1.8 support) - Merged.
> DRILL-4410/pr 380 (listvector should initiatlize bits...) - Merged
> DRILL-4383/pr 375 (Allow custom configs for S3, Kerberos, etc) - Merged
> DRILL-4465/pr 401 (Simplify Calcite parsing & planning integration) -
> Waiting to be merged
> DRILL-4437 (and others)/pr 394 (Operator unit test framework). Waiting to
> be merged.
>
> DRILL-4281/pr 400 (Drill should support inbound impersonation) (Jacques to
> review)
> DRILL-4372/pr 377(?) (Drill Operators and Functions should correctly expose
> their types within Calcite.) - Waiting for Aman to review. (Owners: Hsuan,
> Jinfeng, Aman, Sudheesh)
> DRILL-4313/pr 396  (Improved client randomization. Update JIRA with
> warnings about using the feature ) (Sudheesh to review.)
> DRILL-4449/pr 389 (Wrong results when metadata cache is used..) (Aman to
> review)
> DRILL-4069/pr 352 Enable RPC thread offload by default (Owner: Sudheesh)
>
> Need review -
> DRILL-4375/pr 402 (Fix the maven release profile)
> DRILL-4452/pr 395 (Update Avatica Driver to latest Calcite)
> DRILL-4332/pr 389 (Make vector comparison order stable in test framework)
> DRILL-4411/pr 381 (hash join over-memory condition)
> DRILL-4387/pr 379 (GroupScan should not use star column)
> DRILL-4184/pr 372 (support variable length decimal fields in parquet)
> DRILL-4120 - dir0 does not work when the directory structure contains Avro
> files - Partial patch available.
> DRILL-4203/pr 341 (fix dates written into parquet files to conform to
> parquet format spec)
>
> Not included (yet) -
> DRILL-3149 - No patch available
> DRILL-4441 - IN operator does not work with Avro reader - No patch
> available
> DRILL-3745/pr 399 - Hive char support - New feature - Needs QA - Not
> included in 1.6
> DRILL-3623 - Limit 0 should avoid execution when querying a known schema.
> (Need to add limitations of current impl). Intrusive change; should be
> included at beginning of release cycle.
> DRILL-4416/pr 385 (quote path separator) (Owner: Hanifi) - Causes leak.
>
> Others -
> DRILL-2517   - Already resolved.
> DRILL-3688/pr 382 (skip.header.line.count in hive). - Already merged. PR
> needs to be closed.
>
>
> On Thu, Mar 3, 2016 at 9:44 PM, Parth Chandra  wrote:
>
> > Right. My mistake. Thanks, Jacques, for reviewing.
> >
> > On Thu, Mar 3, 2016 at 9:08 PM, Zelaine Fong  wrote:
> >
> >> DRILL-4281/pr 400 (Drill should support inbound impersonation) (Sudheesh
> >> to
> >> review)
> >>
> >> Sudheesh is the fixer of DRILL-4281, so I don't think he can be the
> >> reviewer :).
> >>
> >> -- Zelaine
> >>
> >> On Thu, Mar 3, 2016 at 6:30 PM, Parth Chandra 
> wrote:
> >>
> >> > Here's an updated list with names of reviewers added. If anyone else
> is
> >> > reviewing the open PRs please let me know. Some PRs have owners names
> >> that
> >> > I will follow up with.
> >> > Jason, I've included your JIRA in the list.
> >> >
> >> >
> >> > Committed for 1.6 -
> >> >
> >> > DRILL-4384 - Query profile is missing important information on WebUi -
> >> > Merged
> >> > DRILL-3488/pr 388 (Java 1.8 support) - Merged.
> >> > DRILL-4410/pr 380 (listvector should initiatlize bits...) - Merged
> >> > DRILL-4383/pr 375 (Allow custom configs for S3, Kerberos, etc) -
> Merged
> >> > DRILL-4465/pr 401 (Simplify Calcite parsing & planning integration) -
> >> > Waiting to be merged
> >> >
> >> > DRILL-4281/pr 400 (Drill should support inbound impersonation)
> >> (Sudheesh to
> >> > review)
> >> > DRILL-4372/pr 377(?) (Drill Operators and Functions should correctly
> >> expose
> >> > their types within Calcite.) - Waiting for Aman to review. (Owners:
> >> Hsuan,
> >> > Jinfeng, Aman, Sudheesh)
> >> > DRILL-4313/pr 396  (Improved client randomization. Update JIRA with
> >> > warnings about using the feature ) (Sudheesh to review.)
> >> > DRILL-4437 (and others)/pr 394 (Operator unit test framework). (Parth
> to
> >> > review)
> >> > DRILL-4449/pr 389 (Wrong results when metadata cache is used..) (Aman
> to
> >> > review)
> >> > DRILL-4416/pr 385 (quote path separator) (Owner: Hanifi)
> >> > DRILL-4069/pr 352 Enable RPC thread offload by default (Owner:
> Sudheesh)
> >> >
> >> > Need review -
> >> > DRILL-4375/pr 402 (Fix the maven release profile)
> >> > DRILL-4452/pr 395 (Update Avatica Driver to latest Calcite)
> >> > DRILL-4332/pr 389 (Make vector comparison 

[GitHub] drill pull request: DRILL-4465: Simplify Calcite parsing & plannin...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/401


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4281: Support authorized users to delega...

[yufeldman]

Github user yufeldman commented on the pull request:

https://github.com/apache/drill/pull/400#issuecomment-192171659
  
Couple of general comments:
1. Since you are using Hadoop UGI it probably makes sense to be more 
compliant with Hadoop auth definitions. Which are: "superuser" can proxy for 
"user(s), group(s) and host(s)". May be adding group that can proxy is OK, but 
it is not what is done in Hadoop world today.
-
hadoop.proxyuser.superuser.hostscomma separated hosts from 
which superuser access are allowed to impersonation. * means wildcard.
hadoop.proxyuser.superuser.groups   comma separated groups to which 
users impersonated by superuser belongs. * means wildcard.
-
2. I think what we call here delegate/delegator is a true impersonation, 
what we call "chained impersonation" is kind of opposite of impersonation as it 
is increasing privileges versus restricting them. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---