[GitHub] drill pull request: DRILL-3623: For limit 0 queries, use a shorter...

[hsuanyi]

Github user hsuanyi commented on the pull request:

https://github.com/apache/drill/pull/405#issuecomment-199650953
  
Except for my comments (which are for future improvements and can be filed 
as follow-up jira issues)
LGTM +1 (non-binding)



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] Remove required type

[Jacques Nadeau]

Definitely in support of this. The required type is a huge maintenance and
code complexity nightmare that provides little to no benefit. As you point
out, we can do better performance optimizations though null count
observation since most sources are nullable anyway.
On Mar 21, 2016 7:41 PM, "Steven Phillips"  wrote:

> I have been thinking about this for a while now, and I feel it would be a
> good idea to remove the Required vector types from Drill, and only use the
> Nullable version of vectors. I think this will greatly simplify the code.
> It will also simplify the creation of UDFs. As is, if a function has custom
> null handling (i.e. INTERNAL), the function has to be separately
> implemented for each permutation of nullability of the inputs. But if drill
> data types are always nullable, this wouldn't be a problem.
>
> I don't think there would be much impact on performance. In practice, I
> think the required type is used very rarely. And there are other ways we
> can optimize for when a column is known to have no nulls.
>
> Thoughts?
>

[DISCUSS] Remove required type

[Steven Phillips]

I have been thinking about this for a while now, and I feel it would be a
good idea to remove the Required vector types from Drill, and only use the
Nullable version of vectors. I think this will greatly simplify the code.
It will also simplify the creation of UDFs. As is, if a function has custom
null handling (i.e. INTERNAL), the function has to be separately
implemented for each permutation of nullability of the inputs. But if drill
data types are always nullable, this wouldn't be a problem.

I don't think there would be much impact on performance. In practice, I
think the required type is used very rarely. And there are other ways we
can optimize for when a column is known to have no nulls.

Thoughts?

[GitHub] drill pull request: Map variance() and stddev() to var_samp() and ...

[minji-kim]

Github user minji-kim commented on a diff in the pull request:

https://github.com/apache/drill/pull/437#discussion_r56927626
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/TestAggregationQueries.java ---
@@ -0,0 +1,53 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill;
+
+import org.junit.Test;
+
+public class TestAggregationQueries extends PlanTestBase {
+
+  @Test // DRILL-4521
+  public void testVariance() throws Exception {
--- End diff --

Done!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4237 DRILL-4478 fully implement hash to ...

[jacques-n]

Github user jacques-n commented on a diff in the pull request:

https://github.com/apache/drill/pull/430#discussion_r56925560
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/HashHelper.java 
---
@@ -17,47 +17,77 @@
  */
 package org.apache.drill.exec.expr.fn.impl;
 
+import io.netty.buffer.DrillBuf;
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.common.exceptions.DrillConfigurationException;
+
 import java.nio.ByteBuffer;
 import java.nio.ByteOrder;
 
-public class HashHelper {
+public abstract class HashHelper {
   static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(HashHelper.class);
+  public static final String defaultHashClassName = new 
String("org.apache.drill.exec.expr.fn.impl.MurmurHash3");
+  static final String HASH_CLASS_PROP = "drill.exec.hash.class";
 
+  static String actualHashClassName = defaultHashClassName;
+  static DrillHash hashCall = new MurmurHash3();
+  static {
 
-  /** taken from mahout **/
-  public static int hash(ByteBuffer buf, int seed) {
-// save byte order for later restoration
-
-int m = 0x5bd1e995;
-int r = 24;
+try {
+  DrillConfig config = DrillConfig.create();
+  String configuredClassName = config.getString(HASH_CLASS_PROP);
+  if(configuredClassName != null && configuredClassName != "") {
+actualHashClassName = configuredClassName;
+hashCall = config.getInstanceOf(HASH_CLASS_PROP, DrillHash.class);
+  }
+  logger.debug("HashHelper initializes with " + actualHashClassName);
+}
+catch(Exception ex){
+  logger.error("Could not initialize Hash %s", ex.getMessage());
+}
+  }
 
-int h = seed ^ buf.remaining();
+  public static String getHashClassName(){
+return actualHashClassName;
+  }
 
-while (buf.remaining() >= 4) {
-  int k = buf.getInt();
+  public static int hash32(int val, long seed) {
+double converted = val;
+return hash32(converted, seed);
+  }
+  public static int hash32(long val, long seed) {
+double converted = val;
+return hash32(converted, seed);
+  }
+  public static int hash32(float val, long seed){
+double converted = val;
+return hash32(converted, seed);
+  }
 
-  k *= m;
-  k ^= k >>> r;
-  k *= m;
+  public static int hash32(double val, long seed){
+return hashCall.hash32(val, seed);
+  }
 
-  h *= m;
-  h ^= k;
-}
+  public static  int hash32(int start, int end, DrillBuf buffer, int seed){
+return hashCall.hash32(start, end, buffer, seed);
--- End diff --

Yes, I'm worried about the extra performance hit. I believe we already 
spend a reasonable amount of processing time applying hash functions and have 
considered it an opportunity for improvement. Give your current construction, 
we would need to dereference the field everytime we call the hash function. In 
the past my analysis of assembly out of the JVM is that this isn't typically 
removed. Directly binding to a static function doesn't require this overhead. 
Take a look at the jvm bytecode (or assembly) to see the difference. In 
general, our goal inside individual functions is to avoid indirection as much 
as possible, especially with a hot path such as the hash function. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4237 DRILL-4478 fully implement hash to ...

[chunhui-shi]

Github user chunhui-shi commented on a diff in the pull request:

https://github.com/apache/drill/pull/430#discussion_r56924863
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/HashHelper.java 
---
@@ -17,47 +17,77 @@
  */
 package org.apache.drill.exec.expr.fn.impl;
 
+import io.netty.buffer.DrillBuf;
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.common.exceptions.DrillConfigurationException;
+
 import java.nio.ByteBuffer;
 import java.nio.ByteOrder;
 
-public class HashHelper {
+public abstract class HashHelper {
   static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(HashHelper.class);
+  public static final String defaultHashClassName = new 
String("org.apache.drill.exec.expr.fn.impl.MurmurHash3");
+  static final String HASH_CLASS_PROP = "drill.exec.hash.class";
 
+  static String actualHashClassName = defaultHashClassName;
+  static DrillHash hashCall = new MurmurHash3();
+  static {
 
-  /** taken from mahout **/
-  public static int hash(ByteBuffer buf, int seed) {
-// save byte order for later restoration
-
-int m = 0x5bd1e995;
-int r = 24;
+try {
+  DrillConfig config = DrillConfig.create();
+  String configuredClassName = config.getString(HASH_CLASS_PROP);
+  if(configuredClassName != null && configuredClassName != "") {
+actualHashClassName = configuredClassName;
+hashCall = config.getInstanceOf(HASH_CLASS_PROP, DrillHash.class);
+  }
+  logger.debug("HashHelper initializes with " + actualHashClassName);
+}
+catch(Exception ex){
+  logger.error("Could not initialize Hash %s", ex.getMessage());
+}
+  }
 
-int h = seed ^ buf.remaining();
+  public static String getHashClassName(){
+return actualHashClassName;
+  }
 
-while (buf.remaining() >= 4) {
-  int k = buf.getInt();
+  public static int hash32(int val, long seed) {
+double converted = val;
+return hash32(converted, seed);
+  }
+  public static int hash32(long val, long seed) {
+double converted = val;
+return hash32(converted, seed);
+  }
+  public static int hash32(float val, long seed){
+double converted = val;
+return hash32(converted, seed);
+  }
 
-  k *= m;
-  k ^= k >>> r;
-  k *= m;
+  public static int hash32(double val, long seed){
+return hashCall.hash32(val, seed);
+  }
 
-  h *= m;
-  h ^= k;
-}
+  public static  int hash32(int start, int end, DrillBuf buffer, int seed){
+return hashCall.hash32(start, end, buffer, seed);
--- End diff --

@jacques-n Do you think this indirection may cause some performance 
overhead? The indirection was introduced to allow us to test different hash 
functions without rebuilding the whole thing. 
As to the second ask, can you elaborate "this class dereference to the 
hashCall class field gets eliminated" for me to understand the concern here? 
Thanks. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (DRILL-4526) WebFOCUS Configuration Document

[Andries Engelbrecht (JIRA)]

Andries Engelbrecht created DRILL-4526:
--

 Summary: WebFOCUS Configuration Document
 Key: DRILL-4526
 URL: https://issues.apache.org/jira/browse/DRILL-4526
 Project: Apache Drill
  Issue Type: Improvement
  Components: Documentation
Affects Versions: 1.4.0
Reporter: Andries Engelbrecht
 Attachments: WebFocus 8.2 Configuration with Drill-v1.1.doc

Please add attached configuration document for Information Builders WebFOCUS.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] drill pull request: Map variance() and stddev() to var_samp() and ...

[jacques-n]

Github user jacques-n commented on a diff in the pull request:

https://github.com/apache/drill/pull/437#discussion_r56920816
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/TestAggregationQueries.java ---
@@ -0,0 +1,53 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill;
+
+import org.junit.Test;
+
+public class TestAggregationQueries extends PlanTestBase {
+
+  @Test // DRILL-4521
+  public void testVariance() throws Exception {
--- End diff --

Can you call these ensureVarianceIsAggregateReduced and 
ensureStddevIsAggregateReduced or something similar?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: Map variance() and stddev() to var_samp() and ...

[minji-kim]

GitHub user minji-kim opened a pull request:

https://github.com/apache/drill/pull/437

Map variance() and stddev() to var_samp() and stddev_samp() respectiv…

…ely, and allow for DrillReduceAggregateRule to apply to them

Right now, variance() and stddev() is not mapped to var_samp() and 
stddev_samp() respectively, and they cannot be reduced to sums of squares (by 
DrillReduceAggregateRule).  DrillReduceAggregateRule allows variance/stddev 
functions to be aggregated over 2-phase aggregation.  Without this, 
stddev/variance is always done in a single phase aggregation.  Added test cases 
to verify.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/minji-kim/drill DRILL-4521

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/437.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #437


commit 033786aaaf0d1fd16cac84ba2a2010e610a8e9a4
Author: Minji Kim 
Date:   2016-03-22T00:20:10Z

Map variance() and stddev() to var_samp() and stddev_samp() respectively, 
and allow for DrillReduceAggregateRule to apply to them




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4514 : Add describe schema ...

[jacques-n]

Github user jacques-n commented on a diff in the pull request:

https://github.com/apache/drill/pull/436#discussion_r56919932
  
--- Diff: exec/java-exec/src/main/codegen/includes/parserImpls.ftl ---
@@ -278,3 +278,19 @@ SqlNode SqlRefreshMetadata() :
 }
 }
 
+/**
--- End diff --

What do you think about including this syntax directly in Calcite? 
@julianhyde, do you think this is generic enough to be shared or should we keep 
only in Drill code?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-3623: For limit 0 queries, use a shorter...

[StevenMPhillips]

Github user StevenMPhillips commented on a diff in the pull request:

https://github.com/apache/drill/pull/405#discussion_r56918321
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/limit/TestLimit0.java
 ---
@@ -0,0 +1,677 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.impl.limit;
+
+import com.google.common.collect.Lists;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.drill.BaseTestQuery;
+import org.apache.drill.PlanTestBase;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.common.types.Types;
+import org.joda.time.DateTime;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Ignore;
+import org.junit.Test;
+
+import java.util.List;
+
+public class TestLimit0 extends BaseTestQuery {
+
+  private static final String viewName = "limitZeroEmployeeView";
+
+  private static String wrapLimit0(final String query) {
+return "SELECT * FROM (" + query + ") LZT LIMIT 0";
+  }
+
+  @BeforeClass
+  public static void createView() throws Exception {
+test("USE dfs_test.tmp");
+test(String.format("CREATE OR REPLACE VIEW %s AS SELECT " +
+"CAST(employee_id AS INT) AS employee_id, " +
+"CAST(full_name AS VARCHAR(25)) AS full_name, " +
+"CAST(position_id AS INTEGER) AS position_id, " +
+"CAST(department_id AS BIGINT) AS department_id," +
+"CAST(birth_date AS DATE) AS birth_date, " +
+"CAST(hire_date AS TIMESTAMP) AS hire_date, " +
+"CAST(salary AS DOUBLE) AS salary, " +
+"CAST(salary AS FLOAT) AS fsalary, " +
+"CAST((CASE WHEN marital_status = 'S' THEN true ELSE false END) AS 
BOOLEAN) AS single, " +
+"CAST(education_level AS VARCHAR(60)) AS education_level," +
+"CAST(gender AS CHAR) AS gender " +
+"FROM cp.`employee.json` " +
+"ORDER BY employee_id " +
+"LIMIT 1;", viewName));
+// { "employee_id":1,"full_name":"Sheri 
Nowmer","first_name":"Sheri","last_name":"Nowmer","position_id":1,
+// 
"position_title":"President","store_id":0,"department_id":1,"birth_date":"1961-08-26",
+// "hire_date":"1994-12-01 
00:00:00.0","end_date":null,"salary":8.,"supervisor_id":0,
+// "education_level":"Graduate 
Degree","marital_status":"S","gender":"F","management_role":"Senior Management" 
}
+  }
+
+  @AfterClass
+  public static void tearDownView() throws Exception {
+test("DROP VIEW " + viewName + ";");
+  }
+
+  //  SIMPLE QUERIES 
+
+  @Test
+  public void infoSchema() throws Exception {
+testBuilder()
+.sqlQuery(String.format("DESCRIBE %s", viewName))
+.unOrdered()
+.baselineColumns("COLUMN_NAME", "DATA_TYPE", "IS_NULLABLE")
+.baselineValues("employee_id", "INTEGER", "YES")
+.baselineValues("full_name", "CHARACTER VARYING", "YES")
+.baselineValues("position_id", "INTEGER", "YES")
+.baselineValues("department_id", "BIGINT", "YES")
+.baselineValues("birth_date", "DATE", "YES")
+.baselineValues("hire_date", "TIMESTAMP", "YES")
+.baselineValues("salary", "DOUBLE", "YES")
+.baselineValues("fsalary", "FLOAT", "YES")
+.baselineValues("single", "BOOLEAN", "NO")
+.baselineValues("education_level", "CHARACTER VARYING", "YES")
+.baselineValues("gender", "CHARACTER", "YES")
+.go();
+  }
+
+  @Test
+  @Ignore("DateTime timezone error needs to be fixed.")
+  public void simpleSelect() throws Exception {
+testBuilder()
+.sqlQuery(String.format("SELECT * FROM %s", viewName))
+.ordered()
+.baselineColumns("employee_id", "full_name", "position_id", 
"depa

[GitHub] drill pull request: DRILL-4514 : Add describe schema ...

[hnfgns]

Github user hnfgns commented on a diff in the pull request:

https://github.com/apache/drill/pull/436#discussion_r56915385
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/parser/SqlDescribeSchema.java
 ---
@@ -0,0 +1,82 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql.parser;
+
+import org.apache.calcite.sql.SqlCall;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlLiteral;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperator;
+import org.apache.calcite.sql.SqlSpecialOperator;
+import org.apache.calcite.sql.SqlWriter;
+import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.drill.exec.planner.sql.handlers.AbstractSqlHandler;
+import org.apache.drill.exec.planner.sql.handlers.DescribeSchemaHandler;
+import org.apache.drill.exec.planner.sql.handlers.SqlHandlerConfig;
+
+import java.util.Collections;
+import java.util.List;
+
+/**
+ * Sql parse tree node to represent statement:
+ * SHOW FILES [{FROM | IN} db_name] [LIKE 'pattern' | WHERE expr]
+ */
+public class SqlDescribeSchema extends DrillSqlCall {
+
+  private final SqlIdentifier schema;
+
+  public static final SqlSpecialOperator OPERATOR =
+  new SqlSpecialOperator("DESCRIBE_SCHEMA", SqlKind.OTHER) {
+@Override
+public SqlCall createCall(SqlLiteral functionQualifier, 
SqlParserPos pos, SqlNode... operands) {
+  return new SqlDescribeSchema(pos, (SqlIdentifier) operands[0]);
+}
+  };
+
+  public SqlDescribeSchema(SqlParserPos pos, SqlIdentifier schema) {
+super(pos);
+this.schema = schema;
+assert schema != null;
--- End diff --

Sure. I meant we should likely use an explicit null-checking pattern like 
`this.schema = Preconditions.checkNotNull(schema, "msg")` instead of an 
assertion here. 

ps: Guava's Preconditions.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (DRILL-4525) Query with BETWEEN clause on Date and Timestamp values fails with Validation Error

[Abhishek Girish (JIRA)]

Abhishek Girish created DRILL-4525:
--

 Summary: Query with BETWEEN clause on Date and Timestamp values 
fails with Validation Error
 Key: DRILL-4525
 URL: https://issues.apache.org/jira/browse/DRILL-4525
 Project: Apache Drill
  Issue Type: Bug
  Components: Query Planning & Optimization
Reporter: Abhishek Girish
Assignee: Sean Hsuan-Yi Chu
Priority: Critical


Query: (simplified variant of TPC-DS Query37)
{code}
SELECT
   *
FROM   
   date_dim
WHERE   
   d_date BETWEEN Cast('1999-03-06' AS DATE) AND  (
  Cast('1999-03-06' AS DATE) + INTERVAL '60' day)
LIMIT 10;
{code}
Error:
{code}
Error: VALIDATION ERROR: From line 6, column 8 to line 7, column 64: Cannot 
apply 'BETWEEN ASYMMETRIC' to arguments of type ' BETWEEN ASYMMETRIC 
 AND '. Supported form(s): ' BETWEEN 
 AND '
SQL Query null
[Error Id: 223fb37c-f561-4a37-9283-871dc6f4d6d0 on abhi2:31010] (state=,code=0)
{code}

This is a regression from 1.6.0. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] drill pull request: DRILL-4237 DRILL-4478 fully implement hash to ...

[jacques-n]

Github user jacques-n commented on a diff in the pull request:

https://github.com/apache/drill/pull/430#discussion_r56907606
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/HashHelper.java 
---
@@ -17,47 +17,77 @@
  */
 package org.apache.drill.exec.expr.fn.impl;
 
+import io.netty.buffer.DrillBuf;
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.common.exceptions.DrillConfigurationException;
+
 import java.nio.ByteBuffer;
 import java.nio.ByteOrder;
 
-public class HashHelper {
+public abstract class HashHelper {
   static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(HashHelper.class);
+  public static final String defaultHashClassName = new 
String("org.apache.drill.exec.expr.fn.impl.MurmurHash3");
+  static final String HASH_CLASS_PROP = "drill.exec.hash.class";
 
+  static String actualHashClassName = defaultHashClassName;
+  static DrillHash hashCall = new MurmurHash3();
+  static {
 
-  /** taken from mahout **/
-  public static int hash(ByteBuffer buf, int seed) {
-// save byte order for later restoration
-
-int m = 0x5bd1e995;
-int r = 24;
+try {
+  DrillConfig config = DrillConfig.create();
+  String configuredClassName = config.getString(HASH_CLASS_PROP);
+  if(configuredClassName != null && configuredClassName != "") {
+actualHashClassName = configuredClassName;
+hashCall = config.getInstanceOf(HASH_CLASS_PROP, DrillHash.class);
+  }
+  logger.debug("HashHelper initializes with " + actualHashClassName);
+}
+catch(Exception ex){
+  logger.error("Could not initialize Hash %s", ex.getMessage());
+}
+  }
 
-int h = seed ^ buf.remaining();
+  public static String getHashClassName(){
+return actualHashClassName;
+  }
 
-while (buf.remaining() >= 4) {
-  int k = buf.getInt();
+  public static int hash32(int val, long seed) {
+double converted = val;
+return hash32(converted, seed);
+  }
+  public static int hash32(long val, long seed) {
+double converted = val;
+return hash32(converted, seed);
+  }
+  public static int hash32(float val, long seed){
+double converted = val;
+return hash32(converted, seed);
+  }
 
-  k *= m;
-  k ^= k >>> r;
-  k *= m;
+  public static int hash32(double val, long seed){
+return hashCall.hash32(val, seed);
+  }
 
-  h *= m;
-  h ^= k;
-}
+  public static  int hash32(int start, int end, DrillBuf buffer, int seed){
+return hashCall.hash32(start, end, buffer, seed);
--- End diff --

It seems like we shouldn't do this indirection. Have you confirmed that 
this class dereference to the hashCall class field gets eliminated?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Moving to HBase 1.1 [DRILL-4199]

[Jacques Nadeau]

+1

--
Jacques Nadeau
CTO and Co-Founder, Dremio

On Mon, Mar 21, 2016 at 1:18 PM, Aditya  wrote:

> Hi,
>
> HBase has moved to 1.1 branch as their latest stable release[1] and since
> it is wire compatible with 0.98 releases, I'd like to propose that Drill
> updates its supported HBase release to 1.1.
>
> Essentially, it means that we update the HBase clients bundled with Drill
> distribution to latest stable version of 1.1 branch. I do not expect any
> code change.
>
> I have assigned DRILL-4199 to myself and unless someone has a reason to not
> to, I'd like to move to HBase 1.1 in Drill 1.7 release.
>
> aditya...
>
> [1] https://dist.apache.org/repos/dist/release/hbase/stable
> [2] https://issues.apache.org/jira/browse/DRILL-4199
>

[jira] [Created] (DRILL-4524) case-sensitive view names

[Bridget Bevens (JIRA)]

Bridget Bevens created DRILL-4524:
-

 Summary: case-sensitive view names
 Key: DRILL-4524
 URL: https://issues.apache.org/jira/browse/DRILL-4524
 Project: Apache Drill
  Issue Type: Bug
  Components: Documentation
Reporter: Bridget Bevens
Assignee: Bridget Bevens


I think that part of the doc needs to be updated. Table names & view names are 
case-sensitive. Also the doc section about double-quoting names is in-correct. 
Using double quotes gives us a SQL parse error. 

On Mon, Mar 21, 2016 at 1:27 PM, Christopher Matta  wrote:
I’m experiencing case-sensitive view names, the drill documentation here:
https://drill.apache.org/docs/lexical-structure/#case-sensitivity doesn't
explicitly state weather view names are case-insensitive:

0: jdbc:drill:> select state, count(1) from
`yelp/yelp_academic_dataset_business.json` group by state;
++-+
| state  | EXPR$1  |
++-+
| AZ | 15582   |
| CA | 2   |
| SC | 1   |
++-+
3 rows selected (0.533 seconds)
0: jdbc:drill:> create or replace view BUSINESS_COUNT as select state,
count(1) from `yelp/yelp_academic_dataset_business.json` group by
state;
+---+---+
|  ok   |summary
 |
+---+---+
| true  | View 'BUSINESS_COUNT' created successfully in
'maprfs.cmatta' schema  |
+---+---+
1 row selected (0.36 seconds)
0: jdbc:drill:> select * from business_count;
Error: VALIDATION ERROR: From line 1, column 15 to line 1, column 28:
Table 'business_count' not found

[Error Id: f2371e7b-f3de-4aef-9879-bae10fddea6b on
se-node13.se.lab:31010] (state=,code=0)
0: jdbc:drill:> select * from BUSINESS_COUNT;
++-+
| state  | EXPR$1  |
++-+
| AZ | 15582   |
| CA | 2   |
| SC | 1   |
++-+
3 rows selected (0.554 seconds)
0: jdbc:drill:>

I see this JIRA, but it’s related to the jdbc storage plugin:
https://issues.apache.org/jira/browse/DRILL-4458

Is this expected behavior?

Chris Matta

cma...@mapr.com
215-701-3146
​



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] drill pull request: DRILL-4237 DRILL-4478 fully implement hash to ...

[amansinha100]

Github user amansinha100 commented on a diff in the pull request:

https://github.com/apache/drill/pull/430#discussion_r56901020
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/MurmurHash3.java
 ---
@@ -0,0 +1,264 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+
+package org.apache.drill.exec.expr.fn.impl;
+
+import io.netty.buffer.DrillBuf;
+import io.netty.util.internal.PlatformDependent;
+
+
+/**
+ *
+ * MurmurHash3 was written by Austin Appleby, and is placed in the public
+ * domain.
+ * See http://smhasher.googlecode.com/svn/trunk/MurmurHash3.cpp
+ * MurmurHash3_x64_128
+ * MurmurHash3_x86_32
+ */
+public final class MurmurHash3 extends DrillHash{
+
+  public final class LongPair {
+public long val1;
+public long val2;
+  }
+
+  public static final long fmix64(long k) {
+k ^= k >>> 33;
+k *= 0xff51afd7ed558ccdL;
+k ^= k >>> 33;
+k *= 0xc4ceb9fe1a85ec53L;
+k ^= k >>> 33;
+return k;
+  }
+
+
+
+  /*
+  Take 64 bit of murmur3_128's output
+   */
+  public static long murmur3_64(long bStart, long bEnd, DrillBuf buffer, 
int seed) {
+
+long h1 = seed & 0xL;
+long h2 = seed & 0xL;
+
+final long c1 = 0x87c37b91114253d5L;
+final long c2 = 0x4cf5ad432745937fL;
+long start = buffer.memoryAddress() + bStart;
+long end = buffer.memoryAddress() + bEnd;
+long length = end - start;
+long roundedEnd = start + ( length & 0xFFF0);  // round down to 16 
byte block
+for (long i=start; i

[GitHub] drill pull request: DRILL-4237 DRILL-4478 fully implement hash to ...

[amansinha100]

Github user amansinha100 commented on a diff in the pull request:

https://github.com/apache/drill/pull/430#discussion_r56900515
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/MurmurHash3.java
 ---
@@ -0,0 +1,264 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+
+package org.apache.drill.exec.expr.fn.impl;
+
+import io.netty.buffer.DrillBuf;
+import io.netty.util.internal.PlatformDependent;
+
+
+/**
+ *
+ * MurmurHash3 was written by Austin Appleby, and is placed in the public
+ * domain.
+ * See http://smhasher.googlecode.com/svn/trunk/MurmurHash3.cpp
+ * MurmurHash3_x64_128
+ * MurmurHash3_x86_32
+ */
+public final class MurmurHash3 extends DrillHash{
+
+  public final class LongPair {
+public long val1;
+public long val2;
+  }
+
+  public static final long fmix64(long k) {
+k ^= k >>> 33;
+k *= 0xff51afd7ed558ccdL;
+k ^= k >>> 33;
+k *= 0xc4ceb9fe1a85ec53L;
+k ^= k >>> 33;
+return k;
+  }
+
+
+
+  /*
+  Take 64 bit of murmur3_128's output
+   */
+  public static long murmur3_64(long bStart, long bEnd, DrillBuf buffer, 
int seed) {
+
+long h1 = seed & 0xL;
+long h2 = seed & 0xL;
+
+final long c1 = 0x87c37b91114253d5L;
+final long c2 = 0x4cf5ad432745937fL;
+long start = buffer.memoryAddress() + bStart;
+long end = buffer.memoryAddress() + bEnd;
+long length = end - start;
+long roundedEnd = start + ( length & 0xFFF0);  // round down to 16 
byte block
+for (long i=start; i

[GitHub] drill pull request: DRILL-4237 DRILL-4478 fully implement hash to ...

[amansinha100]

Github user amansinha100 commented on a diff in the pull request:

https://github.com/apache/drill/pull/430#discussion_r56900362
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/MurmurHash3.java
 ---
@@ -0,0 +1,264 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+
+package org.apache.drill.exec.expr.fn.impl;
+
+import io.netty.buffer.DrillBuf;
+import io.netty.util.internal.PlatformDependent;
+
+
+/**
+ *
+ * MurmurHash3 was written by Austin Appleby, and is placed in the public
+ * domain.
+ * See http://smhasher.googlecode.com/svn/trunk/MurmurHash3.cpp
+ * MurmurHash3_x64_128
+ * MurmurHash3_x86_32
+ */
+public final class MurmurHash3 extends DrillHash{
+
+  public final class LongPair {
+public long val1;
+public long val2;
+  }
+
+  public static final long fmix64(long k) {
+k ^= k >>> 33;
+k *= 0xff51afd7ed558ccdL;
+k ^= k >>> 33;
+k *= 0xc4ceb9fe1a85ec53L;
+k ^= k >>> 33;
+return k;
+  }
+
+
+
+  /*
+  Take 64 bit of murmur3_128's output
+   */
+  public static long murmur3_64(long bStart, long bEnd, DrillBuf buffer, 
int seed) {
+
+long h1 = seed & 0xL;
+long h2 = seed & 0xL;
+
+final long c1 = 0x87c37b91114253d5L;
+final long c2 = 0x4cf5ad432745937fL;
+long start = buffer.memoryAddress() + bStart;
+long end = buffer.memoryAddress() + bEnd;
+long length = end - start;
+long roundedEnd = start + ( length & 0xFFF0);  // round down to 16 
byte block
+for (long i=start; i

[GitHub] drill pull request: DRILL-4237 DRILL-4478 fully implement hash to ...

[amansinha100]

Github user amansinha100 commented on a diff in the pull request:

https://github.com/apache/drill/pull/430#discussion_r56900294
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/MurmurHash3.java
 ---
@@ -0,0 +1,264 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+
+package org.apache.drill.exec.expr.fn.impl;
+
+import io.netty.buffer.DrillBuf;
+import io.netty.util.internal.PlatformDependent;
+
+
+/**
+ *
+ * MurmurHash3 was written by Austin Appleby, and is placed in the public
+ * domain.
+ * See http://smhasher.googlecode.com/svn/trunk/MurmurHash3.cpp
+ * MurmurHash3_x64_128
+ * MurmurHash3_x86_32
+ */
+public final class MurmurHash3 extends DrillHash{
+
+  public final class LongPair {
+public long val1;
+public long val2;
+  }
+
+  public static final long fmix64(long k) {
+k ^= k >>> 33;
+k *= 0xff51afd7ed558ccdL;
+k ^= k >>> 33;
+k *= 0xc4ceb9fe1a85ec53L;
+k ^= k >>> 33;
+return k;
+  }
+
+
+
+  /*
+  Take 64 bit of murmur3_128's output
+   */
+  public static long murmur3_64(long bStart, long bEnd, DrillBuf buffer, 
int seed) {
+
+long h1 = seed & 0xL;
+long h2 = seed & 0xL;
+
+final long c1 = 0x87c37b91114253d5L;
+final long c2 = 0x4cf5ad432745937fL;
+long start = buffer.memoryAddress() + bStart;
+long end = buffer.memoryAddress() + bEnd;
+long length = end - start;
+long roundedEnd = start + ( length & 0xFFF0);  // round down to 16 
byte block
+for (long i=start; i

[GitHub] drill pull request: DRILL-4237 DRILL-4478 fully implement hash to ...

[amansinha100]

Github user amansinha100 commented on a diff in the pull request:

https://github.com/apache/drill/pull/430#discussion_r56900085
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/MurmurHash3.java
 ---
@@ -0,0 +1,264 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+
+package org.apache.drill.exec.expr.fn.impl;
+
+import io.netty.buffer.DrillBuf;
+import io.netty.util.internal.PlatformDependent;
+
+
+/**
+ *
+ * MurmurHash3 was written by Austin Appleby, and is placed in the public
+ * domain.
+ * See http://smhasher.googlecode.com/svn/trunk/MurmurHash3.cpp
+ * MurmurHash3_x64_128
+ * MurmurHash3_x86_32
+ */
+public final class MurmurHash3 extends DrillHash{
+
+  public final class LongPair {
+public long val1;
+public long val2;
+  }
+
+  public static final long fmix64(long k) {
+k ^= k >>> 33;
+k *= 0xff51afd7ed558ccdL;
+k ^= k >>> 33;
+k *= 0xc4ceb9fe1a85ec53L;
+k ^= k >>> 33;
+return k;
+  }
+
+
+
+  /*
+  Take 64 bit of murmur3_128's output
+   */
+  public static long murmur3_64(long bStart, long bEnd, DrillBuf buffer, 
int seed) {
+
+long h1 = seed & 0xL;
+long h2 = seed & 0xL;
+
+final long c1 = 0x87c37b91114253d5L;
+final long c2 = 0x4cf5ad432745937fL;
+long start = buffer.memoryAddress() + bStart;
+long end = buffer.memoryAddress() + bEnd;
+long length = end - start;
+long roundedEnd = start + ( length & 0xFFF0);  // round down to 16 
byte block
+for (long i=start; i

[GitHub] drill pull request: DRILL-4237 DRILL-4478 fully implement hash to ...

[amansinha100]

Github user amansinha100 commented on a diff in the pull request:

https://github.com/apache/drill/pull/430#discussion_r56899651
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/MurmurHash3.java
 ---
@@ -0,0 +1,264 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+
+package org.apache.drill.exec.expr.fn.impl;
+
+import io.netty.buffer.DrillBuf;
+import io.netty.util.internal.PlatformDependent;
+
+
+/**
+ *
+ * MurmurHash3 was written by Austin Appleby, and is placed in the public
+ * domain.
+ * See http://smhasher.googlecode.com/svn/trunk/MurmurHash3.cpp
+ * MurmurHash3_x64_128
+ * MurmurHash3_x86_32
+ */
+public final class MurmurHash3 extends DrillHash{
+
+  public final class LongPair {
--- End diff --

The LongPair class is not getting used anywhere ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4237 DRILL-4478 fully implement hash to ...

[amansinha100]

Github user amansinha100 commented on a diff in the pull request:

https://github.com/apache/drill/pull/430#discussion_r56899529
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/HashHelper.java 
---
@@ -17,47 +17,77 @@
  */
 package org.apache.drill.exec.expr.fn.impl;
 
+import io.netty.buffer.DrillBuf;
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.common.exceptions.DrillConfigurationException;
+
 import java.nio.ByteBuffer;
 import java.nio.ByteOrder;
 
-public class HashHelper {
+public abstract class HashHelper {
   static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(HashHelper.class);
+  public static final String defaultHashClassName = new 
String("org.apache.drill.exec.expr.fn.impl.MurmurHash3");
+  static final String HASH_CLASS_PROP = "drill.exec.hash.class";
 
+  static String actualHashClassName = defaultHashClassName;
+  static DrillHash hashCall = new MurmurHash3();
+  static {
 
-  /** taken from mahout **/
-  public static int hash(ByteBuffer buf, int seed) {
--- End diff --

I think we should keep the mahout hash function for future use...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4237 DRILL-4478 fully implement hash to ...

[amansinha100]

Github user amansinha100 commented on a diff in the pull request:

https://github.com/apache/drill/pull/430#discussion_r56899331
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/MurmurHash3.java
 ---
@@ -0,0 +1,264 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+
+package org.apache.drill.exec.expr.fn.impl;
+
+import io.netty.buffer.DrillBuf;
+import io.netty.util.internal.PlatformDependent;
+
+
+/**
+ *
+ * MurmurHash3 was written by Austin Appleby, and is placed in the public
+ * domain.
+ * See http://smhasher.googlecode.com/svn/trunk/MurmurHash3.cpp
+ * MurmurHash3_x64_128
+ * MurmurHash3_x86_32
+ */
+public final class MurmurHash3 extends DrillHash{
+
+  public final class LongPair {
+public long val1;
+public long val2;
+  }
+
+  public static final long fmix64(long k) {
+k ^= k >>> 33;
+k *= 0xff51afd7ed558ccdL;
+k ^= k >>> 33;
+k *= 0xc4ceb9fe1a85ec53L;
+k ^= k >>> 33;
+return k;
+  }
+
+
+
+  /*
+  Take 64 bit of murmur3_128's output
+   */
+  public static long murmur3_64(long bStart, long bEnd, DrillBuf buffer, 
int seed) {
+
+long h1 = seed & 0xL;
+long h2 = seed & 0xL;
+
+final long c1 = 0x87c37b91114253d5L;
+final long c2 = 0x4cf5ad432745937fL;
+long start = buffer.memoryAddress() + bStart;
+long end = buffer.memoryAddress() + bEnd;
+long length = end - start;
+long roundedEnd = start + ( length & 0xFFF0);  // round down to 16 
byte block
+for (long i=start; i

[GitHub] drill pull request: DRILL-4237 DRILL-4478 fully implement hash to ...

[amansinha100]

Github user amansinha100 commented on a diff in the pull request:

https://github.com/apache/drill/pull/430#discussion_r56899122
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/MurmurHash3.java
 ---
@@ -0,0 +1,264 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+
+package org.apache.drill.exec.expr.fn.impl;
+
+import io.netty.buffer.DrillBuf;
+import io.netty.util.internal.PlatformDependent;
+
+
+/**
+ *
+ * MurmurHash3 was written by Austin Appleby, and is placed in the public
+ * domain.
+ * See http://smhasher.googlecode.com/svn/trunk/MurmurHash3.cpp
+ * MurmurHash3_x64_128
+ * MurmurHash3_x86_32
+ */
+public final class MurmurHash3 extends DrillHash{
+
+  public final class LongPair {
+public long val1;
+public long val2;
+  }
+
+  public static final long fmix64(long k) {
+k ^= k >>> 33;
+k *= 0xff51afd7ed558ccdL;
+k ^= k >>> 33;
+k *= 0xc4ceb9fe1a85ec53L;
+k ^= k >>> 33;
+return k;
+  }
+
+
+
+  /*
+  Take 64 bit of murmur3_128's output
+   */
+  public static long murmur3_64(long bStart, long bEnd, DrillBuf buffer, 
int seed) {
+
+long h1 = seed & 0xL;
+long h2 = seed & 0xL;
+
+final long c1 = 0x87c37b91114253d5L;
+final long c2 = 0x4cf5ad432745937fL;
+long start = buffer.memoryAddress() + bStart;
+long end = buffer.memoryAddress() + bEnd;
+long length = end - start;
+long roundedEnd = start + ( length & 0xFFF0);  // round down to 16 
byte block
+for (long i=start; i

[jira] [Created] (DRILL-4523) Add the IP address if we enable debug for org.apache.drill.exec.coord.zk

[Arina Ielchiieva (JIRA)]

Arina Ielchiieva created DRILL-4523:
---

 Summary: Add the IP address if we enable debug for 
org.apache.drill.exec.coord.zk
 Key: DRILL-4523
 URL: https://issues.apache.org/jira/browse/DRILL-4523
 Project: Apache Drill
  Issue Type: Improvement
  Components:  Server
Affects Versions: 1.6.0
Reporter: Arina Ielchiieva
Assignee: Arina Ielchiieva
 Fix For: 1.7.0


If we enable debug for org.apache.drill.exec.coord.zk in logback.xml, we only 
get the hostname and ports information. For example:
{code}
2015-11-04 19:47:02,927 [ServiceCache-0] DEBUG 
o.a.d.e.c.zk.ZKClusterCoordinator - Cache changed, updating.
2015-11-04 19:47:02,932 [ServiceCache-0] DEBUG 
o.a.d.e.c.zk.ZKClusterCoordinator - Active drillbit set changed.  Now includes 
2 total bits.  New active drillbits:
 h3.poc.com:31010:31011:31012
 h2.poc.com:31010:31011:31012
{code}

We need to know the IP address of each hostname to do further troubleshooting.

Imagine if any drillbit registers itself as "localhost.localdomain" in 
zookeeper, we will never know where it comes from. Enabling IP address tracking 
can help this case.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] drill pull request: DRILL-4514 : Add describe schema ...

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/436#discussion_r56893518
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/parser/SqlDescribeSchema.java
 ---
@@ -0,0 +1,82 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql.parser;
+
+import org.apache.calcite.sql.SqlCall;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlLiteral;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperator;
+import org.apache.calcite.sql.SqlSpecialOperator;
+import org.apache.calcite.sql.SqlWriter;
+import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.drill.exec.planner.sql.handlers.AbstractSqlHandler;
+import org.apache.drill.exec.planner.sql.handlers.DescribeSchemaHandler;
+import org.apache.drill.exec.planner.sql.handlers.SqlHandlerConfig;
+
+import java.util.Collections;
+import java.util.List;
+
+/**
+ * Sql parse tree node to represent statement:
+ * SHOW FILES [{FROM | IN} db_name] [LIKE 'pattern' | WHERE expr]
+ */
+public class SqlDescribeSchema extends DrillSqlCall {
+
+  private final SqlIdentifier schema;
+
+  public static final SqlSpecialOperator OPERATOR =
+  new SqlSpecialOperator("DESCRIBE_SCHEMA", SqlKind.OTHER) {
+@Override
+public SqlCall createCall(SqlLiteral functionQualifier, 
SqlParserPos pos, SqlNode... operands) {
+  return new SqlDescribeSchema(pos, (SqlIdentifier) operands[0]);
+}
+  };
+
+  public SqlDescribeSchema(SqlParserPos pos, SqlIdentifier schema) {
+super(pos);
+this.schema = schema;
+assert schema != null;
--- End diff --

Not sure what you mean. Could you please expand on your comment?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4514 : Add describe schema ...

[hnfgns]

Github user hnfgns commented on a diff in the pull request:

https://github.com/apache/drill/pull/436#discussion_r56891914
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/parser/SqlDescribeSchema.java
 ---
@@ -0,0 +1,82 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql.parser;
+
+import org.apache.calcite.sql.SqlCall;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlLiteral;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperator;
+import org.apache.calcite.sql.SqlSpecialOperator;
+import org.apache.calcite.sql.SqlWriter;
+import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.drill.exec.planner.sql.handlers.AbstractSqlHandler;
+import org.apache.drill.exec.planner.sql.handlers.DescribeSchemaHandler;
+import org.apache.drill.exec.planner.sql.handlers.SqlHandlerConfig;
+
+import java.util.Collections;
+import java.util.List;
+
+/**
+ * Sql parse tree node to represent statement:
+ * SHOW FILES [{FROM | IN} db_name] [LIKE 'pattern' | WHERE expr]
+ */
+public class SqlDescribeSchema extends DrillSqlCall {
+
+  private final SqlIdentifier schema;
+
+  public static final SqlSpecialOperator OPERATOR =
+  new SqlSpecialOperator("DESCRIBE_SCHEMA", SqlKind.OTHER) {
+@Override
+public SqlCall createCall(SqlLiteral functionQualifier, 
SqlParserPos pos, SqlNode... operands) {
+  return new SqlDescribeSchema(pos, (SqlIdentifier) operands[0]);
+}
+  };
+
+  public SqlDescribeSchema(SqlParserPos pos, SqlIdentifier schema) {
+super(pos);
+this.schema = schema;
+assert schema != null;
--- End diff --

What about a precondition here?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4514 : Add describe schema ...

[hnfgns]

Github user hnfgns commented on a diff in the pull request:

https://github.com/apache/drill/pull/436#discussion_r56891737
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/DescribeSchemaCommandResult.java
 ---
@@ -0,0 +1,30 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql.handlers;
+
+public class DescribeSchemaCommandResult {
+
+  public String name;
--- End diff --

You may want to make these final.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Moving to HBase 1.1 [DRILL-4199]

[Jason Altekruse]

With the recent issues that have been discussed on other threads related to
correctness issues when using our current client I agree we should upgrade.
+1

Jason Altekruse
Software Engineer at Dremio
Apache Drill Committer

On Mon, Mar 21, 2016 at 1:18 PM, Aditya  wrote:

> Hi,
>
> HBase has moved to 1.1 branch as their latest stable release[1] and since
> it is wire compatible with 0.98 releases, I'd like to propose that Drill
> updates its supported HBase release to 1.1.
>
> Essentially, it means that we update the HBase clients bundled with Drill
> distribution to latest stable version of 1.1 branch. I do not expect any
> code change.
>
> I have assigned DRILL-4199 to myself and unless someone has a reason to not
> to, I'd like to move to HBase 1.1 in Drill 1.7 release.
>
> aditya...
>
> [1] https://dist.apache.org/repos/dist/release/hbase/stable
> [2] https://issues.apache.org/jira/browse/DRILL-4199
>

Moving to HBase 1.1 [DRILL-4199]

[Aditya]

Hi,

HBase has moved to 1.1 branch as their latest stable release[1] and since
it is wire compatible with 0.98 releases, I'd like to propose that Drill
updates its supported HBase release to 1.1.

Essentially, it means that we update the HBase clients bundled with Drill
distribution to latest stable version of 1.1 branch. I do not expect any
code change.

I have assigned DRILL-4199 to myself and unless someone has a reason to not
to, I'd like to move to HBase 1.1 in Drill 1.7 release.

aditya...

[1] https://dist.apache.org/repos/dist/release/hbase/stable
[2] https://issues.apache.org/jira/browse/DRILL-4199

Re: Drill query does not return all results from HBase

[Aditya]

I did not see any issue when running with HBase 0.98.7 client bundled with
Drill against HBase 1.1 servers.

I have just assigned DRILL-4199[1] to myself to evaluate moving to HBase
1.1 in next Drill release.

[1] https://issues.apache.org/jira/browse/DRILL-4199

On Mon, Mar 21, 2016 at 12:13 PM, Kevin Verhoeven  wrote:

> Aditya,
>
> Looking into the bug we read that the behavior will still occur if the
> hbase-client version does not include the fix (between a 0.98 client and
> 1.0 server). The hbase-client used by Drill under jars/3rdparty is
> hbase-client-0.98.7-hadoop2.jar which does not include the fix. I updated
> the hbase-client jar with hbase-client-0.98.17-hadoop2.jar, but I receive a
> java.lang.NoClassDefFoundError error. Are you able to test Drill with an
> updated hbase-client jar against CDH? Here is the error I received:
>
> 2016-03-21 18:58:36,874 [USER-rpc-event-queue] ERROR
> o.a.d.exec.server.rest.QueryWrapper - Query Failed
> org.apache.drill.common.exceptions.UserRemoteException: DATA_READ ERROR:
> Failure while loading table test6c in database hbase.
> Message:  com.google.protobuf.ServiceException:
> java.lang.NoClassDefFoundError: com/yammer/metrics/core/Gauge
>
> at
> org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:119)
> [drill-java-exec-1.4.0.jar:1.4.0]
> at
> org.apache.drill.exec.rpc.user.UserClient.handleReponse(UserClient.java:113)
> [drill-java-exec-1.4.0.jar:1.4.0]
> at
> org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:46)
> [drill-rpc-1.4.0.jar:1.4.0]
> at
> org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:31)
> [drill-rpc-1.4.0.jar:1.4.0]
> at org.apache.drill.exec.rpc.RpcBus.handle(RpcBus.java:69)
> [drill-rpc-1.4.0.jar:1.4.0]
> at
> org.apache.drill.exec.rpc.RpcBus$RequestEvent.run(RpcBus.java:400)
> [drill-rpc-1.4.0.jar:1.4.0]
> at
> org.apache.drill.common.SerializedExecutor$RunnableProcessor.run(SerializedExecutor.java:105)
> [drill-rpc-1.4.0.jar:1.4.0]
> at
> org.apache.drill.exec.rpc.RpcBus$SameExecutor.execute(RpcBus.java:264)
> [drill-rpc-1.4.0.jar:1.4.0]
> at
> org.apache.drill.common.SerializedExecutor.execute(SerializedExecutor.java:142)
> [drill-rpc-1.4.0.jar:1.4.0]
> at
> org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:298)
> [drill-rpc-1.4.0.jar:1.4.0]
> at
> org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:269)
> [drill-rpc-1.4.0.jar:1.4.0]
> at
> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89)
> [netty-codec-4.0.27.Final.jar:4.0.27.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
> [netty-transport-4.0.27.Final.jar:4.0.27.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
> [netty-transport-4.0.27.Final.jar:4.0.27.Final]
> at
> io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:254)
> [netty-handler-4.0.27.Final.jar:4.0.27.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
> [netty-transport-4.0.27.Final.jar:4.0.27.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
> [netty-transport-4.0.27.Final.jar:4.0.27.Final]
> at
> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
> [netty-codec-4.0.27.Final.jar:4.0.27.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
> [netty-transport-4.0.27.Final.jar:4.0.27.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
> [netty-transport-4.0.27.Final.jar:4.0.27.Final]
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242)
> [netty-codec-4.0.27.Final.jar:4.0.27.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
> [netty-transport-4.0.27.Final.jar:4.0.27.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
> [netty-transport-4.0.27.Final.jar:4.0.27.Final]
> at
> io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
> [netty-transport-4.0.27.Final.jar:4.0.27.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
> [netty-transport-4.0.27.Final.jar:4.0.27.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
> [netty-transpo

[jira] [Resolved] (DRILL-4515) Fix an documentation error related to text file splitting

[Bridget Bevens (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bridget Bevens resolved DRILL-4515.
---
Resolution: Fixed

Updated the text: 
http://drill.apache.org/docs/text-files-csv-tsv-psv/#use-a-distributed-file-system

> Fix an documentation error related to text file splitting
> -
>
> Key: DRILL-4515
> URL: https://issues.apache.org/jira/browse/DRILL-4515
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Documentation
>Reporter: Deneche A. Hakim
>Assignee: Bridget Bevens
>
> In this documentation page:
> http://drill.apache.org/docs/text-files-csv-tsv-psv/
> We can read the following:
> {quote}
> Using a distributed file system, such as HDFS, instead of a local file system 
> to query the files also improves performance because currently Drill does not 
> split files on block splits.
> {quote}
> Drill actually attempts to split files on block boundaries when running on 
> HDFS and MapRFS



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Drill query does not return all results from HBase

[Aditya]

Since I suspected that it was a bug in HBase, I tried it with the original
version you reported in the first post in this thread, i.e. CDH 5.4.3.

If it was back-ported to 5.4.7, upgrading should fix this issue.

On Mon, Mar 21, 2016 at 10:18 AM, Kevin Verhoeven  wrote:

> Aditya,
>
>
>
> Thank you for your help. What version of CDH are you running? I contacted
> Cloudera and they stated that bug HBASE-13262 is backported into CDH 5.4.7.
>
>
>
> Thanks,
>
>
>
> Kevin
>
>
>
> *From:* Aditya [mailto:adityakish...@gmail.com]
> *Sent:* Sunday, March 20, 2016 10:45 PM
>
> *To:* Kumiko Yada 
> *Cc:* u...@drill.apache.org; dev@drill.apache.org;
> altekruseja...@gmail.com; Ki Kang ; Kevin Verhoeven <
> kevin.verhoe...@ds-iq.com>
> *Subject:* Re: Drill query does not return all results from HBase
>
>
>
> Finally managed to reproduce it with CDH distribution (So far I was
> testing with HBase 1.1 distributed with MapR, which does not have this bug).
>
> This is essentially an HBase bug, HBASE-13262[1], which has been fixed in
> 1.0.1, 1.1.0.
>
> Please update your HBase distribution.
>
>
> [1] https://issues.apache.org/jira/browse/HBASE-13262
>
>
>
> On Thu, Mar 17, 2016 at 3:19 PM, Kumiko Yada 
> wrote:
>
> Aditya,
>
>
>
> When we were exchanging the emails, you mentioned to me that you
> discovered another issue in case where the table is spit into multiple
> regions and the first region returned to the client did not have any rows.
> I think this issue is related to the issue that I’m seeing.  Have you
> opened the JIRA for this issue?  Have you investigated/fixed this issue?
>
>
>
> Thanks
>
> Kumiko
>
>
>
> *From:* Aditya [mailto:adityakish...@gmail.com]
> *Sent:* Thursday, March 17, 2016 3:02 PM
> *To:* Kumiko Yada 
> *Cc:* u...@drill.apache.org; dev@drill.apache.org;
> altekruseja...@gmail.com; Ki Kang ; Kevin Verhoeven <
> kevin.verhoe...@ds-iq.com>
>
>
> *Subject:* Re: Drill query does not return all results from HBase
>
>
>
> Hi Kumiko,
>
> I have tried to reproduce this locally with Apache 1.x release but have
> failed so far.
>
> From my mail exchange with Kevin on another thread, it appears that the
> HBase scanner stops returning rows after a while which seem odd.
>
> Probably it is unique to CDH distribution. I am planning to setup a single
> node CDH cluster to see if it I can reproduce it there.
>
>
>
> On Thu, Mar 17, 2016 at 2:56 PM, Kumiko Yada 
> wrote:
>
> Hello,
>
> I provided all information that was requested; however, I haven't heard
> back anything since February 24.
>
> Is anyone taking look at this?  Are there any workarounds?
>
> https://issues.apache.org/jira/browse/DRILL-4271
>
> Thanks
> Kumiko
>
> -Original Message-
> From: Aditya [mailto:adityakish...@gmail.com]
> Sent: Friday, February 19, 2016 12:48 PM
> To: user 
>
> Cc: altekruseja...@gmail.com; Ki Kang ; Kevin
> Verhoeven 
> Subject: Re: Drill query does not return all results from HBase
>
> Hi Kumiko,
>
> I apologies for not chiming in until now, considering that if there is a
> bug here it is most probably put in by me :)
>
> I've assigned the JIRA to myself and going to take a l look.
>
> Would it be possible for you to either attach to the JIRA or send me
> privately the Drill query profiles form both the correct and the incorrect
> executions?
>
> Regards,
> aditya...
>
> On Fri, Feb 19, 2016 at 12:34 PM, Kumiko Yada 
> wrote:
>
> > Hello,
> >
> > Does anyone have any update on this issue,
> > https://issues.apache.org/jira/browse/DRILL-4271?  Are there any plan
> > that this would be investigated/fixed?
> >
> > Thanks
> > Kumiko
> >
> > -Original Message-
> > From: Kumiko Yada [mailto:kumiko.y...@ds-iq.com]
> > Sent: Thursday, January 14, 2016 3:44 PM
> > To: u...@drill.apache.org; altekruseja...@gmail.com
> > Subject: RE: Drill query does not return all results from HBase
> >
> > The query time was very short on the one with the incorrect result.
> >
> > Thanks
> > Kumiko
> >
> > -Original Message-
> > From: Jason Altekruse [mailto:altekruseja...@gmail.com]
> > Sent: Thursday, January 14, 2016 1:25 PM
> > To: user 
> > Subject: Fwd: Drill query does not return all results from HBase
> >
> > Thanks for the update, I'm forwarding your message back to the list.
> >
> > Just to confirm, was the query time longer on the the one with the
> > incorrect result? In the incorrect case I think we are just misreading
> > the HBase metadata during our optimization to return row counts
> > without reading any data. This should be really fast, and noticeably
> > different than running a complete query, even with a small dataset as
> > we have to read in your table and run an aggregation over it.
> >
> > This would just be a final confirmation of where the issue is
> > occurring, I will hopefully have time soon to get this fixed but I'm
> > wrapping up some other things right now.
> >
> >
> > -- Forwarded message --
> > From: Kumiko Yada 
> > Date: Thu, Jan 14, 2016 at 12:53 PM
> > Subject: RE: D

[GitHub] drill pull request: DRILL-4514 : Add describe schema ...

[arina-ielchiieva]

GitHub user arina-ielchiieva opened a pull request:

https://github.com/apache/drill/pull/436

DRILL-4514 : Add describe schema  command

Syntax:
describe database 
describe schema 

Current implementation covers only dfs schema.
For all other "" will be returned.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/arina-ielchiieva/drill DRILL-4514

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/436.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #436


commit 86b7115a49f40c206a30b96288c13d9ff7ed53ac
Author: Arina Ielchiieva 
Date:   2016-03-16T16:33:41Z

DRILL-4514 : Add describe schema  command




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

RE: Drill query does not return all results from HBase

[Kevin Verhoeven]

Aditya,

Thank you for your help. What version of CDH are you running? I contacted 
Cloudera and they stated that bug HBASE-13262 is backported into CDH 5.4.7.

Thanks,

Kevin

From: Aditya [mailto:adityakish...@gmail.com]
Sent: Sunday, March 20, 2016 10:45 PM
To: Kumiko Yada 
Cc: u...@drill.apache.org; dev@drill.apache.org; altekruseja...@gmail.com; Ki 
Kang ; Kevin Verhoeven 
Subject: Re: Drill query does not return all results from HBase

Finally managed to reproduce it with CDH distribution (So far I was testing 
with HBase 1.1 distributed with MapR, which does not have this bug).
This is essentially an HBase bug, HBASE-13262[1], which has been fixed in 
1.0.1, 1.1.0.
Please update your HBase distribution.

[1] https://issues.apache.org/jira/browse/HBASE-13262

On Thu, Mar 17, 2016 at 3:19 PM, Kumiko Yada 
mailto:kumiko.y...@ds-iq.com>> wrote:
Aditya,

When we were exchanging the emails, you mentioned to me that you discovered 
another issue in case where the table is spit into multiple regions and the 
first region returned to the client did not have any rows.  I think this issue 
is related to the issue that I’m seeing.  Have you opened the JIRA for this 
issue?  Have you investigated/fixed this issue?

Thanks
Kumiko

From: Aditya [mailto:adityakish...@gmail.com]
Sent: Thursday, March 17, 2016 3:02 PM
To: Kumiko Yada mailto:kumiko.y...@ds-iq.com>>
Cc: u...@drill.apache.org; 
dev@drill.apache.org; 
altekruseja...@gmail.com; Ki Kang 
mailto:ki.k...@ds-iq.com>>; Kevin Verhoeven 
mailto:kevin.verhoe...@ds-iq.com>>

Subject: Re: Drill query does not return all results from HBase

Hi Kumiko,

I have tried to reproduce this locally with Apache 1.x release but have failed 
so far.
From my mail exchange with Kevin on another thread, it appears that the HBase 
scanner stops returning rows after a while which seem odd.
Probably it is unique to CDH distribution. I am planning to setup a single node 
CDH cluster to see if it I can reproduce it there.

On Thu, Mar 17, 2016 at 2:56 PM, Kumiko Yada 
mailto:kumiko.y...@ds-iq.com>> wrote:
Hello,

I provided all information that was requested; however, I haven't heard back 
anything since February 24.

Is anyone taking look at this?  Are there any workarounds?

https://issues.apache.org/jira/browse/DRILL-4271

Thanks
Kumiko

-Original Message-
From: Aditya [mailto:adityakish...@gmail.com]
Sent: Friday, February 19, 2016 12:48 PM
To: user mailto:u...@drill.apache.org>>
Cc: altekruseja...@gmail.com; Ki Kang 
mailto:ki.k...@ds-iq.com>>; Kevin Verhoeven 
mailto:kevin.verhoe...@ds-iq.com>>
Subject: Re: Drill query does not return all results from HBase

Hi Kumiko,

I apologies for not chiming in until now, considering that if there is a bug 
here it is most probably put in by me :)

I've assigned the JIRA to myself and going to take a l look.

Would it be possible for you to either attach to the JIRA or send me privately 
the Drill query profiles form both the correct and the incorrect executions?

Regards,
aditya...

On Fri, Feb 19, 2016 at 12:34 PM, Kumiko Yada 
mailto:kumiko.y...@ds-iq.com>> wrote:

> Hello,
>
> Does anyone have any update on this issue,
> https://issues.apache.org/jira/browse/DRILL-4271?  Are there any plan
> that this would be investigated/fixed?
>
> Thanks
> Kumiko
>
> -Original Message-
> From: Kumiko Yada [mailto:kumiko.y...@ds-iq.com]
> Sent: Thursday, January 14, 2016 3:44 PM
> To: u...@drill.apache.org; 
> altekruseja...@gmail.com
> Subject: RE: Drill query does not return all results from HBase
>
> The query time was very short on the one with the incorrect result.
>
> Thanks
> Kumiko
>
> -Original Message-
> From: Jason Altekruse 
> [mailto:altekruseja...@gmail.com]
> Sent: Thursday, January 14, 2016 1:25 PM
> To: user mailto:u...@drill.apache.org>>
> Subject: Fwd: Drill query does not return all results from HBase
>
> Thanks for the update, I'm forwarding your message back to the list.
>
> Just to confirm, was the query time longer on the the one with the
> incorrect result? In the incorrect case I think we are just misreading
> the HBase metadata during our optimization to return row counts
> without reading any data. This should be really fast, and noticeably
> different than running a complete query, even with a small dataset as
> we have to read in your table and run an aggregation over it.
>
> This would just be a final confirmation of where the issue is
> occurring, I will hopefully have time soon to get this fixed but I'm
> wrapping up some other things right now.
>
>
> -- Forwarded message --
> From: Kumiko Yada mailto:kumiko.y...@ds-iq.com>>
> Date: Thu, Jan 14, 2016 at 12:53 PM
> Subje

[jira] [Resolved] (DRILL-4271) Drill query does not return all results from HBase when the Hbase contains more than 6 columns

[Aditya Kishore (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aditya Kishore resolved DRILL-4271.
---
Resolution: Duplicate

Finally managed to reproduce it with CDH distribution (So far I was testing 
with HBase 1.1 distributed with MapR, which does not have this bug).

This is essentially an HBase bug, HBASE-13262, which has been fixed in HBase 
release 1.0.1, 1.1.0.

> Drill query does not return all results from HBase when the Hbase contains 
> more than 6 columns
> --
>
> Key: DRILL-4271
> URL: https://issues.apache.org/jira/browse/DRILL-4271
> Project: Apache Drill
>  Issue Type: Bug
> Environment: Drill 1.4, HBase 1.0.0, CDH 5.4
>Reporter: Kumiko Yada
>Assignee: Aditya Kishore
>Priority: Critical
> Attachments: Correct.drill, Incorrect.drill, Table2.png, table1.png, 
> table3.png, table4.png
>
>
> Hbase `test1` contains 1 column family & 5 columns, 1000 rows
> Hbase `test2` contains 1 column family & 6 columns, 1000 rows
> Query: select count ( * ) from `hbase`.`test1`
> Result:  1000 rows
> Query: select count ( * ) from `hbase`.`test2`
> Result:  6724 rows
> Note:
> Correct row count returned with the hbase table with 1 column family 5 or 
> less columns



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] drill pull request: Fixed by adding new where qlause TABLE_TYPE = ...

[dr-wolf]

GitHub user dr-wolf opened a pull request:

https://github.com/apache/drill/pull/435

Fixed by adding new where qlause TABLE_TYPE = TABLE



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dr-wolf/drill DRILL-4391

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/435.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #435


commit be7668cfc994c98f48bfa15af1fc8d9fd0f8b06e
Author: dr-wolf 
Date:   2016-03-21T13:27:26Z

Fixed by adding new where qlause TABLE_TYPE = TABLE




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (DRILL-4522) Apache Drill Schema Support for SQL Server

[Sanjiv Kumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sanjiv Kumar resolved DRILL-4522.
-
   Resolution: Fixed
Fix Version/s: 1.5.0

select * from ...;
Ex:

seleect * from SqlServer.Test.core.Category;
This query will work for all type of user created schemas. But for dbo(default) 
schema Databases name is not required. If you write database name while query 
through dbo schema, it will through error.

> Apache Drill Schema Support for SQL Server
> --
>
> Key: DRILL-4522
> URL: https://issues.apache.org/jira/browse/DRILL-4522
> Project: Apache Drill
>  Issue Type: Bug
>  Components:  Server
>Affects Versions: 1.5.0
> Environment: Sql Server, Window 8.1 
>Reporter: Sanjiv Kumar
> Fix For: 1.5.0
>
>
> I want to know whether Apache Drill supports only dbo schema?? or will it 
> support all type of schema?
> I am running my system in window 8 and with latest version of Drill(1.5) with 
> embedded mode.
> I am trying to search with same storage plugin.
> My Storage Plugin(for SQLServer):
> {
>   "type" : "jdbc",
>   "driver" : "com.microsoft.sqlserver.jdbc.SQLServerDriver",
>   "url" : "jdbc:sqlserver://;databaseName=",
>   "username" : "",
>   "password" : "<>",
>   "enabled" : true
> }
> This Plugin has dbo & core schema (both have same type, no special 
> permission).Its work for dbo schema where core schema is not working.
> DBO Query:
> select * from SqlServer.dbo.Attribute; //Its working.
> Core Query:
> select * from SqlServer.core.Users //Its not working
> My question is whether Drill Supports only dbo schemna or all type of schema?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)