[jira] [Created] (DRILL-5334) Questionable code in the TimeVector class

[Paul Rogers (JIRA)]

Paul Rogers created DRILL-5334:
--

 Summary: Questionable code in the TimeVector class
 Key: DRILL-5334
 URL: https://issues.apache.org/jira/browse/DRILL-5334
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.9.0
Reporter: Paul Rogers


The {{TimeVector}} class, which holds Time data, should hold a simple local 
time with no associated date or time zone. (A local time cannot be converted to 
UTC without a date since the conversion depends on when daylight savings is in 
effect.)

But, the implementation of {{TimeVector}} uses the following very questionable 
code:

{code}
@Override
public DateTime getObject(int index) {
org.joda.time.DateTime time = new org.joda.time.DateTime(get(index), 
org.joda.time.DateTimeZone.UTC);
time = 
time.withZoneRetainFields(org.joda.time.DateTimeZone.getDefault());
return time;
}
{code}

That is, we convert a date-less, local time into a Joda UTC DateTime object, 
then reset the time zone to local time. This is abusing the Joda library and is 
the very kind of fun & games that Joda was designed to prevent.

The conversion of a time into Joda should use the {{LocalTime}} class.

In fact, according to 
[Oracle|http://www.oracle.com/technetwork/articles/java/jf14-date-time-2125367.html],
 the following is the mapping from ANSI SQL date/time types to Java 8 (and thus 
Joda) classes:

||ANSI SQL||Java SE 8
|DATE|LocalDate
|TIME|LocalTime
|TIMESTAMP|LocalDateTime
|TIME WITH TIMEZONE|OffsetTime
|TIMESTAMP WITH TIMEZONE|OffsetDateTime




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (DRILL-5333) Documentation error in TIME data type description

[Paul Rogers (JIRA)]

Paul Rogers created DRILL-5333:
--

 Summary: Documentation error in TIME data type description
 Key: DRILL-5333
 URL: https://issues.apache.org/jira/browse/DRILL-5333
 Project: Apache Drill
  Issue Type: Bug
  Components: Documentation
Affects Versions: 1.9.0
Reporter: Paul Rogers
Priority: Minor


Consider the following description of the TIME data type from the 
[documentation|http://drill.apache.org/docs/supported-data-types/]:

{quote}
TIME

24-hour based time before or after January 1, 2001 in hours, minutes, seconds 
format: HH:mm:ss

22:55:55.23
{quote}

First, TIME has no associated date, so there can be no limitation on the days 
that can be represented. (If I tell you the bank closes at 5 PM, that statement 
is not just true after Jan. 1, 2001 -- it is true for as long as the bank 
exists.)

Second, the example implies that Drill stores milliseconds, which is consistent 
with the implementation of the {{TimeVector}} data type. But, the format 
suggests that granularity is seconds.

Finally, Time, as stored internally, has no format: it is a number; the number 
of milliseconds since the epoch. The format only comes into play when 
converting two or from text.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (DRILL-5332) DateVector support uses questionable conversions to a time

[Paul Rogers (JIRA)]

Paul Rogers created DRILL-5332:
--

 Summary: DateVector support uses questionable conversions to a time
 Key: DRILL-5332
 URL: https://issues.apache.org/jira/browse/DRILL-5332
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.9.0
Reporter: Paul Rogers


The following code in {{DateVector}} is worrisome:

{code}
@Override
public DateTime getObject(int index) {
org.joda.time.DateTime date = new org.joda.time.DateTime(get(index), 
org.joda.time.DateTimeZone.UTC);
date = 
date.withZoneRetainFields(org.joda.time.DateTimeZone.getDefault());
return date;
}
{code}

This code takes a date/time value stored in a value vector, converts it to UTC, 
then strips the time zone and replaces it with local time. The result it a 
"timestamp" in Java format (seconds since the epoch), but not really, it really 
the time since the epoch, as if the epoch had started in the local time zone 
rather than UTC.

This is the kind of fun & games that people used to do in Java with the 
{{Date}}  type before the advent of Joda time (and the migration of Joda into 
Java 8.)

It is, in short, very bad practice and nearly impossible to get right.

Further, converting a pure date (since this is a {{DateVector}}) into a 
date/time is fraught with peril. A date has no corresponding time. 1 AM on 
Friday in one time zone might be 11 PM on Thursday in another. Converting from 
dates to times is very difficult.

If the {{DateVector}} corresponds to a date, then it should be simple date with 
no implied time zone and no implied relationship to time. If there is to be a 
mapping of time, it must be to a {{LocalTime}} (in Joda and Java 8) that has no 
implied time zone.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] drill issue #775: DRILL-5326: Unit tests failures related to the SERVER_META...

[jinfengni]

Github user jinfengni commented on the issue:

https://github.com/apache/drill/pull/775
  
@laurentgo , @vdiravka , thanks. I'll run regression & merge.

+1



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill issue #775: DRILL-5326: Unit tests failures related to the SERVER_META...

[laurentgo]

Github user laurentgo commented on the issue:

https://github.com/apache/drill/pull/775
  
LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (DRILL-5331) NPE in FunctionImplementationRegistry.findDrillFunction() if dynamic UDFs disabled

[Paul Rogers (JIRA)]

Paul Rogers created DRILL-5331:
--

 Summary: NPE in FunctionImplementationRegistry.findDrillFunction() 
if dynamic UDFs disabled
 Key: DRILL-5331
 URL: https://issues.apache.org/jira/browse/DRILL-5331
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.10.0
Reporter: Paul Rogers
Assignee: Paul Rogers
 Fix For: 1.11.0


Drill provides the Dynamic UDF (DUDF) functionality. DUFDs can be disabled 
using the following option in {{ExecConstants}}:

{code}
  String USE_DYNAMIC_UDFS_KEY = "exec.udf.use_dynamic";
  BooleanValidator USE_DYNAMIC_UDFS = new 
BooleanValidator(USE_DYNAMIC_UDFS_KEY, true);
{code}

In a unit test, we created a setup in which we wish to use only the local 
function registry, no DUDF support is needed. Run the code. The following code 
is invoked when asking for a non-existent function:

{code}
  public DrillFuncHolder findDrillFunction(FunctionResolver functionResolver, 
FunctionCall functionCall) {
...
if (holder == null) {
  syncWithRemoteRegistry(version.get());
  List updatedFunctions = 
localFunctionRegistry.getMethods(newFunctionName, version);
  holder = functionResolver.getBestMatch(updatedFunctions, functionCall);
}
{code}

The result is an NPE:

{code}
ERROR o.a.d.e.e.f.r.RemoteFunctionRegistry - Problem during trying to access 
remote function registry [registry]
java.lang.NullPointerException: null
at 
org.apache.drill.exec.expr.fn.registry.RemoteFunctionRegistry.getRegistryVersion(RemoteFunctionRegistry.java:119)
 ~[classes/:na]
{code}

The fix is simply to add a DUDF-enabled check:

{code}
if (holder == null) {
  boolean useDynamicUdfs = optionManager != null && 
optionManager.getOption(ExecConstants.USE_DYNAMIC_UDFS);
  if (useDynamicUdfs) {
syncWithRemoteRegistry(version.get());
...
{code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] drill issue #775: DRILL-5326: Unit tests failures related to the SERVER_META...

[vdiravka]

Github user vdiravka commented on the issue:

https://github.com/apache/drill/pull/775
  
@laurentgo I changed the names for the server meta method. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #775: DRILL-5326: Unit tests failures related to the SERV...

[jinfengni]

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/775#discussion_r104816830
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/work/metadata/ServerMetaProvider.java
 ---
@@ -76,7 +76,7 @@
   .setReadOnly(false)
   .setGroupBySupport(GroupBySupport.GB_UNRELATED)
   .setLikeEscapeClauseSupported(true)
-  .setNullCollation(NullCollation.NC_AT_END)
+  .setNullCollation(NullCollation.NC_HIGH)
--- End diff --

If only specify NC here (no sort order), then NC_HIGH looks more 
reasonable. The change looks fine to me.  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (DRILL-5330) NPE in FunctionImplementationRegistry.functionReplacement()

[Paul Rogers (JIRA)]

Paul Rogers created DRILL-5330:
--

 Summary: NPE in 
FunctionImplementationRegistry.functionReplacement()
 Key: DRILL-5330
 URL: https://issues.apache.org/jira/browse/DRILL-5330
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.10.0
Reporter: Paul Rogers
Assignee: Paul Rogers
 Fix For: 1.11.0


The code in {{FunctionImplementationRegistry.functionReplacement()}} will 
produce an NPE if ever it is called:

{code}
  if (optionManager != null
  && optionManager.getOption(
   ExecConstants.CAST_TO_NULLABLE_NUMERIC).bool_val
  ...
{code}

If an option manager is provided, then get the specified option. The option 
manager will contain a value for that option only if the user has explicitly 
set that option. Suppose the user had not set the option. Then the return from 
{{getOption()}} will be null.

The next thing we do is *assume* that the option exists and is a boolean by 
dereferencing the option. This will trigger an NPE. This NPE was seen in 
detail-level unit tests.

The proper way to handle such options is to use an option validator. Strangely, 
one actually exists in {{ExecConstants}}:

{code}
  String CAST_TO_NULLABLE_NUMERIC = 
"drill.exec.functions.cast_empty_string_to_null";
  OptionValidator CAST_TO_NULLABLE_NUMERIC_OPTION = new 
BooleanValidator(CAST_TO_NULLABLE_NUMERIC, false);
{code}

Then do:

{code}
optionManager.getOption(
 ExecConstants.CAST_TO_NULLABLE_NUMERIC_OPTION)
{code}




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (DRILL-3111) Drill UI should support fast schema return and streaming results

[Abhishek Girish (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-3111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Girish resolved DRILL-3111.

   Resolution: Fixed
Fix Version/s: (was: Future)

> Drill UI should support fast schema return and streaming results
> 
>
> Key: DRILL-3111
> URL: https://issues.apache.org/jira/browse/DRILL-3111
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Client - HTTP
>Affects Versions: 1.0.0
>Reporter: Abhishek Girish
>
> On sqlline, a query which returns several hundred rows, supports fast schema 
> return and streams results as they are fetched. 
> Drill UI doesn't support either of these. It waits until all results are 
> fetched and displays them at once. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (DRILL-5329) External sort does not support TinyInt type

[Paul Rogers (JIRA)]

Paul Rogers created DRILL-5329:
--

 Summary: External sort does not support TinyInt type
 Key: DRILL-5329
 URL: https://issues.apache.org/jira/browse/DRILL-5329
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.10.0
Reporter: Paul Rogers


A unit test was created to exercise the "Sorter" mechanism within the External 
Sort, which is used to sort each incoming batch. The sorter was tested with 
each Drill data type.

The following types fail:

* TinyInt




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] drill pull request #775: DRILL-5326: Unit tests failures related to the SERV...

[laurentgo]

Github user laurentgo commented on a diff in the pull request:

https://github.com/apache/drill/pull/775#discussion_r104813282
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/work/metadata/ServerMetaProvider.java
 ---
@@ -76,7 +76,7 @@
   .setReadOnly(false)
   .setGroupBySupport(GroupBySupport.GB_UNRELATED)
   .setLikeEscapeClauseSupported(true)
-  .setNullCollation(NullCollation.NC_AT_END)
+  .setNullCollation(NullCollation.NC_HIGH)
--- End diff --

I'm not sure if the last sentence is correct. If user specified DESC, and 
null collation is unspecified, then null should sort first, no (if always 
HIGH). NC_HIGH seems then the correct value (unless the default can be changed 
in the planner configuration).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #775: DRILL-5326: Unit tests failures related to the SERV...

[vdiravka]

Github user vdiravka commented on a diff in the pull request:

https://github.com/apache/drill/pull/775#discussion_r104812618
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/work/metadata/ServerMetaProvider.java
 ---
@@ -76,7 +76,7 @@
   .setReadOnly(false)
   .setGroupBySupport(GroupBySupport.GB_UNRELATED)
   .setLikeEscapeClauseSupported(true)
-  .setNullCollation(NullCollation.NC_AT_END)
+  .setNullCollation(NullCollation.NC_HIGH)
--- End diff --

@jinfengni Agree with Paul. The difference between NC_HIGH and NC_AT_END 
for the DESC case. I checked that Drill uses NC_HIGH for sorting. Please, see 
it at 
[jira](https://issues.apache.org/jira/browse/DRILL-5326?focusedCommentId=15900411=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15900411),
 since github doesn't show formatted code correctly in the comments.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (DRILL-5328) Trim down physical plan size - replace StoragePluginConfig with storage name

[Chunhui Shi (JIRA)]

Chunhui Shi created DRILL-5328:
--

 Summary: Trim down physical plan size - replace 
StoragePluginConfig with storage name
 Key: DRILL-5328
 URL: https://issues.apache.org/jira/browse/DRILL-5328
 Project: Apache Drill
  Issue Type: Improvement
Reporter: Chunhui Shi


For a physical plan, we now pass StoragePluginConfig as part of plan, then the 
destination use the config to fetch the storage plugin in 
StoragePluginRegistry. However, we can also fetch a storage plugin with the 
name which is identical to all Drillbits. 

In the example of simple physical plan of 150 lines shown below,  the storage 
plugin config took 60 lines. In a typical large system, FileSystem's 
StoragePluginConfig could be >500 lines. So this improvement should save the 
cost of passing a larger physical plan among nodes.

0: jdbc:drill:zk=10.10.88.126:5181> explain plan for select * from 
dfs.tmp.employee1 where last_name='Blumberg';
+--+--+
| text | json |
+--+--+
| 00-00Screen
00-01  Project(*=[$0])
00-02Project(T1¦¦*=[$0])
00-03  SelectionVectorRemover
00-04Filter(condition=[=($1, 'Blumberg')])
00-05  Project(T1¦¦*=[$0], last_name=[$1])
00-06Scan(groupscan=[ParquetGroupScan 
[entries=[ReadEntryWithPath [path=/tmp/employee1/0_0_0.parquet]], 
selectionRoot=/tmp/employee1, numFiles=1, usedMetadataFile=true, 
cacheFileRoot=/tmp/employee1, columns=[`*`]]])
 | {
  "head" : {
"version" : 1,
"generator" : {
  "type" : "ExplainHandler",
  "info" : ""
},
"type" : "APACHE_DRILL_PHYSICAL",
"options" : [ ],
"queue" : 0,
"resultMode" : "EXEC"
  },
  "graph" : [ {
"pop" : "parquet-scan",
"@id" : 6,
"userName" : "root",
"entries" : [ {
  "path" : "/tmp/employee1/0_0_0.parquet"
} ],
"storage" : {
  "type" : "file",
  "enabled" : true,
  "connection" : "maprfs:///",
  "config" : null,
  "workspaces" : {
"root" : {
  "location" : "/",
  "writable" : false,
  "defaultInputFormat" : null
},
"tmp" : {
  "location" : "/tmp",
  "writable" : true,
  "defaultInputFormat" : null
},
"shi" : {
  "location" : "/user/shi",
  "writable" : true,
  "defaultInputFormat" : null
},
"dir700" : {
  "location" : "/user/shi/dir700",
  "writable" : true,
  "defaultInputFormat" : null
},
"dir775" : {
  "location" : "/user/shi/dir775",
  "writable" : true,
  "defaultInputFormat" : null
},
"xyz" : {
  "location" : "/user/xyz",
  "writable" : true,
  "defaultInputFormat" : null
}
  },
  "formats" : {
"psv" : {
  "type" : "text",
  "extensions" : [ "tbl" ],
  "delimiter" : "|"
},
"csv" : {
  "type" : "text",
  "extensions" : [ "csv" ],
  "delimiter" : ","
},
"tsv" : {
  "type" : "text",
  "extensions" : [ "tsv" ],
  "delimiter" : "\t"
},
"parquet" : {
  "type" : "parquet"
},
"json" : {
  "type" : "json",
  "extensions" : [ "json" ]
},
"maprdb" : {
  "type" : "maprdb"
}
  }
},
"format" : {
  "type" : "parquet"
},
"columns" : [ "`*`" ],
"selectionRoot" : "/tmp/employee1",
"filter" : "true",
"fileSet" : [ "/tmp/employee1/0_0_0.parquet" ],
"files" : [ "/tmp/employee1/0_0_0.parquet" ],
"cost" : 1155.0
  }, {
"pop" : "project",
"@id" : 5,
"exprs" : [ {
  "ref" : "`T1¦¦*`",
  "expr" : "`*`"
}, {
  "ref" : "`last_name`",
  "expr" : "`last_name`"
} ],
"child" : 6,
"initialAllocation" : 100,
"maxAllocation" : 100,
"cost" : 1155.0
  }, {
"pop" : "filter",
"@id" : 4,
"child" : 5,
"expr" : "equal(`last_name`, 'Blumberg') ",
"initialAllocation" : 100,
"maxAllocation" : 100,
"cost" : 173.25
  }, {
"pop" : "selection-vector-remover",
"@id" : 3,
"child" : 4,
"initialAllocation" : 100,
"maxAllocation" : 100,
"cost" : 173.25
  }, {
"pop" : "project",
"@id" : 2,
"exprs" : [ {
  "ref" : "`T1¦¦*`",
  "expr" : "`T1¦¦*`"
} ],
"child" : 3,
"initialAllocation" : 100,
"maxAllocation" : 100,
"cost" : 173.25
  }, {
"pop" : "project",
"@id" : 1,
"exprs" : [ {
  "ref" : "`*`",
  "expr" : "`T1¦¦*`"
} ],
"child" : 2,
"initialAllocation" : 100,
"maxAllocation" : 100,
"cost" : 173.25
  }, {
"pop" : "screen",
"@id" : 0,
"child" : 1,
"initialAllocation" : 100,
"maxAllocation" : 100,

[GitHub] drill pull request #775: DRILL-5326: Unit tests failures related to the SERV...

[paul-rogers]

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/775#discussion_r104809964
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/work/metadata/ServerMetaProvider.java
 ---
@@ -76,7 +76,7 @@
   .setReadOnly(false)
   .setGroupBySupport(GroupBySupport.GB_UNRELATED)
   .setLikeEscapeClauseSupported(true)
-  .setNullCollation(NullCollation.NC_AT_END)
+  .setNullCollation(NullCollation.NC_HIGH)
--- End diff --

When sorting in Drill, the detailed sort spec is set in the 
{{ExternalSort}} operator definition. This thing is a bit complex. One can 
control sort order (ASC, DESC) and nulls position (LOW, HIGH, UNSPECIFIED.)

Data sorts according to ASC, DESC.
Nulls sort as follows:

HIGH: last if ASC, first if DESC
LOW: first if ASC, last if DESC
UNSPECIFIED: always high

If the planner has no way of setting the nulls ordering from a SQL query, 
then the value is UNSPECIFIED, which means nulls always sort last as Jinfeng 
explained.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #775: DRILL-5326: Unit tests failures related to the SERV...

[jinfengni]

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/775#discussion_r104807540
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/work/metadata/ServerMetaProvider.java
 ---
@@ -76,7 +76,7 @@
   .setReadOnly(false)
   .setGroupBySupport(GroupBySupport.GB_UNRELATED)
   .setLikeEscapeClauseSupported(true)
-  .setNullCollation(NullCollation.NC_AT_END)
+  .setNullCollation(NullCollation.NC_HIGH)
--- End diff --

My understanding is NULL collation should be specified together with 
ASC/DESC.  ASC/NULL LAST as the default option essentially implies NC_HIGH. 
However, I'm not sure if we just specify NC_HIGH alone, or ASC/NC_AT_END 
combined.   


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #775: DRILL-5326: Unit tests failures related to the SERV...

[zfong]

Github user zfong commented on a diff in the pull request:

https://github.com/apache/drill/pull/775#discussion_r104805550
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/work/metadata/ServerMetaProvider.java
 ---
@@ -76,7 +76,7 @@
   .setReadOnly(false)
   .setGroupBySupport(GroupBySupport.GB_UNRELATED)
   .setLikeEscapeClauseSupported(true)
-  .setNullCollation(NullCollation.NC_AT_END)
+  .setNullCollation(NullCollation.NC_HIGH)
--- End diff --

@jinfengni  - See @vdiravka's comments in the Jira.  The Drill 
documentation at https://drill.apache.org/docs/order-by-clause/#usage-notes 
says NULLs sort highest.  If the doc is wrong, then we should fix the doc. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (DRILL-5327) Hash aggregate can return empty batch which can cause schema change exception

[Chun Chang (JIRA)]

Chun Chang created DRILL-5327:
-

 Summary: Hash aggregate can return empty batch which can cause 
schema change exception
 Key: DRILL-5327
 URL: https://issues.apache.org/jira/browse/DRILL-5327
 Project: Apache Drill
  Issue Type: Bug
  Components: Functions - Drill
Affects Versions: 1.10.0
Reporter: Chun Chang
Priority: Blocker


Hash aggregate can return empty batches which cause drill to throw schema 
change exception (not handling this type of schema change). This is not a new 
bug. But a recent hash function change (a theoretically correct change) may 
have increased the chance of hitting this issue. I don't have scientific data 
to support my claim (in fact I don't believe it's the case), but a regular 
regression run used to pass fails now due to this bug. My concern is that 
existing drill users out there may have queries that used to work but fail now. 
It will be difficult to explain why the new release is better for them. I put 
this bug as blocker so we can discuss it before releasing 1.10.

{noformat}
/root/drillAutomation/framework-master/framework/resources/Advanced/tpcds/tpcds_sf1/original/text/query66.sql
Query: 
-- start query 66 in stream 0 using template query66.tpl 
SELECT w_warehouse_name, 
   w_warehouse_sq_ft, 
   w_city, 
   w_county, 
   w_state, 
   w_country, 
   ship_carriers, 
   year1,
   Sum(jan_sales) AS jan_sales, 
   Sum(feb_sales) AS feb_sales, 
   Sum(mar_sales) AS mar_sales, 
   Sum(apr_sales) AS apr_sales, 
   Sum(may_sales) AS may_sales, 
   Sum(jun_sales) AS jun_sales, 
   Sum(jul_sales) AS jul_sales, 
   Sum(aug_sales) AS aug_sales, 
   Sum(sep_sales) AS sep_sales, 
   Sum(oct_sales) AS oct_sales, 
   Sum(nov_sales) AS nov_sales, 
   Sum(dec_sales) AS dec_sales, 
   Sum(jan_sales / w_warehouse_sq_ft) AS jan_sales_per_sq_foot, 
   Sum(feb_sales / w_warehouse_sq_ft) AS feb_sales_per_sq_foot, 
   Sum(mar_sales / w_warehouse_sq_ft) AS mar_sales_per_sq_foot, 
   Sum(apr_sales / w_warehouse_sq_ft) AS apr_sales_per_sq_foot, 
   Sum(may_sales / w_warehouse_sq_ft) AS may_sales_per_sq_foot, 
   Sum(jun_sales / w_warehouse_sq_ft) AS jun_sales_per_sq_foot, 
   Sum(jul_sales / w_warehouse_sq_ft) AS jul_sales_per_sq_foot, 
   Sum(aug_sales / w_warehouse_sq_ft) AS aug_sales_per_sq_foot, 
   Sum(sep_sales / w_warehouse_sq_ft) AS sep_sales_per_sq_foot, 
   Sum(oct_sales / w_warehouse_sq_ft) AS oct_sales_per_sq_foot, 
   Sum(nov_sales / w_warehouse_sq_ft) AS nov_sales_per_sq_foot, 
   Sum(dec_sales / w_warehouse_sq_ft) AS dec_sales_per_sq_foot, 
   Sum(jan_net)   AS jan_net, 
   Sum(feb_net)   AS feb_net, 
   Sum(mar_net)   AS mar_net, 
   Sum(apr_net)   AS apr_net, 
   Sum(may_net)   AS may_net, 
   Sum(jun_net)   AS jun_net, 
   Sum(jul_net)   AS jul_net, 
   Sum(aug_net)   AS aug_net, 
   Sum(sep_net)   AS sep_net, 
   Sum(oct_net)   AS oct_net, 
   Sum(nov_net)   AS nov_net, 
   Sum(dec_net)   AS dec_net 
FROM   (SELECT w_warehouse_name, 
   w_warehouse_sq_ft, 
   w_city, 
   w_county, 
   w_state, 
   w_country, 
   'ZOUROS' 
   || ',' 
   || 'ZHOU' AS ship_carriers, 
   d_yearAS year1, 
   Sum(CASE 
 WHEN d_moy = 1 THEN ws_ext_sales_price * ws_quantity 
 ELSE 0 
   END)  AS jan_sales, 
   Sum(CASE 
 WHEN d_moy = 2 THEN ws_ext_sales_price * ws_quantity 
 ELSE 0 
   END)  AS feb_sales, 
   Sum(CASE 
 WHEN d_moy = 3 THEN ws_ext_sales_price * ws_quantity 
 ELSE 0 
   END)  AS mar_sales, 
   Sum(CASE 
 WHEN d_moy = 4 THEN ws_ext_sales_price * ws_quantity 
 ELSE 0 
   END)  AS apr_sales, 
  

[GitHub] drill pull request #775: DRILL-5326: Unit tests failures related to the SERV...

[jinfengni]

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/775#discussion_r104802660
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/work/metadata/ServerMetaProvider.java
 ---
@@ -76,7 +76,7 @@
   .setReadOnly(false)
   .setGroupBySupport(GroupBySupport.GB_UNRELATED)
   .setLikeEscapeClauseSupported(true)
-  .setNullCollation(NullCollation.NC_AT_END)
+  .setNullCollation(NullCollation.NC_HIGH)
--- End diff --

I'm not completely sure why we should change from NC_AT_END to NC_HIGH, in 
stead of NC_AT_END.  I thought Drill is using ASC as default ordering, and 
NULLS LAST as default null collation for ASC. This is consistent to what Oracle 
[1] and Postgres [2] is doing : ASC /NULL LAST is the default option.  

1. http://docs.oracle.com/javadb/10.6.2.1/ref/rrefsqlj13658.html
2. https://www.postgresql.org/docs/9.4/static/queries-order.html



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Time for 1.10 release

[Jinfeng Ni]

Yes, DRILL-5326 is blocking issue for RC0, if build with unit test fails.


On Tue, Mar 7, 2017 at 12:55 PM, Zelaine Fong  wrote:
> It looks like DRILL-5326 has just been logged.  I assume we need this for RC0?
>
> Vitalii – it looks like you have a fix.  Can you post a pull request as soon 
> as you’re ready with the change.
>
> Thanks.
>
> -- Zelaine
>
> On 3/3/17, 2:13 PM, "Jinfeng Ni"  wrote:
>
> Pushed the patch for DRILL-5313, as well as two other patches which
> were reviewed before 3/1.
>
> I will start build RC0 after QA sanity test is completed.
>
>
> On Thu, Mar 2, 2017 at 11:00 PM, Jinfeng Ni  wrote:
> > Looks like the C++ client is not able to built successfully [1], after
> > the change of DRILL-5301 / DRILL-5167.
> >
> > This seems to be a blocking issue for 1.10.0, and I'll merge the patch
> > once it's verified/reviewed.
> >
> > 1. https://issues.apache.org/jira/browse/DRILL-5313
> >
> > On Thu, Mar 2, 2017 at 5:37 PM, Jinfeng Ni  wrote:
> >> I missed 5208, because it did not show up in Paul's list when he 
> replied to
> >> this thread.
> >>
> >> On Thu, Mar 2, 2017 at 2:58 PM Zelaine Fong  wrote:
> >>>
> >>> Jinfeng,
> >>>
> >>> I notice the following Jira has the ready-to-commit label but isn’t on
> >>> your list:
> >>>
> >>> DRILL-5208
> >>>
> >>> Was this one overlooked?
> >>>
> >>> -- Zelaine
> >>>
> >>> On 3/2/17, 1:04 PM, "Jinfeng Ni"  wrote:
> >>>
> >>> The following PRs have been merged to Apache master.
> >>>
> >>> DRILL-4994
> >>> DRILL-4730
> >>> DRILL-5301
> >>> DRILL-5167
> >>> DRILL-5221
> >>> DRILL-5258
> >>> DRILL-5034
> >>> DRILL-4963
> >>> DRILL-5252
> >>> DRILL-5266
> >>> DRILL-5284
> >>> DRILL-5304
> >>> DRILL-5290
> >>> DRILL-5287
> >>>
> >>> QA folks will run tests. If no issue found, I'll build a RC0 
> candidate
> >>> for 1.10 and start the vote.
> >>>
> >>> Thanks,
> >>>
> >>> Jinfeng
> >>>
> >>>
> >>>
> >>> On Thu, Mar 2, 2017 at 8:30 AM, Jinfeng Ni  
> wrote:
> >>> > I'm building a merge branch, and hopefully push to master branch
> >>> today
> >>> > if things go smoothly.
> >>> >
> >>> >
> >>> > On Wed, Mar 1, 2017 at 7:13 PM, Padma Penumarthy
> >>>  wrote:
> >>> >> Hi Jinfeng,
> >>> >>
> >>> >> Please include DRILL-5287, DRILL-5290 and DRILL-5304.
> >>> >>
> >>> >> Thanks,
> >>> >> Padma
> >>> >>
> >>> >>
> >>> >>> On Feb 22, 2017, at 11:16 PM, Jinfeng Ni  
> wrote:
> >>> >>>
> >>> >>> Hi Drillers,
> >>> >>>
> >>> >>> It has been almost 3 months since we release Drill 1.9. We 
> have
> >>> >>> resolved plenty of fixes and improvements (closed around 88 
> JIRAs
> >>> >>> [1]). I propose that we start the 1.10 release process, and 
> set
> >>> >>> Wednesday 3/1 as the cutoff day for code checkin. After 3/1, 
> we
> >>> should
> >>> >>> start build a release candidate.
> >>> >>>
> >>> >>> Please reply in this email thread if you have something near
> >>> complete
> >>> >>> and you would like to include in 1.10 release.
> >>> >>>
> >>> >>> I volunteer as the release manager, unless someone else come
> >>> forward.
> >>> >>>
> >>> >>> Thanks,
> >>> >>>
> >>> >>> Jinfeng
> >>> >>>
> >>> >>> [1]
> >>> https://issues.apache.org/jira/browse/DRILL/fixforversion/12338769
> >>> >>
> >>>
> >>>
> >>
>
>

Re: Time for 1.10 release

[Vitalii Diravka]

I posted a PR  for the DRILL-5326
 since it can be blocker
for a new release.

Jinfeng, please take a look at this.

Kind regards
Vitalii

On Tue, Mar 7, 2017 at 8:55 PM, Zelaine Fong  wrote:

> It looks like DRILL-5326 has just been logged.  I assume we need this for
> RC0?
>
> Vitalii – it looks like you have a fix.  Can you post a pull request as
> soon as you’re ready with the change.
>
> Thanks.
>
> -- Zelaine
>
> On 3/3/17, 2:13 PM, "Jinfeng Ni"  wrote:
>
> Pushed the patch for DRILL-5313, as well as two other patches which
> were reviewed before 3/1.
>
> I will start build RC0 after QA sanity test is completed.
>
>
> On Thu, Mar 2, 2017 at 11:00 PM, Jinfeng Ni  wrote:
> > Looks like the C++ client is not able to built successfully [1],
> after
> > the change of DRILL-5301 / DRILL-5167.
> >
> > This seems to be a blocking issue for 1.10.0, and I'll merge the
> patch
> > once it's verified/reviewed.
> >
> > 1. https://issues.apache.org/jira/browse/DRILL-5313
> >
> > On Thu, Mar 2, 2017 at 5:37 PM, Jinfeng Ni  wrote:
> >> I missed 5208, because it did not show up in Paul's list when he
> replied to
> >> this thread.
> >>
> >> On Thu, Mar 2, 2017 at 2:58 PM Zelaine Fong  wrote:
> >>>
> >>> Jinfeng,
> >>>
> >>> I notice the following Jira has the ready-to-commit label but
> isn’t on
> >>> your list:
> >>>
> >>> DRILL-5208
> >>>
> >>> Was this one overlooked?
> >>>
> >>> -- Zelaine
> >>>
> >>> On 3/2/17, 1:04 PM, "Jinfeng Ni"  wrote:
> >>>
> >>> The following PRs have been merged to Apache master.
> >>>
> >>> DRILL-4994
> >>> DRILL-4730
> >>> DRILL-5301
> >>> DRILL-5167
> >>> DRILL-5221
> >>> DRILL-5258
> >>> DRILL-5034
> >>> DRILL-4963
> >>> DRILL-5252
> >>> DRILL-5266
> >>> DRILL-5284
> >>> DRILL-5304
> >>> DRILL-5290
> >>> DRILL-5287
> >>>
> >>> QA folks will run tests. If no issue found, I'll build a RC0
> candidate
> >>> for 1.10 and start the vote.
> >>>
> >>> Thanks,
> >>>
> >>> Jinfeng
> >>>
> >>>
> >>>
> >>> On Thu, Mar 2, 2017 at 8:30 AM, Jinfeng Ni 
> wrote:
> >>> > I'm building a merge branch, and hopefully push to master
> branch
> >>> today
> >>> > if things go smoothly.
> >>> >
> >>> >
> >>> > On Wed, Mar 1, 2017 at 7:13 PM, Padma Penumarthy
> >>>  wrote:
> >>> >> Hi Jinfeng,
> >>> >>
> >>> >> Please include DRILL-5287, DRILL-5290 and DRILL-5304.
> >>> >>
> >>> >> Thanks,
> >>> >> Padma
> >>> >>
> >>> >>
> >>> >>> On Feb 22, 2017, at 11:16 PM, Jinfeng Ni 
> wrote:
> >>> >>>
> >>> >>> Hi Drillers,
> >>> >>>
> >>> >>> It has been almost 3 months since we release Drill 1.9. We
> have
> >>> >>> resolved plenty of fixes and improvements (closed around
> 88 JIRAs
> >>> >>> [1]). I propose that we start the 1.10 release process,
> and set
> >>> >>> Wednesday 3/1 as the cutoff day for code checkin. After
> 3/1, we
> >>> should
> >>> >>> start build a release candidate.
> >>> >>>
> >>> >>> Please reply in this email thread if you have something
> near
> >>> complete
> >>> >>> and you would like to include in 1.10 release.
> >>> >>>
> >>> >>> I volunteer as the release manager, unless someone else
> come
> >>> forward.
> >>> >>>
> >>> >>> Thanks,
> >>> >>>
> >>> >>> Jinfeng
> >>> >>>
> >>> >>> [1]
> >>> https://issues.apache.org/jira/browse/DRILL/fixforversion/12338769
> >>> >>
> >>>
> >>>
> >>
>
>
>

[jira] [Resolved] (DRILL-4850) TPCDS Query 33 failed in the second and 3rd runs, but succeeded in the 1st run

[Dechang Gu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dechang Gu resolved DRILL-4850.
---
   Resolution: Cannot Reproduce
Fix Version/s: 1.10.0

Cannot repro in 1.10.0 (gitid: 3bfb497), hence close it.

> TPCDS Query 33 failed in the second and 3rd runs, but succeeded in the 1st run
> --
>
> Key: DRILL-4850
> URL: https://issues.apache.org/jira/browse/DRILL-4850
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Drill
>Affects Versions: 1.8.0
> Environment: REL 6.0 
>Reporter: Dechang Gu
>Assignee: Parth Chandra
> Fix For: 1.10.0
>
>
> I run tpcds query 33 on SF100 database, 3 times consecutively. The first one 
> succeeded, but the 2nd and 3rd runs hit the following error:
>  
> 2016-08-16 20:20:52,530 ucs-node6.perf.lab 
> [284c27f1-ee13-dfd0-6cbb-37f49810e93f:frag:3:9] ERROR 
> o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException: 
> Failure while reading vector.  Expected vector class of 
> org.apache.drill.exec.vector.NullableIntVector but was holding vector class 
> org.apache.drill.exec.vector.NullableBigIntVector, field= 
> i_manufact_id(BIGINT:OPTIONAL)[$bits$(UINT1:REQUIRED), 
> i_manufact_id(BIGINT:OPTIONAL)]
> Fragment 3:9
> [Error Id: 7fc06ab9-6c63-402b-a1b4-465526aa7dc7 on ucs-node6.perf.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
> IllegalStateException: Failure while reading vector.  Expected vector class 
> of org.apache.drill.exec.vector.NullableIntVector but was holding vector 
> class org.apache.drill.exec.vector.NullableBigIntVector, field= 
> i_manufact_id(BIGINT:OPTIONAL)[$bits$(UINT1:REQUIRED), 
> i_manufact_id(BIGINT:OPTIONAL)]
> Fragment 3:9
> [Error Id: 7fc06ab9-6c63-402b-a1b4-465526aa7dc7 on ucs-node6.perf.lab:31010]
> at 
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543)
>  ~[drill-common-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:293)
>  [drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
>  [drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:262)
>  [drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
>  [drill-common-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  [na:1.7.0_65]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  [na:1.7.0_65]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65]
> Caused by: java.lang.IllegalStateException: Failure while reading vector.  
> Expected vector class of org.apache.drill.exec.vector.NullableIntVector but 
> was holding vector class org.apache.drill.exec.vector.NullableBigIntVector, 
> field= i_manufact_id(BIGINT:OPTIONAL)[$bits$(UINT1:REQUIRED), 
> i_manufact_id(BIGINT:OPTIONAL)]
> at 
> org.apache.drill.exec.record.VectorContainer.getValueAccessorById(VectorContainer.java:290)
>  ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.record.RecordBatchLoader.getValueAccessorById(RecordBatchLoader.java:178)
>  ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.physical.impl.unorderedreceiver.UnorderedReceiverBatch.getValueAccessorById(UnorderedReceiverBatch.java:135)
>  ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.test.generated.PartitionerGen36655$OutgoingRecordBatch.doSetup(PartitionerTemplate.java:64)
>  ~[na:na]
> at 
> org.apache.drill.exec.test.generated.PartitionerGen36655$OutgoingRecordBatch.initializeBatch(PartitionerTemplate.java:358)
>  ~[na:na]
> at 
> org.apache.drill.exec.test.generated.PartitionerGen36655.flushOutgoingBatches(PartitionerTemplate.java:163)
>  ~[na:na]
> at 
> org.apache.drill.exec.physical.impl.partitionsender.PartitionerDecorator$FlushBatchesHandlingClass.execute(PartitionerDecorator.java:266)
>  ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.physical.impl.partitionsender.PartitionerDecorator.executeMethodLogic(PartitionerDecorator.java:138)
>  ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.physical.impl.partitionsender.PartitionerDecorator.flushOutgoingBatches(PartitionerDecorator.java:82)
>  

[jira] [Resolved] (DRILL-4851) TPCDS Query 64 just hang there at "starting" state

[Dechang Gu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dechang Gu resolved DRILL-4851.
---
Resolution: Duplicate

duplicate of DRILL-4347

> TPCDS Query 64 just hang there at "starting" state
> --
>
> Key: DRILL-4851
> URL: https://issues.apache.org/jira/browse/DRILL-4851
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Drill
>Affects Versions: 1.8.0
> Environment: REL 6.0
>Reporter: Dechang Gu
>Assignee: Jinfeng Ni
>
> TPC-DS Query 64 on SF100 (views on parquet files) hung there at "starting" 
> state. drillbit.log on the foreman node show the following info:
> 2016-08-17 14:26:00,785 ucs-node4.perf.lab 
> [284b2996-d25a-d9af-178c-143b32ea6969:foreman] INFO  
> o.a.drill.exec.work.foreman.Foreman - Query text for query id 
> 284b2996-d25a-d9af-178c-143b32ea6969: WITH cs_ui AS (SELECT cs_item_sk, 
> Sum(cs_ext_list_price) AS sale, Sum(cr_refunded_cash + cr_reversed_charge + 
> cr_store_credit) AS refund FROM   catalog_sales, catalog_returns WHERE  
> cs_item_sk = cr_item_sk AND cs_order_number = cr_order_number GROUP  BY 
> cs_item_sk HAVING Sum(cs_ext_list_price) > 2 * Sum( cr_refunded_cash + 
> cr_reversed_charge + cr_store_credit)), cross_sales AS (SELECT i_product_name 
> product_name, i_item_sk  item_sk, s_store_name   
> store_name, s_zip  store_zip, ad1.ca_street_number   
> b_street_number, ad1.ca_street_name b_streen_name, ad1.ca_city
> b_city, ad1.ca_zip b_zip, ad2.ca_street_number   c_street_number, 
> ad2.ca_street_name c_street_name, ad2.ca_cityc_city, 
> ad2.ca_zip c_zip, d1.d_year  AS syear, d2.d_year  
> AS fsyear, d3.d_year  s2year, Count(*)   cnt, 
> Sum(ss_wholesale_cost) s1, Sum(ss_list_price) s2, Sum(ss_coupon_amt) 
> s3 FROM   store_sales, store_returns, cs_ui, date_dim d1, date_dim d2, 
> date_dim d3, store, customer, customer_dd
> emographics cd1, customer_demographics cd2, promotion, household_demographics 
> hd1, household_demographics hd2, customer_address ad1, customer_address ad2, 
> income_band ib1, income_band ib2, item WHERE  ss_store_sk = s_stt
> ore_sk AND ss_sold_date_sk = d1.d_date_sk AND ss_customer_sk = c_customer_sk 
> AND ss_cdemo_sk = cd1.cd_demo_sk AND ss_hdemo_sk = hd1.hd_demo_sk AND 
> ss_addr_sk = ad1.ca_address_sk AND ss_item_sk = i_item_sk AND ss_item_skk
>  = sr_item_sk AND ss_ticket_number = sr_ticket_number AND ss_item_sk = 
> cs_ui.cs_item_sk AND c_current_cdemo_sk = cd2.cd_demo_sk AND 
> c_current_hdemo_sk = hd2.hd_demo_sk AND c_current_addr_sk = ad2.ca_address_sk 
> AND c_firr
> st_sales_date_sk = d2.d_date_sk AND c_first_shipto_date_sk = d3.d_date_sk AND 
> ss_promo_sk = p_promo_sk AND hd1.hd_income_band_sk = ib1.ib_income_band_sk 
> AND hd2.hd_income_band_sk = ib2.ib_income_band_sk AND cd1.cd_maritt
> al_status <> cd2.cd_marital_status AND i_color IN ( 'cyan', 'peach', 'blush', 
> 'frosted', 'powder', 'orange' ) AND i_current_price BETWEEN 58 AND 58 + 10 
> AND i_current_price BETWEEN 58 + 1 AND 58 + 15 GROUP  BY i_productt
> _name, i_item_sk, s_store_name, s_zip, ad1.ca_street_number, 
> ad1.ca_street_name, ad1.ca_city, ad1.ca_zip, ad2.ca_street_number, 
> ad2.ca_street_name, ad2.ca_city, ad2.ca_zip, d1.d_year, d2.d_year, d3.d_year) 
> SELECT cs1.prr
> oduct_name, cs1.store_name, cs1.store_zip, cs1.b_street_number, 
> cs1.b_streen_name, cs1.b_city, cs1.b_zip, cs1.c_street_number, 
> cs1.c_street_name, cs1.c_city, cs1.c_zip, cs1.syear, cs1.cnt, cs1.s1, cs1.s2, 
> cs1.s3, cs2.s11
> , cs2.s2, cs2.s3, cs2.syear, cs2.cnt FROM   cross_sales cs1, cross_sales cs2 
> WHERE  cs1.item_sk = cs2.item_sk AND cs1.syear = 2001 AND cs2.syear = 2001 + 
> 1 AND cs2.cnt <= cs1.cnt AND cs1.store_name = cs2.store_name AND
> cs1.store_zip = cs2.store_zip ORDER  BY cs1.product_name, cs1.store_name, 
> cs2.cnt
> 2016-08-17 14:26:04,045 ucs-node4.perf.lab 
> [284b2996-d25a-d9af-178c-143b32ea6969:foreman] INFO  
> o.a.d.exec.store.parquet.Metadata - Took 1 ms to get file statuses
> 2016-08-17 14:26:04,051 ucs-node4.perf.lab 
> [284b2996-d25a-d9af-178c-143b32ea6969:foreman] INFO  
> o.a.d.exec.store.parquet.Metadata - Fetch parquet metadata: Executed 1 out of 
> 1 using 1 threads. Time: 4ms total, 4.753323mm
> s avg, 4ms max.
> 2016-08-17 14:26:04,051 ucs-node4.perf.lab 
> [284b2996-d25a-d9af-178c-143b32ea6969:foreman] INFO  
> o.a.d.exec.store.parquet.Metadata - Fetch parquet metadata: Executed 1 out of 
> 1 using 1 threads. Earliest start: 12.261000
> μs, Latest start: 12.261000 μs, Average start: 12.261000 μs .
> 2016-08-17 14:26:04,051 ucs-node4.perf.lab 
> [284b2996-d25a-d9af-178c-143b32ea6969:foreman] INFO  
> o.a.d.exec.store.parquet.Metadata - Took 6 

[GitHub] drill pull request #775: DRILL-5326: Unit tests failures related to the SERV...

[vdiravka]

GitHub user vdiravka opened a pull request:

https://github.com/apache/drill/pull/775

DRILL-5326: Unit tests failures related to the SERVER_METADTA

- adding of the sql type name for the "GENERIC_OBJECT";
- changing "NullCollation" in the "ServerMetaProvider" to the correct 
default value;

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vdiravka/drill DRILL-5326

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/775.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #775


commit e7ca7650fa1bcc32638cfe4aade96aa56406a362
Author: Vitalii Diravka 
Date:   2017-03-07T20:53:03Z

DRILL-5326: Unit tests failures related to the SERVER_METADTA
- adding of the sql type name for the "GENERIC_OBJECT";
- changing "NullCollation" in the "ServerMetaProvider" to the correct 
default value;




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Time for 1.10 release

[Zelaine Fong]

It looks like DRILL-5326 has just been logged.  I assume we need this for RC0?

Vitalii – it looks like you have a fix.  Can you post a pull request as soon as 
you’re ready with the change.

Thanks.

-- Zelaine

On 3/3/17, 2:13 PM, "Jinfeng Ni"  wrote:

Pushed the patch for DRILL-5313, as well as two other patches which
were reviewed before 3/1.

I will start build RC0 after QA sanity test is completed.


On Thu, Mar 2, 2017 at 11:00 PM, Jinfeng Ni  wrote:
> Looks like the C++ client is not able to built successfully [1], after
> the change of DRILL-5301 / DRILL-5167.
>
> This seems to be a blocking issue for 1.10.0, and I'll merge the patch
> once it's verified/reviewed.
>
> 1. https://issues.apache.org/jira/browse/DRILL-5313
>
> On Thu, Mar 2, 2017 at 5:37 PM, Jinfeng Ni  wrote:
>> I missed 5208, because it did not show up in Paul's list when he replied 
to
>> this thread.
>>
>> On Thu, Mar 2, 2017 at 2:58 PM Zelaine Fong  wrote:
>>>
>>> Jinfeng,
>>>
>>> I notice the following Jira has the ready-to-commit label but isn’t on
>>> your list:
>>>
>>> DRILL-5208
>>>
>>> Was this one overlooked?
>>>
>>> -- Zelaine
>>>
>>> On 3/2/17, 1:04 PM, "Jinfeng Ni"  wrote:
>>>
>>> The following PRs have been merged to Apache master.
>>>
>>> DRILL-4994
>>> DRILL-4730
>>> DRILL-5301
>>> DRILL-5167
>>> DRILL-5221
>>> DRILL-5258
>>> DRILL-5034
>>> DRILL-4963
>>> DRILL-5252
>>> DRILL-5266
>>> DRILL-5284
>>> DRILL-5304
>>> DRILL-5290
>>> DRILL-5287
>>>
>>> QA folks will run tests. If no issue found, I'll build a RC0 
candidate
>>> for 1.10 and start the vote.
>>>
>>> Thanks,
>>>
>>> Jinfeng
>>>
>>>
>>>
>>> On Thu, Mar 2, 2017 at 8:30 AM, Jinfeng Ni  wrote:
>>> > I'm building a merge branch, and hopefully push to master branch
>>> today
>>> > if things go smoothly.
>>> >
>>> >
>>> > On Wed, Mar 1, 2017 at 7:13 PM, Padma Penumarthy
>>>  wrote:
>>> >> Hi Jinfeng,
>>> >>
>>> >> Please include DRILL-5287, DRILL-5290 and DRILL-5304.
>>> >>
>>> >> Thanks,
>>> >> Padma
>>> >>
>>> >>
>>> >>> On Feb 22, 2017, at 11:16 PM, Jinfeng Ni  
wrote:
>>> >>>
>>> >>> Hi Drillers,
>>> >>>
>>> >>> It has been almost 3 months since we release Drill 1.9. We have
>>> >>> resolved plenty of fixes and improvements (closed around 88 
JIRAs
>>> >>> [1]). I propose that we start the 1.10 release process, and set
>>> >>> Wednesday 3/1 as the cutoff day for code checkin. After 3/1, we
>>> should
>>> >>> start build a release candidate.
>>> >>>
>>> >>> Please reply in this email thread if you have something near
>>> complete
>>> >>> and you would like to include in 1.10 release.
>>> >>>
>>> >>> I volunteer as the release manager, unless someone else come
>>> forward.
>>> >>>
>>> >>> Thanks,
>>> >>>
>>> >>> Jinfeng
>>> >>>
>>> >>> [1]
>>> https://issues.apache.org/jira/browse/DRILL/fixforversion/12338769
>>> >>
>>>
>>>
>>

[jira] [Created] (DRILL-5326) Unexpected/unhandled MinorType value GENERIC_OBJECT

[Vitalii Diravka (JIRA)]

Vitalii Diravka created DRILL-5326:
--

 Summary: Unexpected/unhandled MinorType value GENERIC_OBJECT
 Key: DRILL-5326
 URL: https://issues.apache.org/jira/browse/DRILL-5326
 Project: Apache Drill
  Issue Type: Bug
  Components: Metadata
Affects Versions: 1.10.0
Reporter: Vitalii Diravka
Assignee: Vitalii Diravka
Priority: Blocker
 Fix For: 1.10.0


In DRILL-5301 a new SERVER_META rpc call was introduced. The server will 
support this method only from 1.10.0 drill version. For drill 1.10.0-SNAPHOT it 
is disabled. 
When I enabled this method (by way of upgrading drill version to 1.10.0 or 
1.11.0-SNAPSHOT) I found the following exception:
{code}
java.lang.AssertionError: Unexpected/unhandled MinorType value GENERIC_OBJECT
{code}
It appears in several tests (for example in 
DatabaseMetadataTest#testNullsAreSortedMethodsSaySortedHigh).
The reason of it is "GENERIC_OBJECT" RPC-/protobuf-level type is appeared in 
the ServerMetadata#ConvertSupportList. (Supporting of GENERIC_OBJECT was added 
in DRILL-1126).

The proposed solution is to add the appropriate "JAVA_OBJECT" sql type name for 
this "GENERIC_OBJECT" RPC-/protobuf-level data type.




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (DRILL-5324) Provide simplified column reader/writer for use in tests

[Paul Rogers (JIRA)]

Paul Rogers created DRILL-5324:
--

 Summary: Provide simplified column reader/writer for use in tests
 Key: DRILL-5324
 URL: https://issues.apache.org/jira/browse/DRILL-5324
 Project: Apache Drill
  Issue Type: Sub-task
Affects Versions: 1.11.0
Reporter: Paul Rogers
Assignee: Paul Rogers
 Fix For: 1.11.0


In support of DRILL-, we wish to provide a very easy way to work with row 
sets. See the comment section for examples of the target API.

Drill provides over 100 different value vectors, any of which may be required 
to perform a specific unit test. Creating these vectors, populating them, and 
retrieving values, is very tedious. The work is so complex that it acts to 
discourage developers from writing such tests.

To simplify the task, we wish to provide a simplified row set reader and 
writer. To do that, we need to generate the corresponding column reader and 
writer for each value vector. This ticket focuses on the column-level readers 
and writers, and the required code generation.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (DRILL-5323) Provide test tools to create, populate and compare row sets

[Paul Rogers (JIRA)]

Paul Rogers created DRILL-5323:
--

 Summary: Provide test tools to create, populate and compare row 
sets
 Key: DRILL-5323
 URL: https://issues.apache.org/jira/browse/DRILL-5323
 Project: Apache Drill
  Issue Type: Sub-task
Affects Versions: 1.11.0
Reporter: Paul Rogers
Assignee: Paul Rogers
 Fix For: 1.11.0


Operators work with individual row sets. A row set is a collection of records 
stored as column vectors. (Drill uses various terms for this concept. A record 
batch is a row set with an operator implementation wrapped around it. A vector 
container is a row set, but with much functionality left as an exercise for the 
developer. And so on.)

To simplify tests, we need a {{TestRowSet}} concept that wraps a 
{{VectorContainer}} and provides easy ways to:

* Define a schema for the row set.
* Create a set of vectors that implement the schema.
* Populate the row set with test data via code.
* Add an SV2 to the row set.
* Pass the row set to operator components (such as generated code blocks.)
* Compare the results of the operation with an expected result set.
* Dispose of the underling direct memory when work is done.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (DRILL-5322) Provide an OperatorFixture for sub-operator unit testing setup

[Paul Rogers (JIRA)]

Paul Rogers created DRILL-5322:
--

 Summary: Provide an OperatorFixture for sub-operator unit testing 
setup
 Key: DRILL-5322
 URL: https://issues.apache.org/jira/browse/DRILL-5322
 Project: Apache Drill
  Issue Type: Sub-task
Affects Versions: 1.11.0
Reporter: Paul Rogers
Assignee: Paul Rogers
 Fix For: 1.11.0


We recently created various "fixture" classes to assist with system-level 
testing: {{LogFixture}}, {{ClusterFixture}} and {{ClientFixture}}. Each handles 
the tedious work of setting up the conditions to run certain kinds of tests.

In the same way, we need an {{OperatorFixture}} to set up the low-level bits 
and pieces needed for operator-level, and sub-operator-level unit testing.

The {{DrillConfig}} is used by both the system-level and operator-level 
fixtures. So, pull the config-setup tasks our of (cluster) {{FixtureBuilder}} 
(should rename) and into a new {{ConfigBuilder}}. Leave the existing methods in 
{{FixtureBuilder}}, but modify them to be wrappers around the new config 
builder.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (DRILL-5321) Refactor FragmentContext for unit testing

[Paul Rogers (JIRA)]

Paul Rogers created DRILL-5321:
--

 Summary: Refactor FragmentContext for unit testing
 Key: DRILL-5321
 URL: https://issues.apache.org/jira/browse/DRILL-5321
 Project: Apache Drill
  Issue Type: Sub-task
Affects Versions: 1.10.0
Reporter: Paul Rogers
Assignee: Paul Rogers
 Fix For: Future


Each operator has visibility to the {{FragmentContext}} class. 
{{FragmentContext}} provides access to all of Drill internals: the Drillbit 
context, the network interfaces, RPC messages and so on.

Further, all the code generation mechanisms require a {{FragmentContext}} 
object.

This structure creates a large barrier to unit testing. To test, say, a 
particular bit of generated code, we must have the entire Drillbit running so 
we can obtain a {{FragmentContext}}. Clearly, this is less than ideal.

Upon inspection, it turns out that the {{FragmentContext}} is mostly needed, by 
many operators, to generate code. Of the many methods in {{FragmentContext}}, 
code generation uses only six.

The solution is to create a new super-interface, {{CodeGenContext}}, which 
holds those six methods. The {{CodeGenContext}} can be easily re-implemented 
for unit testing.

Then, modify all the code-generation classes that currently take 
{{FragmentContext}} to take {{CodeGenContext}} instead.

Since {{FragmentContext}} derives from {{CodeGenContext}}, existing operator 
code "just works."



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (DRILL-5320) Refactor OptionManager to allow better unit testing

[Paul Rogers (JIRA)]

Paul Rogers created DRILL-5320:
--

 Summary: Refactor OptionManager to allow better unit testing
 Key: DRILL-5320
 URL: https://issues.apache.org/jira/browse/DRILL-5320
 Project: Apache Drill
  Issue Type: Sub-task
Affects Versions: 1.10.0
Reporter: Paul Rogers
Assignee: Paul Rogers
 Fix For: Future


The {{OptionManager}} interface serves two purposes:

* Create and modify options
* Access option values

The implementations  of this class are integrated with the rest of Drill, 
making it difficult to use the classes in isolation in unit testing. Further, 
since operators are given the full interface, the operator has the ability to 
modify options, and so each unit test should either verify that no modification 
is, in fact, done, or must track down modifications and test them.

For operator and sub-operator unit tests we need a simpler interface. As it 
turns out, most low-level uses of {{OptionManager}} are all read-only. This 
allows a simple refactoring to enhance unit testability: create a new 
super-interface {{OptionSet}}, which provides only the read-only methods.

Then, refactor low-level classes (code generation, compilers, and so on) to use 
the restricted {{OptionSet}} interface.

Finally, for unit tests, create a trivial, map-based implementation that can be 
populated as needed for each specific test.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (DRILL-5318) Create a sub-operator test framework

[Paul Rogers (JIRA)]

Paul Rogers created DRILL-5318:
--

 Summary: Create a sub-operator test framework
 Key: DRILL-5318
 URL: https://issues.apache.org/jira/browse/DRILL-5318
 Project: Apache Drill
  Issue Type: Improvement
  Components: Tools, Build & Test
Affects Versions: 1.10.0
Reporter: Paul Rogers
Assignee: Paul Rogers
 Fix For: Future


Drill provides two unit test frameworks for whole-server, SQL-based testing: 
the original {{BaseTestQuery}} and the newer {{ClusterFixture}}. Both use the 
{{TestBuilder}} mechanism to build system-level functional tests that run 
queries and check results.

Jason provided an operator-level test framework based, in part on mocks: 

As Drill operators become more complex, we have a crying need for true 
unit-level tests at a level below the whole system and below operators. That 
is, we need to test the individual pieces that, together, form the operator.

This umbrella ticket includes a number of tasks needed to create the 
sub-operator framework. Our intention is that, over time, as we find the need 
to revisit existing operators, or create new ones, we can employ the 
sub-operator test framework to exercise code at a finer granularity than is 
possible prior to this framework.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] drill pull request #774: OpenTSDB storage plugin

[dmitriyHavrilovich]

GitHub user dmitriyHavrilovich opened a pull request:

https://github.com/apache/drill/pull/774

OpenTSDB storage plugin

Implementation of TSDB storage plugin. Plugin uses REST API to work with 
TSDB. Supported queries for now are listed below: 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mapr-demos/drill master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/774.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #774


commit 83cb33523f42b0abf9e5dfed6b1ee104fe3e1bcf
Author: Vlad Storona 
Date:   2016-11-25T18:28:02Z

openTSDB plugin: initial commit

commit 20813645fa72513066c041fdb9ab93aa540fc6f5
Author: Vlad Storona 
Date:   2016-12-02T12:31:42Z

Implement getting table names from openTSDB

commit 319c2d2f21d30cc70d1b183470ea8315d3a5381b
Author: Vlad Storona 
Date:   2016-12-19T11:51:15Z

initial commit, add basic structure

commit 7b94619aca525a54e612708f93fac71dc71d5281
Author: Vlad Storona 
Date:   2016-12-27T07:37:50Z

Partialy implement SELECT from openTSDB

commit c65e2e8f9aa973bbff76d37cc55ba2dc9bd5b5f2
Author: Vlad Storona 
Date:   2017-01-12T20:51:04Z

Change table schema

commit f70bfbb14083fb1e1041f4481257a690eda2d61b
Author: Vlad Storona 
Date:   2017-01-16T17:42:51Z

Implemented getting multiple rows from openTSDB table

commit d9643c220c21ce42e16aa21fea322898b130444a
Author: Vlad Storona 
Date:   2017-01-17T14:06:58Z

Refactor Record Reader

commit 18cd35e53fa77de998e685b3834ec4134a64
Author: Vlad Storona 
Date:   2017-01-17T19:03:32Z

#1 Implemented aggregation function support

commit df81c967922fa4416d64d9e496a880f58b38c6bc
Author: Vlad Storona 
Date:   2017-01-18T14:48:51Z

#4 Fixed issue with data duplication

commit 2505e7cbc885446971f3cb9f632ae4e16f6a8461
Author: Dmitriy Gavrilovich 
Date:   2017-01-31T12:44:26Z

Open TSDB plugin initial commit

commit a2019c9108c500e70b26b6839797f5f542ca6233
Author: dmitriyHavrilovich 
Date:   2017-01-31T23:31:29Z

Merge pull request #1 from mapr-demos/openTSDB-plugin

Open tsdb plugin

commit e85931c84839774d715effea6ec0dd56a0ba87a5
Author: Vlad Storona 
Date:   2017-02-01T18:41:05Z

Change FROM syntax in query

commit 7ce213f44b880f4e4e4fe5567829a66c4e8ae82f
Author: Vlad Storona 
Date:   2017-02-01T18:44:43Z

Fix formatting

commit 25f05816589c83f710c2fed18f3039530e97c797
Author: Vlad Storona 
Date:   2017-02-03T18:50:02Z

Tag map divided into separate columns

commit 3deff2175fd34a456a3152c744e82959348fac09
Author: Vlad Storona 
Date:   2017-02-06T14:57:55Z

#5 Add name validation

commit ffd2da2911b6131f8def9f290b6801fbe7b0589b
Author: Vlad Storona 
Date:   2017-02-07T08:43:25Z

#6 Change kind of Exception to more clear form

commit 871ab712ee23809bd81132e2cca0ee2d6b8d6526
Author: Vlad Storona 
Date:   2017-02-07T18:35:04Z

#7 Add parameters validation

commit f6387452497de2daabb2c95afe73d868ceb5188a
Author: Vlad Storona 
Date:   2017-02-07T18:36:50Z

Refactor Schema

commit b44f861e2817be5aadc2e396d590e2e17ba1
Author: Vlad Storona 
Date:   2017-02-08T16:17:29Z

#8 Add tests

commit 6fb8701daee7de5e2ac74805616340162fedc0e8
Author: Vlad Storona 
Date:   2017-02-09T16:31:15Z

#8 Add printing and counting rows in tests

commit e5d5da107d6b376418a0ff4e352be024048031c0
Author: Vlad Storona 
Date:   2017-02-15T14:12:01Z

Add downsample support

commit 2c0bf049842c79b05378c29c24cdebcfaeecbc37
Author: Vlad Storona 
Date:   2017-02-15T15:35:45Z

Merge

commit 682179916948c028109d4de149c48324932ae21f
Author: Vlad Storona 
Date:   2017-02-15T15:42:39Z

Merge pull request #2 from mapr-demos/openTSDB-plugin

Open tsdb plugin

commit ad4cab46729a1f14269110c78803e4db4a7f3c5b
Author: Vlad Storona 
Date:   2017-02-15T17:55:53Z

Change encoding to UTF_8 in test classes

commit bd53e6ac0db7ec7568657a83321fdbe41cc7b6b2
Author: Vlad Storona 
Date:   2017-03-04T14:56:21Z

Fix reader, add POST API request

commit 2757d8903d0d2f4087c6b74f9b253e4f57acdc99

[jira] [Created] (DRILL-5317) Drill Configuration to access S3 buckets in Mumbai region

[shivamurthy.dee...@gmail.com (JIRA)]

shivamurthy.dee...@gmail.com created DRILL-5317:
---

 Summary: Drill Configuration to access S3 buckets in Mumbai region
 Key: DRILL-5317
 URL: https://issues.apache.org/jira/browse/DRILL-5317
 Project: Apache Drill
  Issue Type: Bug
  Components: Functions - Drill
Affects Versions: 1.8.0
Reporter: shivamurthy.dee...@gmail.com


I am able to access and query S3 buckets in US standard region, but not able to 
access/query buckets in Mumbai region. Is there any specific configuration that 
needs to be enabled on Drill?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] drill pull request #772: DRILL-5316: Check drillbitsVector count from zoo_ge...

[superbstreak]

Github user superbstreak commented on a diff in the pull request:

https://github.com/apache/drill/pull/772#discussion_r104627768
  
--- Diff: contrib/native/client/src/clientlib/drillClientImpl.cpp ---
@@ -86,6 +86,9 @@ connectionStatus_t DrillClientImpl::connect(const char* 
connStr, DrillUserProper
 std::vector drillbits;
 int err = zook.getAllDrillbits(hostPortStr, drillbits);
 if(!err){
+if (drillbits.empty()){
+return handleConnError(CONN_INVALID_INPUT, 
getMessage(ERR_CONN_ZKNODBIT));
--- End diff --

double check CONN_INVALID_INPUT usage.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #772: DRILL-5316: Check drillbitsVector count from zoo_ge...

[superbstreak]

Github user superbstreak commented on a diff in the pull request:

https://github.com/apache/drill/pull/772#discussion_r104607726
  
--- Diff: contrib/native/client/src/clientlib/zookeeperClient.cpp ---
@@ -138,6 +138,11 @@ int ZookeeperClient::getAllDrillbits(const 
std::string& connectStr, std::vector<
 DRILL_MT_LOG(DRILL_LOG(LOG_TRACE) << "\t Unshuffled Drillbit 
id: " << drillbits[i] << std::endl;)
 }
 }
+else{
--- End diff --

Thanks both. Yea make sense, I'll make the change.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #772: DRILL-5316: Check drillbitsVector count from zoo_ge...

[sohami]

Github user sohami commented on a diff in the pull request:

https://github.com/apache/drill/pull/772#discussion_r104606833
  
--- Diff: contrib/native/client/src/clientlib/zookeeperClient.cpp ---
@@ -138,6 +138,11 @@ int ZookeeperClient::getAllDrillbits(const 
std::string& connectStr, std::vector<
 DRILL_MT_LOG(DRILL_LOG(LOG_TRACE) << "\t Unshuffled Drillbit 
id: " << drillbits[i] << std::endl;)
 }
 }
+else{
--- End diff --

Agreed. Should be handled in caller (i.e. DrillClient). If the returned 
vector size is zero then we should check that in DrillClient and close client 
connection with error as `ERR_CONN_ZKNODBIT`. Something like below:

`return handleConnError(CONN_INVALID_INPUT, getMessage(ERR_CONN_ZKNODBIT, 
pathToDrill.c_str()));`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---