[jira] [Commented] (DRILL-5221) cancel message is delayed until queryid or data is received

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852586#comment-15852586
 ] 

ASF GitHub Bot commented on DRILL-5221:
---

Github user superbstreak commented on the issue:

https://github.com/apache/drill/pull/733
  
Did some testing. This allow query to cancel more precisely in certain 
flows. Good.


> cancel message is delayed until queryid or data is received
> ---
>
> Key: DRILL-5221
> URL: https://issues.apache.org/jira/browse/DRILL-5221
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Client - C++
>Affects Versions: 1.9.0
>Reporter: Laurent Goujon
>Assignee: Laurent Goujon
>
> When user is calling the cancel method of the C++ client, the client wait for 
> a message from the server to reply back with a cancellation message.
> In case of queries taking a long time to return batch results, it means 
> cancellation won't be effective until the next batch is received, instead of 
> cancelling right away the query (assuming the query id has already been 
> received, which is generally the case).
> It seems this was foreseen by [~vkorukanti] in his initial patch 
> (https://github.com/vkorukanti/drill/commit/e0ef6349aac48de5828b6d725c2cf013905d18eb)
>  but was omitted when I backported it post metadata changes.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (DRILL-5207) Improve Parquet scan pipelining

[Parth Chandra (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Parth Chandra updated DRILL-5207:
-
Labels: doc-impacting  (was: ready-to-commit)

> Improve Parquet scan pipelining
> ---
>
> Key: DRILL-5207
> URL: https://issues.apache.org/jira/browse/DRILL-5207
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Parquet
>Affects Versions: 1.9.0
>Reporter: Parth Chandra
>Assignee: Parth Chandra
>  Labels: doc-impacting
> Fix For: 1.10.0
>
>
> The parquet reader's async page reader is not quite efficiently pipelined. 
> The default size of the disk read buffer is 4MB while the page reader reads 
> ~1MB at a time. The Parquet decode is also processing 1MB at a time. This 
> means the disk is idle while the data is being processed. Reducing the buffer 
> to 1MB will reduce the time the processing thread waits for the disk read 
> thread.
> Additionally, since the data to process a page may be more or less than 1MB, 
> a queue of pages will help so that the disk scan does not block (until the 
> queue is full), waiting for the processing thread.
> Additionally, the BufferedDirectBufInputStream class reads from disk as soon 
> as it is initialized. Since this is called at setup time, this increases the 
> setup time for the query and query execution does not begin until this is 
> completed.
> There are a few other inefficiencies - options are read every time a page 
> reader is created. Reading options can be expensive.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5238) CTTAS: unable to resolve temporary table if workspace is indicated without schema

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852562#comment-15852562
 ] 

ASF GitHub Bot commented on DRILL-5238:
---

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/736


> CTTAS: unable to resolve temporary table if workspace is indicated without 
> schema
> -
>
> Key: DRILL-5238
> URL: https://issues.apache.org/jira/browse/DRILL-5238
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.10.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>  Labels: ready-to-commit
> Fix For: 1.10.0
>
>
> Drill is unable to resolve temporary table if default temporary workspace is 
> partially indicated (schema was set using USE command and only workspace name 
> is used in query).
> {noformat}
> 0: jdbc:drill:zk=local> use dfs;
> +---+--+
> |  ok   | summary  |
> +---+--+
> | true  | Default schema changed to [dfs]  |
> +---+--+
> 1 row selected (0.156 seconds)
> 0: jdbc:drill:zk=local> create temporary table tmp.t as select 'A' from 
> (values(1));
> +---++
> | Fragment  | Number of records written  |
> +---++
> | 0_0   | 1  |
> +---++
> 1 row selected (1.525 seconds)
> 0: jdbc:drill:zk=local> select * from tmp.t;
> Feb 02, 2017 11:24:09 AM 
> org.apache.calcite.sql.validate.SqlValidatorException 
> SEVERE: org.apache.calcite.sql.validate.SqlValidatorException: Table 'tmp.t' 
> not found
> Feb 02, 2017 11:24:09 AM org.apache.calcite.runtime.CalciteException 
> SEVERE: org.apache.calcite.runtime.CalciteContextException: From line 1, 
> column 15 to line 1, column 17: Table 'tmp.t' not found
> Error: VALIDATION ERROR: From line 1, column 15 to line 1, column 17: Table 
> 'tmp.t' not found
> SQL Query null
> [Error Id: 5266cb67-9d37-4a94-9a4e-28a4a2f94be5 on localhost:31010] 
> (state=,code=0)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5207) Improve Parquet scan pipelining

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852558#comment-15852558
 ] 

ASF GitHub Bot commented on DRILL-5207:
---

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/723


> Improve Parquet scan pipelining
> ---
>
> Key: DRILL-5207
> URL: https://issues.apache.org/jira/browse/DRILL-5207
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Parquet
>Affects Versions: 1.9.0
>Reporter: Parth Chandra
>Assignee: Parth Chandra
>  Labels: ready-to-commit
> Fix For: 1.10.0
>
>
> The parquet reader's async page reader is not quite efficiently pipelined. 
> The default size of the disk read buffer is 4MB while the page reader reads 
> ~1MB at a time. The Parquet decode is also processing 1MB at a time. This 
> means the disk is idle while the data is being processed. Reducing the buffer 
> to 1MB will reduce the time the processing thread waits for the disk read 
> thread.
> Additionally, since the data to process a page may be more or less than 1MB, 
> a queue of pages will help so that the disk scan does not block (until the 
> queue is full), waiting for the processing thread.
> Additionally, the BufferedDirectBufInputStream class reads from disk as soon 
> as it is initialized. Since this is called at setup time, this increases the 
> setup time for the query and query execution does not begin until this is 
> completed.
> There are a few other inefficiencies - options are read every time a page 
> reader is created. Reading options can be expensive.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5220) Add api to set application name in C++ connector

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852559#comment-15852559
 ] 

ASF GitHub Bot commented on DRILL-5220:
---

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/728


> Add api to set application name in C++ connector
> 
>
> Key: DRILL-5220
> URL: https://issues.apache.org/jira/browse/DRILL-5220
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Client - C++
>Affects Versions: 1.8.0
>Reporter: Laurent Goujon
>Assignee: Laurent Goujon
>Priority: Minor
>  Labels: ready-to-commit
>
> There's no API for a C++ connector user to specify the name of the 
> application, and to provide it to the server (optional field added in 
> DRILL-4369)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5237) FlattenRecordBatch loses nested fields from the schema when returns empty batches for the first time

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852561#comment-15852561
 ] 

ASF GitHub Bot commented on DRILL-5237:
---

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/735


> FlattenRecordBatch loses nested fields from the schema when returns empty 
> batches for the first time
> 
>
> Key: DRILL-5237
> URL: https://issues.apache.org/jira/browse/DRILL-5237
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Serhii Harnyk
>Assignee: Serhii Harnyk
>  Labels: ready-to-commit
> Attachments: test_data.tar.gz
>
>
> Query 
> {code:sql}
> select count(*) as col from (select t1.a.a1 from (select t.*, flatten(t.b) as 
> b from dfs.`/tmp/test_data` t where t.c is not null) t1 where t1.a .a1 like 
> 'a1');
> {code}
> returns different results for different values for option 
> planner.width.max_per_node
> With options 
> {code:sql}
> alter session set `planner.slice_target` = 1;
> alter session set `planner.width.max_per_node` = 1;
> {code}
> query returns correct result:
> {noformat}
> +--+
> | col  |
> +--+
> | 3|
> +--+
> {noformat}
> but with options 
> {code:sql}
> alter session set `planner.slice_target` = 1;
> alter session set `planner.width.max_per_node` = 3;
> {code}
> the same query returns wrong result 
> {noformat}
> +--+
> | col  |
> +--+
> | 2|
> +--+
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5224) CTTAS: fix errors connected with system path delimiters (Windows)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852560#comment-15852560
 ] 

ASF GitHub Bot commented on DRILL-5224:
---

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/731


> CTTAS: fix errors connected with system path delimiters (Windows)
> -
>
> Key: DRILL-5224
> URL: https://issues.apache.org/jira/browse/DRILL-5224
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.10.0
> Environment: Windows 10
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>  Labels: ready-to-commit
> Fix For: 1.10.0
>
>
> Problem 1:
> Error occurs when attempting to create temporary table on Windows:
> {noformat}
> 0: jdbc:drill:zk=local> create temporary table t as select * from sys.version;
> Error: SYSTEM ERROR: InvalidPathException: Illegal char <:> at index 4: 
> file:///\tmp\3191db8e-279d-4ced-b0e5-32b3b477edfb
> {noformat}
> Root cause:
> when creating temporary directory we merge file system uri, temporary 
> workspace location and session id into one path using 
> java.nio.file.Paths.get() but this method cannot resolve path when path has 
> different delimiters.
> Fix:
> Use org.apache.hadoop.fs.Path tools to merge path, path string is normalized 
> during creation.
> {noformat}
> new Path(fs.getUri().toString(), new 
> Path(temporaryWorkspace.getDefaultLocation(), sessionId));
> {noformat}
> Problem 2:
> When temporary table is being manually dropped using drop table command, 
> though actual table is dropped, remnant folder is left.
> Root cause:
> Before adding to temporary table to the list of temporary tables, its 
> generated name is concatenated with session id (as parent and child folders). 
> java.nio.file.Paths.get() is used for concatenation but it preserves current 
> system delimiter. When table is being dropped, passed table name is split 
> using org.apache.hadoop.fs.Path.SEPARATOR, since it's assumed that path was 
> created using org.apache.hadoop.fs.Path tools where path separators are 
> normalized to one format disregarding the system.
> Fix:
> Concatenate session id with generated table name using 
> org.apache.hadoop.fs.Path tools.
> {noformat}
> new Path(sessionId, UUID.randomUUID().toString()).toUri().getPath();
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5215) CTTAS: disallow temp tables in view expansion logic

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852557#comment-15852557
 ] 

ASF GitHub Bot commented on DRILL-5215:
---

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/725


> CTTAS: disallow temp tables in view expansion logic
> ---
>
> Key: DRILL-5215
> URL: https://issues.apache.org/jira/browse/DRILL-5215
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.10.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>  Labels: ready-to-commit
> Fix For: 1.10.0
>
>
> This Jira is addressing last two CR comments in PR for Jira DRILL-4956:
> 1. Disallow temp tables in view expansion logic.
> Steps to reproduce:
> a. use default temporary workspace.
> b. create persistent table with name t:
> c. create view over persistent table.
> d. drop persistent table.
> e. create temporary table with the same name as persistent table.
> f. select from view 
> Currently Drill returns result on step f but should return error message.
> {noformat}
> VALIDATION ERROR: Temporary tables usage is disallowed. Used temporary table 
> name: t
> {noformat}
> {noformat}
> use dfs.tmp;
> create table t as select 'TABLE' from (values(1));
> create view v as select * from t;
> select * from v;
> +-+
> | EXPR$0  |
> +-+
> | TABLE   |
> +-+
> drop table t;
> create temporary table t as select 'TEMP' from (values(1));
> select * from v;
> +-+
> | EXPR$0  |
> +-+
> | TEMP|
> +-+
> {noformat}
> 2. Replace link to gist with CTTAS design doc to Jira link.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5240) Parquet reader creates unnecessary objects when checking for nullability in var length columns

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852563#comment-15852563
 ] 

ASF GitHub Bot commented on DRILL-5240:
---

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/740


> Parquet reader creates unnecessary objects when checking for nullability in 
> var length columns
> --
>
> Key: DRILL-5240
> URL: https://issues.apache.org/jira/browse/DRILL-5240
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Parquet
>Reporter: Parth Chandra
>Assignee: Parth Chandra
>  Labels: ready-to-commit
>
> In {{NullableVarLengthValuesColumn.java}} we have the following line of code:
> {code}
> currentValNull = 
> variableWidthVector.getAccessor().getObject(valuesReadInCurrentPass) == null;
> {code}
> which creates an object to check nullability of every (nullable var char) 
> value being read. This object creation is expensive and can be replaced with 
> a simple check for the null bit.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5224) CTTAS: fix errors connected with system path delimiters (Windows)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852556#comment-15852556
 ] 

ASF GitHub Bot commented on DRILL-5224:
---

Github user parthchandra commented on the issue:

https://github.com/apache/drill/pull/731
  
+1


> CTTAS: fix errors connected with system path delimiters (Windows)
> -
>
> Key: DRILL-5224
> URL: https://issues.apache.org/jira/browse/DRILL-5224
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.10.0
> Environment: Windows 10
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>  Labels: ready-to-commit
> Fix For: 1.10.0
>
>
> Problem 1:
> Error occurs when attempting to create temporary table on Windows:
> {noformat}
> 0: jdbc:drill:zk=local> create temporary table t as select * from sys.version;
> Error: SYSTEM ERROR: InvalidPathException: Illegal char <:> at index 4: 
> file:///\tmp\3191db8e-279d-4ced-b0e5-32b3b477edfb
> {noformat}
> Root cause:
> when creating temporary directory we merge file system uri, temporary 
> workspace location and session id into one path using 
> java.nio.file.Paths.get() but this method cannot resolve path when path has 
> different delimiters.
> Fix:
> Use org.apache.hadoop.fs.Path tools to merge path, path string is normalized 
> during creation.
> {noformat}
> new Path(fs.getUri().toString(), new 
> Path(temporaryWorkspace.getDefaultLocation(), sessionId));
> {noformat}
> Problem 2:
> When temporary table is being manually dropped using drop table command, 
> though actual table is dropped, remnant folder is left.
> Root cause:
> Before adding to temporary table to the list of temporary tables, its 
> generated name is concatenated with session id (as parent and child folders). 
> java.nio.file.Paths.get() is used for concatenation but it preserves current 
> system delimiter. When table is being dropped, passed table name is split 
> using org.apache.hadoop.fs.Path.SEPARATOR, since it's assumed that path was 
> created using org.apache.hadoop.fs.Path tools where path separators are 
> normalized to one format disregarding the system.
> Fix:
> Concatenate session id with generated table name using 
> org.apache.hadoop.fs.Path tools.
> {noformat}
> new Path(sessionId, UUID.randomUUID().toString()).toUri().getPath();
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5196) Could not run a single MongoDB unit test case through command line or IDE

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852500#comment-15852500
 ] 

ASF GitHub Bot commented on DRILL-5196:
---

Github user chunhui-shi commented on a diff in the pull request:

https://github.com/apache/drill/pull/741#discussion_r99455997
  
--- Diff: 
contrib/storage-mongo/src/test/java/org/apache/drill/exec/store/mongo/MongoTestSuit.java
 ---
@@ -204,16 +209,20 @@ private static void cleanup() {
   @BeforeClass
   public static void initMongo() throws Exception {
 synchronized (MongoTestSuit.class) {
-  if (distMode) {
-logger.info("Executing tests in distributed mode");
-DistributedMode.setup();
-  } else {
-logger.info("Executing tests in single mode");
-SingleMode.setup();
+  if (initCount.get() == 0) {
--- End diff --

Then it will increase the count when the init is not done. If init fail in 
the middle, then another test has no chance to try.


> Could not run a single MongoDB unit test case through command line or IDE
> -
>
> Key: DRILL-5196
> URL: https://issues.apache.org/jira/browse/DRILL-5196
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Chunhui Shi
>Assignee: Chunhui Shi
>
> Could not run a single MongoDB's unit test through IDE or command line. The 
> reason is when running a single test case, the MongoDB instance did not get 
> started thus a 'table not found' error for 'mongo.employee.empinfo' would be 
> raised.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5190) Display planning time for a query in its profile page

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852475#comment-15852475
 ] 

ASF GitHub Bot commented on DRILL-5190:
---

Github user kkhatua commented on a diff in the pull request:

https://github.com/apache/drill/pull/738#discussion_r99454561
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/profile/ProfileResources.java
 ---
@@ -77,12 +77,14 @@
*/
   public static String getPrettyDuration(long startTimeMillis, long 
endTimeMillis) {
 long durationInMillis = (startTimeMillis > endTimeMillis ? 
System.currentTimeMillis() : endTimeMillis) - startTimeMillis;
-long hours = TimeUnit.MILLISECONDS.toHours(durationInMillis);
+long days = TimeUnit.MILLISECONDS.toDays(durationInMillis);
--- End diff --

Initializing an object for rendering compact or verbose duration formats 
might be an overkill. 
I'll change the signature for the existing method _getPrettyDuration_ in 
the _utility_ class - `ProfileResources.java` to carry an additional parameter 
to indicating verbose or compact representation and apply that in DRILL-5230. I 
can rebase this commit on top of that. 


> Display planning time for a query in its profile page
> -
>
> Key: DRILL-5190
> URL: https://issues.apache.org/jira/browse/DRILL-5190
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Web Server
>Affects Versions: 1.9.0
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>
> Currently, the Web UI does not display the time spent in planning for a query 
> in its profile page. The estimate needs to be done by seeing how late did the 
> earliest major fragment start. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5242) The UI breaks when trying to render profiles having unknown metrics

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852457#comment-15852457
 ] 

ASF GitHub Bot commented on DRILL-5242:
---

GitHub user kkhatua opened a pull request:

https://github.com/apache/drill/pull/742

DRILL-5242: The UI breaks when rendering profiles having unknown metrics

Skip any metrics whose metric ID is unknown, This prevents any 
ArrayIndexOutOfBoundsException from being thrown and breaking the UI rendering.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kkhatua/drill DRILL-5242

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/742.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #742


commit a949c99e099a46aad5462a115a30004a2712fa37
Author: Kunal Khatua 
Date:   2017-02-04T01:29:47Z

DRILL-5242: The UI breaks when trying to render profiles having unknown 
metrics

Skip any metrics whose metric ID is unknown, This prevents any 
ArrayIndexOutOfBoundsException from being thrown and breaking the UI rendering.




> The UI breaks when trying to render profiles having unknown metrics
> ---
>
> Key: DRILL-5242
> URL: https://issues.apache.org/jira/browse/DRILL-5242
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Web Server
>Affects Versions: 1.9.0
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
> Fix For: 1.10.0
>
>
> When profiles are generated using a fork of Drill that has introduced new 
> metrics, the server of the parent branch will fail in rendering the operator 
> metrics correctly. 
> The workaround should be to simply skip unknown metrics. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (DRILL-5242) The UI breaks when trying to render profiles having unknown metrics

[Kunal Khatua (JIRA)]

Kunal Khatua created DRILL-5242:
---

 Summary: The UI breaks when trying to render profiles having 
unknown metrics
 Key: DRILL-5242
 URL: https://issues.apache.org/jira/browse/DRILL-5242
 Project: Apache Drill
  Issue Type: Bug
  Components: Web Server
Affects Versions: 1.9.0
Reporter: Kunal Khatua
Assignee: Kunal Khatua
 Fix For: 1.10.0


When profiles are generated using a fork of Drill that has introduced new 
metrics, the server of the parent branch will fail in rendering the operator 
metrics correctly. 
The workaround should be to simply skip unknown metrics. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5196) Could not run a single MongoDB unit test case through command line or IDE

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852446#comment-15852446
 ] 

ASF GitHub Bot commented on DRILL-5196:
---

Github user chunhui-shi commented on a diff in the pull request:

https://github.com/apache/drill/pull/741#discussion_r99453293
  
--- Diff: 
contrib/storage-mongo/src/test/java/org/apache/drill/exec/store/mongo/MongoTestSuit.java
 ---
@@ -234,15 +243,25 @@ private static void createDbAndCollections(String 
dbName,
 
   @AfterClass
   public static void tearDownCluster() throws Exception {
-if (mongoClient != null) {
-  mongoClient.dropDatabase(EMPLOYEE_DB);
-  mongoClient.close();
-}
 synchronized (MongoTestSuit.class) {
-  if (distMode) {
-DistributedMode.cleanup();
-  } else {
-SingleMode.cleanup();
+  if (initCount.decrementAndGet() == 0) {
+try {
+  if (mongoClient != null) {
--- End diff --

Then we will end up not to cleanup the cluster if mongoClient for some 
reason was set to null? I think it is better leave it as is.


> Could not run a single MongoDB unit test case through command line or IDE
> -
>
> Key: DRILL-5196
> URL: https://issues.apache.org/jira/browse/DRILL-5196
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Chunhui Shi
>Assignee: Chunhui Shi
>
> Could not run a single MongoDB's unit test through IDE or command line. The 
> reason is when running a single test case, the MongoDB instance did not get 
> started thus a 'table not found' error for 'mongo.employee.empinfo' would be 
> raised.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5196) Could not run a single MongoDB unit test case through command line or IDE

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852438#comment-15852438
 ] 

ASF GitHub Bot commented on DRILL-5196:
---

Github user chunhui-shi commented on a diff in the pull request:

https://github.com/apache/drill/pull/741#discussion_r99452939
  
--- Diff: 
contrib/storage-mongo/src/test/java/org/apache/drill/exec/store/mongo/MongoTestSuit.java
 ---
@@ -234,15 +243,25 @@ private static void createDbAndCollections(String 
dbName,
 
   @AfterClass
   public static void tearDownCluster() throws Exception {
-if (mongoClient != null) {
-  mongoClient.dropDatabase(EMPLOYEE_DB);
-  mongoClient.close();
-}
 synchronized (MongoTestSuit.class) {
-  if (distMode) {
-DistributedMode.cleanup();
-  } else {
-SingleMode.cleanup();
+  if (initCount.decrementAndGet() == 0) {
--- End diff --

Without Synchronized, there is still chance two threads go into this place. 
while thread 0 go to here and is in the middle of tearing down cluster, then 
thread 1 increases initCount to 1 and somehow report error then come to here to 
decrease initCount and continue... Then we don't know what will happen.


> Could not run a single MongoDB unit test case through command line or IDE
> -
>
> Key: DRILL-5196
> URL: https://issues.apache.org/jira/browse/DRILL-5196
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Chunhui Shi
>Assignee: Chunhui Shi
>
> Could not run a single MongoDB's unit test through IDE or command line. The 
> reason is when running a single test case, the MongoDB instance did not get 
> started thus a 'table not found' error for 'mongo.employee.empinfo' would be 
> raised.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5196) Could not run a single MongoDB unit test case through command line or IDE

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852410#comment-15852410
 ] 

ASF GitHub Bot commented on DRILL-5196:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/741#discussion_r99451363
  
--- Diff: 
contrib/storage-mongo/src/test/java/org/apache/drill/exec/store/mongo/MongoTestSuit.java
 ---
@@ -204,16 +209,20 @@ private static void cleanup() {
   @BeforeClass
   public static void initMongo() throws Exception {
 synchronized (MongoTestSuit.class) {
-  if (distMode) {
-logger.info("Executing tests in distributed mode");
-DistributedMode.setup();
-  } else {
-logger.info("Executing tests in single mode");
-SingleMode.setup();
+  if (initCount.get() == 0) {
--- End diff --

non-synchronized and if (initCount.getAndIncrement() == 0) ?


> Could not run a single MongoDB unit test case through command line or IDE
> -
>
> Key: DRILL-5196
> URL: https://issues.apache.org/jira/browse/DRILL-5196
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Chunhui Shi
>Assignee: Chunhui Shi
>
> Could not run a single MongoDB's unit test through IDE or command line. The 
> reason is when running a single test case, the MongoDB instance did not get 
> started thus a 'table not found' error for 'mongo.employee.empinfo' would be 
> raised.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5196) Could not run a single MongoDB unit test case through command line or IDE

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852411#comment-15852411
 ] 

ASF GitHub Bot commented on DRILL-5196:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/741#discussion_r99451485
  
--- Diff: 
contrib/storage-mongo/src/test/java/org/apache/drill/exec/store/mongo/MongoTestSuit.java
 ---
@@ -234,15 +243,25 @@ private static void createDbAndCollections(String 
dbName,
 
   @AfterClass
   public static void tearDownCluster() throws Exception {
-if (mongoClient != null) {
-  mongoClient.dropDatabase(EMPLOYEE_DB);
-  mongoClient.close();
-}
 synchronized (MongoTestSuit.class) {
-  if (distMode) {
-DistributedMode.cleanup();
-  } else {
-SingleMode.cleanup();
+  if (initCount.decrementAndGet() == 0) {
+try {
+  if (mongoClient != null) {
--- End diff --

Move to top of function?
```
if (mongoClient==null) { return; }
if (initCount.decrementAndGet() == 0) { ...
```


> Could not run a single MongoDB unit test case through command line or IDE
> -
>
> Key: DRILL-5196
> URL: https://issues.apache.org/jira/browse/DRILL-5196
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Chunhui Shi
>Assignee: Chunhui Shi
>
> Could not run a single MongoDB's unit test through IDE or command line. The 
> reason is when running a single test case, the MongoDB instance did not get 
> started thus a 'table not found' error for 'mongo.employee.empinfo' would be 
> raised.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5196) Could not run a single MongoDB unit test case through command line or IDE

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852409#comment-15852409
 ] 

ASF GitHub Bot commented on DRILL-5196:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/741#discussion_r99451419
  
--- Diff: 
contrib/storage-mongo/src/test/java/org/apache/drill/exec/store/mongo/MongoTestSuit.java
 ---
@@ -234,15 +243,25 @@ private static void createDbAndCollections(String 
dbName,
 
   @AfterClass
   public static void tearDownCluster() throws Exception {
-if (mongoClient != null) {
-  mongoClient.dropDatabase(EMPLOYEE_DB);
-  mongoClient.close();
-}
 synchronized (MongoTestSuit.class) {
-  if (distMode) {
-DistributedMode.cleanup();
-  } else {
-SingleMode.cleanup();
+  if (initCount.decrementAndGet() == 0) {
--- End diff --

Because using the atomic int, also need the synchronized?


> Could not run a single MongoDB unit test case through command line or IDE
> -
>
> Key: DRILL-5196
> URL: https://issues.apache.org/jira/browse/DRILL-5196
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Chunhui Shi
>Assignee: Chunhui Shi
>
> Could not run a single MongoDB's unit test through IDE or command line. The 
> reason is when running a single test case, the MongoDB instance did not get 
> started thus a 'table not found' error for 'mongo.employee.empinfo' would be 
> raised.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5223) Drill should ensure balanced workload assignment at node level in order to get better query performance

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852407#comment-15852407
 ] 

ASF GitHub Bot commented on DRILL-5223:
---

Github user jinfengni commented on the issue:

https://github.com/apache/drill/pull/730
  
+1


LGTM.



> Drill should ensure balanced workload assignment at node level in order to 
> get better query performance
> ---
>
> Key: DRILL-5223
> URL: https://issues.apache.org/jira/browse/DRILL-5223
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization
>Reporter: Jinfeng Ni
>Assignee: Padma Penumarthy
>
> Drill's work assignment logic currently aims to achieve workload balance 
> across different minor fragment (or slices) and honor data affinity in order 
> to get as much local read as possible. 
> However, when the # of work units could not be evenly divided by # of minor 
> fragments, the remaining work units would tender to go to the first subset of 
> drill endpoints. This means the drill endpoints assigned with the remaining 
> work units could have larger workload than the rest of them.  When 
> MuxExchange is enabled (by default), all the minor fragments on the same node 
> have to send data to a single Muxer per node, and unbalanced workload 
> assignment at node level could impact query elapse time.  which is 
> essentially decided by the slowest drill endpoint.
> Some prototype experimental run shows that with more balanced workload 
> assignment, Drill shows quite significant improvement for most of TPC-H 
> queries. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5088) Error when reading DBRef column

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852380#comment-15852380
 ] 

ASF GitHub Bot commented on DRILL-5088:
---

Github user chunhui-shi commented on a diff in the pull request:

https://github.com/apache/drill/pull/702#discussion_r99448154
  
--- Diff: 
contrib/storage-mongo/src/test/java/org/apache/drill/exec/store/mongo/TestTableGenerator.java
 ---
@@ -58,7 +59,16 @@ public static void generateTable(String dbName, String 
collection,
 .jsonArray(jsonArray).importFile(jsonFile).build();
 MongoImportExecutable importExecutable = MongoImportStarter
 .getDefaultInstance().prepare(mongoImportConfig);
-importExecutable.start();
+MongoImportProcess importProcess = importExecutable.start();
+
+try {
+  while (importProcess.isProcessRunning()) {
+Thread.sleep(1000);
+  }
+}catch (Exception ex) {
+  logger.error("Import mongoDb failed", ex);
--- End diff --

Paul, since there are two JIRAs here, I filed a new pull request  
https://github.com/apache/drill/pull/741 which addressed your comments. Could 
you look at that pull request? Thanks.


> Error when reading DBRef column
> ---
>
> Key: DRILL-5088
> URL: https://issues.apache.org/jira/browse/DRILL-5088
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Data Types
> Environment: drill 1.9.0
> mongo 3.2
>Reporter: Guillaume Champion
>Assignee: Chunhui Shi
>
> In a mongo database with DBRef, when a DBRef is inserted in the first line of 
> a mongo's collection drill query failed :
> {code}
> 0: jdbc:drill:zk=local> select * from mongo.mydb.contact2;
> Error: SYSTEM ERROR: CodecConfigurationException: Can't find a codec for 
> class com.mongodb.DBRef.
> {code}
> Simple example to reproduce:
> In mongo instance
> {code}
> db.contact2.drop();
> db.contact2.insert({ "_id" : ObjectId("582081d96b69060001fd8938"), "account" 
> : DBRef("contact", ObjectId("999cbf116b69060001fd8611")) });
> {code}
> In drill :
> {code}
> 0: jdbc:drill:zk=local> select * from mongo.mydb.contact2;
> Error: SYSTEM ERROR: CodecConfigurationException: Can't find a codec for 
> class com.mongodb.DBRef.
> [Error Id: 2944d766-e483-4453-a706-3d481397b186 on Analytics-Biznet:31010] 
> (state=,code=0)
> {code}
> If the first line doesn't contain de DBRef, drill will querying correctly :
> In a mongo instance :
> {code}
> db.contact2.drop();
> db.contact2.insert({ "_id" : ObjectId("582081d96b69060001fd8939") });
> db.contact2.insert({ "_id" : ObjectId("582081d96b69060001fd8938"), "account" 
> : DBRef("contact", ObjectId("999cbf116b69060001fd8611")) });
> {code}
> In drill :
> {code}
> 0: jdbc:drill:zk=local> select * from mongo.mydb.contact2;
> +--+---+
> | _id  |account   
>  |
> +--+---+
> | {"$oid":"582081d96b69060001fd8939"}  | {"$id":{}}   
>  |
> | {"$oid":"582081d96b69060001fd8938"}  | 
> {"$ref":"contact","$id":{"$oid":"999cbf116b69060001fd8611"}}  |
> +--+---+
> 2 rows selected (0,563 seconds)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5196) Could not run a single MongoDB unit test case through command line or IDE

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852376#comment-15852376
 ] 

ASF GitHub Bot commented on DRILL-5196:
---

GitHub user chunhui-shi opened a pull request:

https://github.com/apache/drill/pull/741

DRILL-5196: init MongoDB cluster when run a single test case directly…

… through command line or IDE.

Other fixes include:
Sync mongo-java-driver versions to newer 3.2.0
update flapdoodle package to latest accordingly

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chunhui-shi/drill DRILL-5196

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/741.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #741


commit 6068d7a21a0cf45ba9f5f3c76ad48dbd94c6e204
Author: chunhui-shi 
Date:   2017-01-14T01:20:46Z

DRILL-5196: init MongoDB cluster when run a single test case directly 
through command line or IDE.
Other fixes include:
Sync mongo-java-driver versions to newer 3.2.0
update flapdoodle package to latest accordingly




> Could not run a single MongoDB unit test case through command line or IDE
> -
>
> Key: DRILL-5196
> URL: https://issues.apache.org/jira/browse/DRILL-5196
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Chunhui Shi
>Assignee: Chunhui Shi
>
> Could not run a single MongoDB's unit test through IDE or command line. The 
> reason is when running a single test case, the MongoDB instance did not get 
> started thus a 'table not found' error for 'mongo.employee.empinfo' would be 
> raised.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-4963) Issues when overloading Drill native functions with dynamic UDFs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852368#comment-15852368
 ] 

ASF GitHub Bot commented on DRILL-4963:
---

Github user paul-rogers commented on the issue:

https://github.com/apache/drill/pull/701
  
As I understand it, the issues are these:

* When parsing/planning a query, function references are ambiguous.
* If a function x() simply does not exist at all, we get a clear signal and 
can check the function registry for updates.
* When names are overridden, there may be many x(arg1) functions: some 
defined locally, some recently added to the registry.
* In this case, we get no clear signal that we should check the registry 
since we might find a good-enough match locally and not know to check for a 
better match in the registry.

The solution is to check the registry version on each new query. This must 
be done for every query (with functions) since we can never be certain whether 
an override exists in the registry.

The problem is that a check of ZK, even to get a version, adds latency. 
99.9% of the time, nothing will have changed. How can we fix that?

Let's take a step back. The original problems we tried to solve were:

If a user executes a CREATE FUNCTION x on Drillbit A and the command 
returns successfully, then:

* If the same user immediately executes a query using x on the same 
Drillbit, that query succeeds.
* If some other user executes a query using x (on any Drillbit, say 
Drillbit B), then the query either fails (x is not found) or succeeds (x is 
found on Drillbit B and on all Drillbits that run fragments.)

In general, the above requires lots of synchronization: we'd want every 
Drillbit to synchronize with the registry before every query parse and fragment 
execution. We know that is expensive. So, we looked for special case 
opportunities to exploit. The "found/not found" semantics above appeared to 
provide that special case. What that trick off the table, we are back to the 
massive synchronization solution, which is costly.

We want to keep the semantics listed above, but without the cost of 
synchronization.
Just tossing out ideas (I'm not (yet) proposing changes), perhaps:

* Each Drillbit maintains a cache of the most recent ZK registry version it 
has seen.
* When a Foreman registers a function, it updates the ZK registry and its 
locally-cached version number.
* When a Foreman runs a query, it includes the registry version number in 
the physical plan sent to each Drillbit for execution.
* The Drillbit checks the plan's version number against its cached version. 
If the plan version is newer, the Drillbit downloads the latest registry data 
and updates its cached version number.

The above ensures that, if a function is registered on Drillbit A, then any 
query that is submitted to A will execute anywhere on the cluster. So far so 
good.

But, what about queries submitted to Drillbits B, C and D? How do they 
learn about the new functions? Here, perhaps we can use an eventually 
consistent pattern. The other Drillbits listen for ZK change notifications and 
refresh their local registries from ZK, and update the locally cached version 
number.

Now, we get the semantics that if a function is defined on Drillbit A, a 
brief time later it will be available on Drillbits B, C and D. Once available, 
the above rules apply: the registry version is written into the plan and all 
other fragment executors will force update their local registries if they 
haven't yet gotten the ZK update notices.

The only drawback is a slight delay in a function becoming available to the 
cluster. But, that delay is fine in an eventually consistent design. Said 
another way, we make no transaction guarantees that updating something on 
Drillbit A will be immediately reflected on B, etc.

This is a rough draft. It may be what the code does. I'll continue to 
review and revise the note as I learn more.


> Issues when overloading Drill native functions with dynamic UDFs
> 
>
> Key: DRILL-4963
> URL: https://issues.apache.org/jira/browse/DRILL-4963
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Drill
>Affects Versions: 1.9.0
>Reporter: Roman
>Assignee: Arina Ielchiieva
> Fix For: Future
>
> Attachments: subquery_udf-1.0.jar, subquery_udf-1.0-sources.jar, 
> test_overloading-1.0.jar, test_overloading-1.0-sources.jar
>
>
> I created jar file which overloads 3 DRILL native functions 
> (LOG(VARCHAR-REQUIRED), CURRENT_DATE(VARCHAR-REQUIRED) and 
> ABS(VARCHAR-REQUIRED,VARCHAR-REQUIRED)) and registered it as dynamic UDF.
> If I try to use 

[jira] [Commented] (DRILL-5196) Could not run a single MongoDB unit test case through command line or IDE

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852369#comment-15852369
 ] 

ASF GitHub Bot commented on DRILL-5196:
---

Github user chunhui-shi closed the pull request at:

https://github.com/apache/drill/pull/722


> Could not run a single MongoDB unit test case through command line or IDE
> -
>
> Key: DRILL-5196
> URL: https://issues.apache.org/jira/browse/DRILL-5196
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Chunhui Shi
>Assignee: Chunhui Shi
>
> Could not run a single MongoDB's unit test through IDE or command line. The 
> reason is when running a single test case, the MongoDB instance did not get 
> started thus a 'table not found' error for 'mongo.employee.empinfo' would be 
> raised.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-4963) Issues when overloading Drill native functions with dynamic UDFs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852335#comment-15852335
 ] 

ASF GitHub Bot commented on DRILL-4963:
---

Github user paul-rogers commented on the issue:

https://github.com/apache/drill/pull/701
  
Thank you for the very clear explanation. It is now obvious that the 
original lazy init design relied on the "found/not found" semantics of a simple 
name match: no ambiguity there. However, there is ambiguity with the overload 
rules. Clearly, at planning stage, we need to check that our entire list of 
candidate functions is consistent with ZK. Your design for that seems quite 
good.

I wonder, at execution time, do we save a fully qualified function name so 
we can go back to the "found/not found" trick? That is, do we save the function 
as "foo( INT, VARCHAR )" or do we repeat the type inference at runtime?

I will do a detailed review of the code next.


> Issues when overloading Drill native functions with dynamic UDFs
> 
>
> Key: DRILL-4963
> URL: https://issues.apache.org/jira/browse/DRILL-4963
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Drill
>Affects Versions: 1.9.0
>Reporter: Roman
>Assignee: Arina Ielchiieva
> Fix For: Future
>
> Attachments: subquery_udf-1.0.jar, subquery_udf-1.0-sources.jar, 
> test_overloading-1.0.jar, test_overloading-1.0-sources.jar
>
>
> I created jar file which overloads 3 DRILL native functions 
> (LOG(VARCHAR-REQUIRED), CURRENT_DATE(VARCHAR-REQUIRED) and 
> ABS(VARCHAR-REQUIRED,VARCHAR-REQUIRED)) and registered it as dynamic UDF.
> If I try to use my functions I will get errors:
> {code:xml}
> SELECT CURRENT_DATE('test') FROM (VALUES(1));
> {code}
> Error: FUNCTION ERROR: CURRENT_DATE does not support operand types (CHAR)
> SQL Query null
> {code:xml}
> SELECT ABS('test','test') FROM (VALUES(1));
> {code}
> Error: FUNCTION ERROR: ABS does not support operand types (CHAR,CHAR)
> SQL Query null
> {code:xml}
> SELECT LOG('test') FROM (VALUES(1));
> {code}
> Error: SYSTEM ERROR: DrillRuntimeException: Failure while materializing 
> expression in constant expression evaluator LOG('test').  Errors: 
> Error in expression at index -1.  Error: Missing function implementation: 
> castTINYINT(VARCHAR-REQUIRED).  Full expression: UNKNOWN EXPRESSION.
> But if I rerun all this queries after "DrillRuntimeException", they will run 
> correctly. It seems that Drill have not updated the function signature before 
> that error. Also if I add jar as usual UDF (copy jar to 
> /drill_home/jars/3rdparty and restart drillbits), all queries will run 
> correctly without errors.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5080) Create a memory-managed version of the External Sort operator

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852283#comment-15852283
 ] 

ASF GitHub Bot commented on DRILL-5080:
---

Github user paul-rogers commented on the issue:

https://github.com/apache/drill/pull/717
  
Build failure related to the slf4j library in the build process itself.


> Create a memory-managed version of the External Sort operator
> -
>
> Key: DRILL-5080
> URL: https://issues.apache.org/jira/browse/DRILL-5080
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
> Fix For: 1.10.0
>
> Attachments: ManagedExternalSortDesign.pdf
>
>
> We propose to create a "managed" version of the external sort operator that 
> works to a clearly-defined memory limit. Attached is a design specification 
> for the work.
> The project will include fixing a number of bugs related to the external 
> sort, include as sub-tasks of this umbrella task.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (DRILL-5224) CTTAS: fix errors connected with system path delimiters (Windows)

[Paul Rogers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paul Rogers updated DRILL-5224:
---
Labels: ready-to-commit  (was: )

> CTTAS: fix errors connected with system path delimiters (Windows)
> -
>
> Key: DRILL-5224
> URL: https://issues.apache.org/jira/browse/DRILL-5224
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.10.0
> Environment: Windows 10
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>  Labels: ready-to-commit
> Fix For: 1.10.0
>
>
> Problem 1:
> Error occurs when attempting to create temporary table on Windows:
> {noformat}
> 0: jdbc:drill:zk=local> create temporary table t as select * from sys.version;
> Error: SYSTEM ERROR: InvalidPathException: Illegal char <:> at index 4: 
> file:///\tmp\3191db8e-279d-4ced-b0e5-32b3b477edfb
> {noformat}
> Root cause:
> when creating temporary directory we merge file system uri, temporary 
> workspace location and session id into one path using 
> java.nio.file.Paths.get() but this method cannot resolve path when path has 
> different delimiters.
> Fix:
> Use org.apache.hadoop.fs.Path tools to merge path, path string is normalized 
> during creation.
> {noformat}
> new Path(fs.getUri().toString(), new 
> Path(temporaryWorkspace.getDefaultLocation(), sessionId));
> {noformat}
> Problem 2:
> When temporary table is being manually dropped using drop table command, 
> though actual table is dropped, remnant folder is left.
> Root cause:
> Before adding to temporary table to the list of temporary tables, its 
> generated name is concatenated with session id (as parent and child folders). 
> java.nio.file.Paths.get() is used for concatenation but it preserves current 
> system delimiter. When table is being dropped, passed table name is split 
> using org.apache.hadoop.fs.Path.SEPARATOR, since it's assumed that path was 
> created using org.apache.hadoop.fs.Path tools where path separators are 
> normalized to one format disregarding the system.
> Fix:
> Concatenate session id with generated table name using 
> org.apache.hadoop.fs.Path tools.
> {noformat}
> new Path(sessionId, UUID.randomUUID().toString()).toUri().getPath();
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5224) CTTAS: fix errors connected with system path delimiters (Windows)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852273#comment-15852273
 ] 

ASF GitHub Bot commented on DRILL-5224:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/731#discussion_r99439632
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/rpc/user/UserSession.java ---
@@ -278,7 +277,7 @@ public boolean setSessionOption(String name, String 
value) {
*/
   public String registerTemporaryTable(AbstractSchema schema, String 
tableName) throws IOException {
   addTemporaryLocation((WorkspaceSchemaFactory.WorkspaceSchema) 
schema);
-  String temporaryTableName = Paths.get(sessionId, 
UUID.randomUUID().toString()).toString();
+  String temporaryTableName = new Path(sessionId, 
UUID.randomUUID().toString()).toUri().getPath();
--- End diff --

Sounds good.


> CTTAS: fix errors connected with system path delimiters (Windows)
> -
>
> Key: DRILL-5224
> URL: https://issues.apache.org/jira/browse/DRILL-5224
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.10.0
> Environment: Windows 10
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>  Labels: ready-to-commit
> Fix For: 1.10.0
>
>
> Problem 1:
> Error occurs when attempting to create temporary table on Windows:
> {noformat}
> 0: jdbc:drill:zk=local> create temporary table t as select * from sys.version;
> Error: SYSTEM ERROR: InvalidPathException: Illegal char <:> at index 4: 
> file:///\tmp\3191db8e-279d-4ced-b0e5-32b3b477edfb
> {noformat}
> Root cause:
> when creating temporary directory we merge file system uri, temporary 
> workspace location and session id into one path using 
> java.nio.file.Paths.get() but this method cannot resolve path when path has 
> different delimiters.
> Fix:
> Use org.apache.hadoop.fs.Path tools to merge path, path string is normalized 
> during creation.
> {noformat}
> new Path(fs.getUri().toString(), new 
> Path(temporaryWorkspace.getDefaultLocation(), sessionId));
> {noformat}
> Problem 2:
> When temporary table is being manually dropped using drop table command, 
> though actual table is dropped, remnant folder is left.
> Root cause:
> Before adding to temporary table to the list of temporary tables, its 
> generated name is concatenated with session id (as parent and child folders). 
> java.nio.file.Paths.get() is used for concatenation but it preserves current 
> system delimiter. When table is being dropped, passed table name is split 
> using org.apache.hadoop.fs.Path.SEPARATOR, since it's assumed that path was 
> created using org.apache.hadoop.fs.Path tools where path separators are 
> normalized to one format disregarding the system.
> Fix:
> Concatenate session id with generated table name using 
> org.apache.hadoop.fs.Path tools.
> {noformat}
> new Path(sessionId, UUID.randomUUID().toString()).toUri().getPath();
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5190) Display planning time for a query in its profile page

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852268#comment-15852268
 ] 

ASF GitHub Bot commented on DRILL-5190:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/738#discussion_r99439263
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/profile/ProfileResources.java
 ---
@@ -77,12 +77,14 @@
*/
   public static String getPrettyDuration(long startTimeMillis, long 
endTimeMillis) {
 long durationInMillis = (startTimeMillis > endTimeMillis ? 
System.currentTimeMillis() : endTimeMillis) - startTimeMillis;
-long hours = TimeUnit.MILLISECONDS.toHours(durationInMillis);
+long days = TimeUnit.MILLISECONDS.toDays(durationInMillis);
--- End diff --

Good point. Maybe have a single utility class with two methods: one for the 
compact form, one for the verbose form. Callers can choose the format best for 
a particular presentation.

Note that, in both cases, we need to tear apart the duration into various 
units. This suggests not a static class, but an actual "real" class:
```
DurationFormat df = new DurationFormat( durationMs );
System.out.println( df.compact() );
System.out.println( df.verbose() );
```


> Display planning time for a query in its profile page
> -
>
> Key: DRILL-5190
> URL: https://issues.apache.org/jira/browse/DRILL-5190
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Web Server
>Affects Versions: 1.9.0
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>
> Currently, the Web UI does not display the time spent in planning for a query 
> in its profile page. The estimate needs to be done by seeing how late did the 
> earliest major fragment start. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (DRILL-5241) JDBC proxy driver: Do not put null value in map

[Paul Rogers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paul Rogers updated DRILL-5241:
---
  Labels: ready-to-commit  (was: )
Reviewer: Paul Rogers

> JDBC proxy driver: Do not put null value in map
> ---
>
> Key: DRILL-5241
> URL: https://issues.apache.org/jira/browse/DRILL-5241
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Client - JDBC
>Affects Versions: 1.9.0
>Reporter: David Haller
>Priority: Minor
>  Labels: ready-to-commit
>   Original Estimate: 0h
>  Remaining Estimate: 0h
>
> See GitHub pull request: https://github.com/apache/drill/pull/724
> Hello everyone,
> proxyReturnClass is always null, so interfacesToProxyClassesMap will contain 
> null values only. Adding newProxyReturnClass should be correct.
> This bug does not affect functionality, but probably decreases performance 
> because you get "cache misses" all the time.
> Best regards,
> David.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5190) Display planning time for a query in its profile page

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852012#comment-15852012
 ] 

ASF GitHub Bot commented on DRILL-5190:
---

Github user kkhatua commented on a diff in the pull request:

https://github.com/apache/drill/pull/738#discussion_r99408793
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/profile/ProfileResources.java
 ---
@@ -77,12 +77,14 @@
*/
   public static String getPrettyDuration(long startTimeMillis, long 
endTimeMillis) {
 long durationInMillis = (startTimeMillis > endTimeMillis ? 
System.currentTimeMillis() : endTimeMillis) - startTimeMillis;
-long hours = TimeUnit.MILLISECONDS.toHours(durationInMillis);
+long days = TimeUnit.MILLISECONDS.toDays(durationInMillis);
--- End diff --

@paul-rogers , The format in DRILL-5230 is compacted to an approximation 
(i.e. we retain the 2 largest units of time) so as to fit within the cells of a 
table. 
This format is more verbose as it gives the complete picture, down to the 
smallest unit of time. Do we want to still merge the two into a single utility 
method?



> Display planning time for a query in its profile page
> -
>
> Key: DRILL-5190
> URL: https://issues.apache.org/jira/browse/DRILL-5190
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Web Server
>Affects Versions: 1.9.0
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>
> Currently, the Web UI does not display the time spent in planning for a query 
> in its profile page. The estimate needs to be done by seeing how late did the 
> earliest major fragment start. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (DRILL-5240) Parquet reader creates unnecessary objects when checking for nullability in var length columns

[Parth Chandra (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Parth Chandra updated DRILL-5240:
-
Labels: ready-to-commit  (was: )

> Parquet reader creates unnecessary objects when checking for nullability in 
> var length columns
> --
>
> Key: DRILL-5240
> URL: https://issues.apache.org/jira/browse/DRILL-5240
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Parquet
>Reporter: Parth Chandra
>Assignee: Parth Chandra
>  Labels: ready-to-commit
>
> In {{NullableVarLengthValuesColumn.java}} we have the following line of code:
> {code}
> currentValNull = 
> variableWidthVector.getAccessor().getObject(valuesReadInCurrentPass) == null;
> {code}
> which creates an object to check nullability of every (nullable var char) 
> value being read. This object creation is expensive and can be replaced with 
> a simple check for the null bit.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5241) JDBC proxy driver: Do not put null value in map

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15851968#comment-15851968
 ] 

ASF GitHub Bot commented on DRILL-5241:
---

Github user davidhaller commented on the issue:

https://github.com/apache/drill/pull/724
  
JIRA issue created at https://issues.apache.org/jira/browse/DRILL-5241


> JDBC proxy driver: Do not put null value in map
> ---
>
> Key: DRILL-5241
> URL: https://issues.apache.org/jira/browse/DRILL-5241
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Client - JDBC
>Affects Versions: 1.9.0
>Reporter: David Haller
>Priority: Minor
>   Original Estimate: 0h
>  Remaining Estimate: 0h
>
> See GitHub pull request: https://github.com/apache/drill/pull/724
> Hello everyone,
> proxyReturnClass is always null, so interfacesToProxyClassesMap will contain 
> null values only. Adding newProxyReturnClass should be correct.
> This bug does not affect functionality, but probably decreases performance 
> because you get "cache misses" all the time.
> Best regards,
> David.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (DRILL-5241) JDBC proxy driver: Do not put null value in map

[David Haller (JIRA)]

David Haller created DRILL-5241:
---

 Summary: JDBC proxy driver: Do not put null value in map
 Key: DRILL-5241
 URL: https://issues.apache.org/jira/browse/DRILL-5241
 Project: Apache Drill
  Issue Type: Bug
  Components: Client - JDBC
Affects Versions: 1.9.0
Reporter: David Haller
Priority: Minor


See GitHub pull request: https://github.com/apache/drill/pull/724

Hello everyone,

proxyReturnClass is always null, so interfacesToProxyClassesMap will contain 
null values only. Adding newProxyReturnClass should be correct.

This bug does not affect functionality, but probably decreases performance 
because you get "cache misses" all the time.

Best regards,
David.





--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5240) Parquet reader creates unnecessary objects when checking for nullability in var length columns

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15851961#comment-15851961
 ] 

ASF GitHub Bot commented on DRILL-5240:
---

Github user jinfengni commented on the issue:

https://github.com/apache/drill/pull/740
  
+1

LGTM


> Parquet reader creates unnecessary objects when checking for nullability in 
> var length columns
> --
>
> Key: DRILL-5240
> URL: https://issues.apache.org/jira/browse/DRILL-5240
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Parquet
>Reporter: Parth Chandra
>Assignee: Parth Chandra
>
> In {{NullableVarLengthValuesColumn.java}} we have the following line of code:
> {code}
> currentValNull = 
> variableWidthVector.getAccessor().getObject(valuesReadInCurrentPass) == null;
> {code}
> which creates an object to check nullability of every (nullable var char) 
> value being read. This object creation is expensive and can be replaced with 
> a simple check for the null bit.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5240) Parquet reader creates unnecessary objects when checking for nullability in var length columns

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15851946#comment-15851946
 ] 

ASF GitHub Bot commented on DRILL-5240:
---

GitHub user parthchandra opened a pull request:

https://github.com/apache/drill/pull/740

DRILL-5240: Parquet - fix unnecessary object creation while checking …

…for null values in nullable var length columns

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/parthchandra/drill DRILL-5240

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/740.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #740


commit d6c65ac7e8b5ff968e23e93ee695467165d0e3bf
Author: Parth Chandra 
Date:   2017-02-02T04:14:52Z

DRILL-5240: Parquet - fix unnecessary object creation while checking for 
null values in nullable var length columns




> Parquet reader creates unnecessary objects when checking for nullability in 
> var length columns
> --
>
> Key: DRILL-5240
> URL: https://issues.apache.org/jira/browse/DRILL-5240
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Parquet
>Reporter: Parth Chandra
>Assignee: Parth Chandra
>
> In {{NullableVarLengthValuesColumn.java}} we have the following line of code:
> {code}
> currentValNull = 
> variableWidthVector.getAccessor().getObject(valuesReadInCurrentPass) == null;
> {code}
> which creates an object to check nullability of every (nullable var char) 
> value being read. This object creation is expensive and can be replaced with 
> a simple check for the null bit.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5088) Error when reading DBRef column

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15851910#comment-15851910
 ] 

ASF GitHub Bot commented on DRILL-5088:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/702#discussion_r99396951
  
--- Diff: 
contrib/storage-mongo/src/test/java/org/apache/drill/exec/store/mongo/TestTableGenerator.java
 ---
@@ -58,7 +59,16 @@ public static void generateTable(String dbName, String 
collection,
 .jsonArray(jsonArray).importFile(jsonFile).build();
 MongoImportExecutable importExecutable = MongoImportStarter
 .getDefaultInstance().prepare(mongoImportConfig);
-importExecutable.start();
+MongoImportProcess importProcess = importExecutable.start();
+
+try {
+  while (importProcess.isProcessRunning()) {
+Thread.sleep(1000);
+  }
+}catch (Exception ex) {
+  logger.error("Import mongoDb failed", ex);
--- End diff --

Here we log the error but go ahead and return. Should we propagate an 
exception upward in the call stack so that the caller knows that "Something Is 
Wrong"? Otherwise, how will the caller know whether the import process is ready 
or not?


> Error when reading DBRef column
> ---
>
> Key: DRILL-5088
> URL: https://issues.apache.org/jira/browse/DRILL-5088
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Data Types
> Environment: drill 1.9.0
> mongo 3.2
>Reporter: Guillaume Champion
>Assignee: Chunhui Shi
>
> In a mongo database with DBRef, when a DBRef is inserted in the first line of 
> a mongo's collection drill query failed :
> {code}
> 0: jdbc:drill:zk=local> select * from mongo.mydb.contact2;
> Error: SYSTEM ERROR: CodecConfigurationException: Can't find a codec for 
> class com.mongodb.DBRef.
> {code}
> Simple example to reproduce:
> In mongo instance
> {code}
> db.contact2.drop();
> db.contact2.insert({ "_id" : ObjectId("582081d96b69060001fd8938"), "account" 
> : DBRef("contact", ObjectId("999cbf116b69060001fd8611")) });
> {code}
> In drill :
> {code}
> 0: jdbc:drill:zk=local> select * from mongo.mydb.contact2;
> Error: SYSTEM ERROR: CodecConfigurationException: Can't find a codec for 
> class com.mongodb.DBRef.
> [Error Id: 2944d766-e483-4453-a706-3d481397b186 on Analytics-Biznet:31010] 
> (state=,code=0)
> {code}
> If the first line doesn't contain de DBRef, drill will querying correctly :
> In a mongo instance :
> {code}
> db.contact2.drop();
> db.contact2.insert({ "_id" : ObjectId("582081d96b69060001fd8939") });
> db.contact2.insert({ "_id" : ObjectId("582081d96b69060001fd8938"), "account" 
> : DBRef("contact", ObjectId("999cbf116b69060001fd8611")) });
> {code}
> In drill :
> {code}
> 0: jdbc:drill:zk=local> select * from mongo.mydb.contact2;
> +--+---+
> | _id  |account   
>  |
> +--+---+
> | {"$oid":"582081d96b69060001fd8939"}  | {"$id":{}}   
>  |
> | {"$oid":"582081d96b69060001fd8938"}  | 
> {"$ref":"contact","$id":{"$oid":"999cbf116b69060001fd8611"}}  |
> +--+---+
> 2 rows selected (0,563 seconds)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5088) Error when reading DBRef column

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15851911#comment-15851911
 ] 

ASF GitHub Bot commented on DRILL-5088:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/702#discussion_r99396637
  
--- Diff: 
contrib/storage-mongo/src/test/java/org/apache/drill/exec/store/mongo/MongoTestSuit.java
 ---
@@ -234,15 +240,17 @@ private static void createDbAndCollections(String 
dbName,
 
   @AfterClass
   public static void tearDownCluster() throws Exception {
-if (mongoClient != null) {
-  mongoClient.dropDatabase(EMPLOYEE_DB);
-  mongoClient.close();
-}
 synchronized (MongoTestSuit.class) {
-  if (distMode) {
-DistributedMode.cleanup();
-  } else {
-SingleMode.cleanup();
+  if (initCount.decrementAndGet() == 0) {
+if (mongoClient != null) {
+  mongoClient.dropDatabase(EMPLOYEE_DB);
+  mongoClient.close();
+}
+if (distMode) {
+  DistributedMode.cleanup();
+} else {
+  SingleMode.cleanup();
--- End diff --

Should the above include error handling? It seems that if the 
`dropDatabase` fails, then `close()` won't be called, nor will the `cleanup` 
method.


> Error when reading DBRef column
> ---
>
> Key: DRILL-5088
> URL: https://issues.apache.org/jira/browse/DRILL-5088
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Data Types
> Environment: drill 1.9.0
> mongo 3.2
>Reporter: Guillaume Champion
>Assignee: Chunhui Shi
>
> In a mongo database with DBRef, when a DBRef is inserted in the first line of 
> a mongo's collection drill query failed :
> {code}
> 0: jdbc:drill:zk=local> select * from mongo.mydb.contact2;
> Error: SYSTEM ERROR: CodecConfigurationException: Can't find a codec for 
> class com.mongodb.DBRef.
> {code}
> Simple example to reproduce:
> In mongo instance
> {code}
> db.contact2.drop();
> db.contact2.insert({ "_id" : ObjectId("582081d96b69060001fd8938"), "account" 
> : DBRef("contact", ObjectId("999cbf116b69060001fd8611")) });
> {code}
> In drill :
> {code}
> 0: jdbc:drill:zk=local> select * from mongo.mydb.contact2;
> Error: SYSTEM ERROR: CodecConfigurationException: Can't find a codec for 
> class com.mongodb.DBRef.
> [Error Id: 2944d766-e483-4453-a706-3d481397b186 on Analytics-Biznet:31010] 
> (state=,code=0)
> {code}
> If the first line doesn't contain de DBRef, drill will querying correctly :
> In a mongo instance :
> {code}
> db.contact2.drop();
> db.contact2.insert({ "_id" : ObjectId("582081d96b69060001fd8939") });
> db.contact2.insert({ "_id" : ObjectId("582081d96b69060001fd8938"), "account" 
> : DBRef("contact", ObjectId("999cbf116b69060001fd8611")) });
> {code}
> In drill :
> {code}
> 0: jdbc:drill:zk=local> select * from mongo.mydb.contact2;
> +--+---+
> | _id  |account   
>  |
> +--+---+
> | {"$oid":"582081d96b69060001fd8939"}  | {"$id":{}}   
>  |
> | {"$oid":"582081d96b69060001fd8938"}  | 
> {"$ref":"contact","$id":{"$oid":"999cbf116b69060001fd8611"}}  |
> +--+---+
> 2 rows selected (0,563 seconds)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (DRILL-5002) Using hive's date functions on top of date column in parquet gives wrong results

[Rahul Challapalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rahul Challapalli updated DRILL-5002:
-
Attachment: 0_0_0.parquet

Check the exact same query that I ran from the above comment. The below query 
is also one good example

*DRILL* : Wrong Result
{code}
select l_shipdate, last_day(l_shipdate) from hive.lineitem where l_shipdate = 
date '1994-02-01' limit 2;
+-+-+
| l_shipdate  |   EXPR$1|
+-+-+
| 1994-02-01  | 1994-01-31  |
| 1994-02-01  | 1994-01-31  |
+-+-+
2 rows selected (0.402 seconds)
{code}

*HIVE* : Correct Result
{code}
hive> select l_shipdate, `last_day`(l_shipdate) from lineitem where l_shipdate 
= date '1994-02-01' limit 2;
OK
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further 
details.
1994-02-01  1994-02-28
1994-02-01  1994-02-28
Time taken: 1.608 seconds, Fetched: 2 row(s)
{code}

> Using hive's date functions on top of date column in parquet gives wrong 
> results
> 
>
> Key: DRILL-5002
> URL: https://issues.apache.org/jira/browse/DRILL-5002
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Hive, Storage - Parquet
>Reporter: Rahul Challapalli
>Assignee: Vitalii Diravka
>Priority: Critical
> Attachments: 0_0_0.parquet
>
>
> git.commit.id.abbrev=190d5d4
> Wrong Result 1 :
> {code}
> select l_shipdate, `month`(l_shipdate) from cp.`tpch/lineitem.parquet` where 
> l_shipdate = date '1994-02-01' limit 2;
> +-+-+
> | l_shipdate  | EXPR$1  |
> +-+-+
> | 1994-02-01  | 1   |
> | 1994-02-01  | 1   |
> +-+-+
> {code}
> Wrong Result 2 : 
> {code}
> select l_shipdate, `day`(l_shipdate) from cp.`tpch/lineitem.parquet` where 
> l_shipdate = date '1998-06-02' limit 2;
> +-+-+
> | l_shipdate  | EXPR$1  |
> +-+-+
> | 1998-06-02  | 1   |
> | 1998-06-02  | 1   |
> +-+-+
> {code}
> Correct Result :
> {code}
> select l_shipdate, `month`(l_shipdate) from cp.`tpch/lineitem.parquet` where 
> l_shipdate = date '1998-06-02' limit 2;
> +-+-+
> | l_shipdate  | EXPR$1  |
> +-+-+
> | 1998-06-02  | 6   |
> | 1998-06-02  | 6   |
> +-+-+
> {code}
> It looks like we are getting wrong results when the 'day' is '01'. I only 
> tried month and day hive functionsbut wouldn't be surprised if they have 
> similar issues too.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5238) CTTAS: unable to resolve temporary table if workspace is indicated without schema

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15851875#comment-15851875
 ] 

ASF GitHub Bot commented on DRILL-5238:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/736#discussion_r99393786
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/SqlConverter.java
 ---
@@ -527,5 +529,31 @@ public RelOptTableImpl getTable(final List 
names) {
   }
   return super.getTable(names);
 }
+
+/**
+ * Check if passed table is temporary or not should be done if:
+ * schema is not indicated (only one element in the names 
list)
+ * current schema or indicated schema is default temporary 
workspace
+ *
+ * Examples (where dfs.tmp is default temporary workspace):
+ * select * from t
+ * select * from dfs.tmp.t
+ * use dfs; select * from tmp.t
+ *
+ * @param names list of schema and table names, table name 
is always the last element
+ * @param defaultSchemaPath current schema path set using USE command
+ * @param drillConfig   drill config
+ * @return true if check for temporary table should be done, false 
otherwise
+ */
+private boolean checkForTemporaryTable(List names, String 
defaultSchemaPath, DrillConfig drillConfig) {
+  if (names.size() == 1) {
+return true;
+  }
--- End diff --

Renamed method to `mightBeTemporaryTable`. You are right the intent here is 
just to filter out cases when we don't need to check is passed table might be 
temporary. When table is passed to `getTableName` we don't know if it's 
temporary or not. If it is temporary we need to substitute the name to 
generated one and there are several indicators when we need to check this: 
table name is without schema or indicated schema is default temporary 
workspace,in all other cases we just check regular tables directly.


> CTTAS: unable to resolve temporary table if workspace is indicated without 
> schema
> -
>
> Key: DRILL-5238
> URL: https://issues.apache.org/jira/browse/DRILL-5238
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.10.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>  Labels: ready-to-commit
> Fix For: 1.10.0
>
>
> Drill is unable to resolve temporary table if default temporary workspace is 
> partially indicated (schema was set using USE command and only workspace name 
> is used in query).
> {noformat}
> 0: jdbc:drill:zk=local> use dfs;
> +---+--+
> |  ok   | summary  |
> +---+--+
> | true  | Default schema changed to [dfs]  |
> +---+--+
> 1 row selected (0.156 seconds)
> 0: jdbc:drill:zk=local> create temporary table tmp.t as select 'A' from 
> (values(1));
> +---++
> | Fragment  | Number of records written  |
> +---++
> | 0_0   | 1  |
> +---++
> 1 row selected (1.525 seconds)
> 0: jdbc:drill:zk=local> select * from tmp.t;
> Feb 02, 2017 11:24:09 AM 
> org.apache.calcite.sql.validate.SqlValidatorException 
> SEVERE: org.apache.calcite.sql.validate.SqlValidatorException: Table 'tmp.t' 
> not found
> Feb 02, 2017 11:24:09 AM org.apache.calcite.runtime.CalciteException 
> SEVERE: org.apache.calcite.runtime.CalciteContextException: From line 1, 
> column 15 to line 1, column 17: Table 'tmp.t' not found
> Error: VALIDATION ERROR: From line 1, column 15 to line 1, column 17: Table 
> 'tmp.t' not found
> SQL Query null
> [Error Id: 5266cb67-9d37-4a94-9a4e-28a4a2f94be5 on localhost:31010] 
> (state=,code=0)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5230) Translation of millisecond duration into hours is incorrect

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15851874#comment-15851874
 ] 

ASF GitHub Bot commented on DRILL-5230:
---

Github user kkhatua commented on a diff in the pull request:

https://github.com/apache/drill/pull/739#discussion_r99393776
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/profile/TableBuilder.java
 ---
@@ -56,6 +47,30 @@ public TableBuilder(final String[] columns) {
 sb.append("\n");
   }
 
+  /**
+   * Representation of a millisecond duration in a short readable text
+   * @param millis  Duration in milliseconds
+   * @returnHuman-Readable Duration Time
+   */
+  public String shortDurationFormat(long millis) {
+long days = TimeUnit.MILLISECONDS.toDays(millis);
+long hours = TimeUnit.MILLISECONDS.toHours(millis) - 
TimeUnit.DAYS.toHours(TimeUnit.MILLISECONDS.toDays(millis));
+long minutes = TimeUnit.MILLISECONDS.toMinutes(millis) - 
TimeUnit.HOURS.toMinutes(TimeUnit.MILLISECONDS.toHours(millis));
+long seconds = TimeUnit.MILLISECONDS.toSeconds(millis) - 
TimeUnit.MINUTES.toSeconds(TimeUnit.MILLISECONDS.toMinutes(millis));
+long milliSeconds = millis - 
TimeUnit.SECONDS.toMillis(TimeUnit.MILLISECONDS.toSeconds(millis));
+String formattedDuration = "";
+if (days >= 1) {
+  formattedDuration = days + "d" + hours + "h" + minutes + "m";
+} else if (hours >= 1) {
+  formattedDuration = hours + "h" + minutes + "m";
+} else if (minutes >= 1) {
+  formattedDuration = minutes + "m" + seconds + "s";
+} else {
+  formattedDuration = seconds + "." + milliSeconds + "s";
--- End diff --

Good catch. I'll update the test case as well and apply the String.format()


> Translation of millisecond duration into hours is incorrect
> ---
>
> Key: DRILL-5230
> URL: https://issues.apache.org/jira/browse/DRILL-5230
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Web Server
>Affects Versions: 1.9.0
>Reporter: Kunal Khatua
>  Labels: easyfix
> Fix For: 1.10.0
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> The method 
> {code:JAVA}org.apache.drill.exec.server.rest.profile.TableBuilder.appendMillis(long,
>  String){code}
> has a bug where the human readable translation of a 1+ hr duration in 
> milliseconds is reported incorrectly. 
> This has to do with the {code:JAVA}SimpleDateFormat.format() {code} method 
> incorectly translating it. 
> For e.g.
> {code:JAVA}
> long x = 4545342L; //1 hour 15 min 45.342 sec
> public void appendMillis(x, null);
> {code}
> This formats the value as {noformat}17h15m{noformat} instead of 
> {noformat}1h15m{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5230) Translation of millisecond duration into hours is incorrect

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15851871#comment-15851871
 ] 

ASF GitHub Bot commented on DRILL-5230:
---

Github user kkhatua commented on a diff in the pull request:

https://github.com/apache/drill/pull/739#discussion_r99393441
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/profile/TableBuilder.java
 ---
@@ -56,6 +47,30 @@ public TableBuilder(final String[] columns) {
 sb.append("\n");
   }
 
+  /**
+   * Representation of a millisecond duration in a short readable text
+   * @param millis  Duration in milliseconds
+   * @returnHuman-Readable Duration Time
+   */
+  public String shortDurationFormat(long millis) {
+long days = TimeUnit.MILLISECONDS.toDays(millis);
+long hours = TimeUnit.MILLISECONDS.toHours(millis) - 
TimeUnit.DAYS.toHours(TimeUnit.MILLISECONDS.toDays(millis));
+long minutes = TimeUnit.MILLISECONDS.toMinutes(millis) - 
TimeUnit.HOURS.toMinutes(TimeUnit.MILLISECONDS.toHours(millis));
+long seconds = TimeUnit.MILLISECONDS.toSeconds(millis) - 
TimeUnit.MINUTES.toSeconds(TimeUnit.MILLISECONDS.toMinutes(millis));
+long milliSeconds = millis - 
TimeUnit.SECONDS.toMillis(TimeUnit.MILLISECONDS.toSeconds(millis));
+String formattedDuration = "";
+if (days >= 1) {
+  formattedDuration = days + "d" + hours + "h" + minutes + "m";
--- End diff --

This display format is kept compact due to limited space within the cells 
of the table. Hence, we aim to keep only the first two major units. Hence, if 
something runs in hours, we report down to the minute, but not any more 
granular.


> Translation of millisecond duration into hours is incorrect
> ---
>
> Key: DRILL-5230
> URL: https://issues.apache.org/jira/browse/DRILL-5230
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Web Server
>Affects Versions: 1.9.0
>Reporter: Kunal Khatua
>  Labels: easyfix
> Fix For: 1.10.0
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> The method 
> {code:JAVA}org.apache.drill.exec.server.rest.profile.TableBuilder.appendMillis(long,
>  String){code}
> has a bug where the human readable translation of a 1+ hr duration in 
> milliseconds is reported incorrectly. 
> This has to do with the {code:JAVA}SimpleDateFormat.format() {code} method 
> incorectly translating it. 
> For e.g.
> {code:JAVA}
> long x = 4545342L; //1 hour 15 min 45.342 sec
> public void appendMillis(x, null);
> {code}
> This formats the value as {noformat}17h15m{noformat} instead of 
> {noformat}1h15m{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Reopened] (DRILL-5002) Using hive's date functions on top of date column in parquet gives wrong results

[Rahul Challapalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rahul Challapalli reopened DRILL-5002:
--

There was some mis-understanding. We are still seeing wrong results. Check the 
below case with the hive plugin. The month should have been 2.
{code}
select l_shipdate, `month`(l_shipdate) from hive.lineitem where l_shipdate = 
date '1994-02-01' limit 2;
+-+-+
| l_shipdate  | EXPR$1  |
+-+-+
| 1994-02-01  | 1   |
| 1994-02-01  | 1   |
+-+-+
2 rows selected (0.28 seconds)
{code}

The same query from hive on the same parquet file returns the correct results
{code}
select l_shipdate, `month`(l_shipdate) from lineitem where l_shipdate = date 
'1994-02-01' limit 2;
OK
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further 
details.
1994-02-01  2
1994-02-01  2
Time taken: 0.536 seconds, Fetched: 2 row(s)
{code}

I attached the parquet file used and below is the hive ddl
{code}
create external table if not exists lineitem (
l_orderkey int,
l_partkey int,
l_suppkey int,
l_linenumber int,
l_quantity double,
l_extendedprice double,
l_discount double,
l_tax double,
l_returnflag string,
l_linestatus string,
l_shipdate date,
l_commitdate date,
l_receiptdate date,
l_shipinstruct string,
l_shipmode string,
l_comment string
)
STORED AS PARQUET
LOCATION '/drill/testdata/lineitem';
{code}

> Using hive's date functions on top of date column in parquet gives wrong 
> results
> 
>
> Key: DRILL-5002
> URL: https://issues.apache.org/jira/browse/DRILL-5002
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Hive, Storage - Parquet
>Reporter: Rahul Challapalli
>Assignee: Vitalii Diravka
>Priority: Critical
>
> git.commit.id.abbrev=190d5d4
> Wrong Result 1 :
> {code}
> select l_shipdate, `month`(l_shipdate) from cp.`tpch/lineitem.parquet` where 
> l_shipdate = date '1994-02-01' limit 2;
> +-+-+
> | l_shipdate  | EXPR$1  |
> +-+-+
> | 1994-02-01  | 1   |
> | 1994-02-01  | 1   |
> +-+-+
> {code}
> Wrong Result 2 : 
> {code}
> select l_shipdate, `day`(l_shipdate) from cp.`tpch/lineitem.parquet` where 
> l_shipdate = date '1998-06-02' limit 2;
> +-+-+
> | l_shipdate  | EXPR$1  |
> +-+-+
> | 1998-06-02  | 1   |
> | 1998-06-02  | 1   |
> +-+-+
> {code}
> Correct Result :
> {code}
> select l_shipdate, `month`(l_shipdate) from cp.`tpch/lineitem.parquet` where 
> l_shipdate = date '1998-06-02' limit 2;
> +-+-+
> | l_shipdate  | EXPR$1  |
> +-+-+
> | 1998-06-02  | 6   |
> | 1998-06-02  | 6   |
> +-+-+
> {code}
> It looks like we are getting wrong results when the 'day' is '01'. I only 
> tried month and day hive functionsbut wouldn't be surprised if they have 
> similar issues too.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5224) CTTAS: fix errors connected with system path delimiters (Windows)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15851857#comment-15851857
 ] 

ASF GitHub Bot commented on DRILL-5224:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/731#discussion_r99391122
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/rpc/user/UserSession.java ---
@@ -278,7 +277,7 @@ public boolean setSessionOption(String name, String 
value) {
*/
   public String registerTemporaryTable(AbstractSchema schema, String 
tableName) throws IOException {
   addTemporaryLocation((WorkspaceSchemaFactory.WorkspaceSchema) 
schema);
-  String temporaryTableName = Paths.get(sessionId, 
UUID.randomUUID().toString()).toString();
+  String temporaryTableName = new Path(sessionId, 
UUID.randomUUID().toString()).toUri().getPath();
--- End diff --

In this case we use path  just to concatenate two paths: session id + temp 
table generated name (to which refer as temp table full name). 
org.apache.hadoop.fs.Path uses common path delimiter which it can resolve for 
any system. The above just returns xxx/xxx without file system specifics.




> CTTAS: fix errors connected with system path delimiters (Windows)
> -
>
> Key: DRILL-5224
> URL: https://issues.apache.org/jira/browse/DRILL-5224
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.10.0
> Environment: Windows 10
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: 1.10.0
>
>
> Problem 1:
> Error occurs when attempting to create temporary table on Windows:
> {noformat}
> 0: jdbc:drill:zk=local> create temporary table t as select * from sys.version;
> Error: SYSTEM ERROR: InvalidPathException: Illegal char <:> at index 4: 
> file:///\tmp\3191db8e-279d-4ced-b0e5-32b3b477edfb
> {noformat}
> Root cause:
> when creating temporary directory we merge file system uri, temporary 
> workspace location and session id into one path using 
> java.nio.file.Paths.get() but this method cannot resolve path when path has 
> different delimiters.
> Fix:
> Use org.apache.hadoop.fs.Path tools to merge path, path string is normalized 
> during creation.
> {noformat}
> new Path(fs.getUri().toString(), new 
> Path(temporaryWorkspace.getDefaultLocation(), sessionId));
> {noformat}
> Problem 2:
> When temporary table is being manually dropped using drop table command, 
> though actual table is dropped, remnant folder is left.
> Root cause:
> Before adding to temporary table to the list of temporary tables, its 
> generated name is concatenated with session id (as parent and child folders). 
> java.nio.file.Paths.get() is used for concatenation but it preserves current 
> system delimiter. When table is being dropped, passed table name is split 
> using org.apache.hadoop.fs.Path.SEPARATOR, since it's assumed that path was 
> created using org.apache.hadoop.fs.Path tools where path separators are 
> normalized to one format disregarding the system.
> Fix:
> Concatenate session id with generated table name using 
> org.apache.hadoop.fs.Path tools.
> {noformat}
> new Path(sessionId, UUID.randomUUID().toString()).toUri().getPath();
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-1328) Support table statistics

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-1328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15851851#comment-15851851
 ] 

ASF GitHub Bot commented on DRILL-1328:
---

Github user paul-rogers commented on the issue:

https://github.com/apache/drill/pull/729
  
Please rename PR to standard format:
```
DRILL-1328: Some textual description
```
For us novices, "r4" is a bit of a terse description of the purpose of this 
PR...


> Support table statistics
> 
>
> Key: DRILL-1328
> URL: https://issues.apache.org/jira/browse/DRILL-1328
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Cliff Buchanan
>Assignee: Gautam Kumar Parai
> Fix For: Future
>
> Attachments: 0001-PRE-Set-value-count-in-splitAndTransfer.patch
>
>
> This consists of several subtasks
> * implement operators to generate statistics
> * add "analyze table" support to parser/planner
> * create a metadata provider to allow statistics to be used by optiq in 
> planning optimization
> * implement statistics functions
> Right now, the bulk of this functionality is implemented, but it hasn't been 
> rigorously tested and needs to have some definite answers for some of the 
> parts "around the edges" (how analyze table figures out where the table 
> statistics are located, how a table "append" should work in a read only file 
> system)
> Also, here are a few known caveats:
> * table statistics are collected by creating a sql query based on the string 
> path of the table. This should probably be done with a Table reference.
> * Case sensitivity for column statistics is probably iffy
> * Math for combining two column NDVs into a joint NDV should be checked.
> * Schema changes aren't really being considered yet.
> * adding getDrillTable is probably unnecessary; it might be better to do 
> getTable().unwrap(DrillTable.class)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5224) CTTAS: fix errors connected with system path delimiters (Windows)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15851848#comment-15851848
 ] 

ASF GitHub Bot commented on DRILL-5224:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/731#discussion_r99389277
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/rpc/user/UserSession.java ---
@@ -278,7 +277,7 @@ public boolean setSessionOption(String name, String 
value) {
*/
   public String registerTemporaryTable(AbstractSchema schema, String 
tableName) throws IOException {
   addTemporaryLocation((WorkspaceSchemaFactory.WorkspaceSchema) 
schema);
-  String temporaryTableName = Paths.get(sessionId, 
UUID.randomUUID().toString()).toString();
+  String temporaryTableName = new Path(sessionId, 
UUID.randomUUID().toString()).toUri().getPath();
--- End diff --

I wonder if we should use one of the many platform-independent path 
representations? We have the original Java `File`, the new Java `Path`, the 
Hadoop HDFS `Path`... If we use strings for paths, we run into the very 
problems that those packages are designed to address.

For example, do the above handle the "D:" directory suffix on Windows? 
Resolving "." and ".." in paths? The "~" indicator of use home folder? These 
all vary across platforms.


> CTTAS: fix errors connected with system path delimiters (Windows)
> -
>
> Key: DRILL-5224
> URL: https://issues.apache.org/jira/browse/DRILL-5224
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.10.0
> Environment: Windows 10
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: 1.10.0
>
>
> Problem 1:
> Error occurs when attempting to create temporary table on Windows:
> {noformat}
> 0: jdbc:drill:zk=local> create temporary table t as select * from sys.version;
> Error: SYSTEM ERROR: InvalidPathException: Illegal char <:> at index 4: 
> file:///\tmp\3191db8e-279d-4ced-b0e5-32b3b477edfb
> {noformat}
> Root cause:
> when creating temporary directory we merge file system uri, temporary 
> workspace location and session id into one path using 
> java.nio.file.Paths.get() but this method cannot resolve path when path has 
> different delimiters.
> Fix:
> Use org.apache.hadoop.fs.Path tools to merge path, path string is normalized 
> during creation.
> {noformat}
> new Path(fs.getUri().toString(), new 
> Path(temporaryWorkspace.getDefaultLocation(), sessionId));
> {noformat}
> Problem 2:
> When temporary table is being manually dropped using drop table command, 
> though actual table is dropped, remnant folder is left.
> Root cause:
> Before adding to temporary table to the list of temporary tables, its 
> generated name is concatenated with session id (as parent and child folders). 
> java.nio.file.Paths.get() is used for concatenation but it preserves current 
> system delimiter. When table is being dropped, passed table name is split 
> using org.apache.hadoop.fs.Path.SEPARATOR, since it's assumed that path was 
> created using org.apache.hadoop.fs.Path tools where path separators are 
> normalized to one format disregarding the system.
> Fix:
> Concatenate session id with generated table name using 
> org.apache.hadoop.fs.Path tools.
> {noformat}
> new Path(sessionId, UUID.randomUUID().toString()).toUri().getPath();
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5238) CTTAS: unable to resolve temporary table if workspace is indicated without schema

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15851838#comment-15851838
 ] 

ASF GitHub Bot commented on DRILL-5238:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/736#discussion_r99388535
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/SqlConverter.java
 ---
@@ -527,5 +529,31 @@ public RelOptTableImpl getTable(final List 
names) {
   }
   return super.getTable(names);
 }
+
+/**
+ * Check if passed table is temporary or not should be done if:
+ * schema is not indicated (only one element in the names 
list)
+ * current schema or indicated schema is default temporary 
workspace
+ *
+ * Examples (where dfs.tmp is default temporary workspace):
+ * select * from t
+ * select * from dfs.tmp.t
+ * use dfs; select * from tmp.t
+ *
+ * @param names list of schema and table names, table name 
is always the last element
+ * @param defaultSchemaPath current schema path set using USE command
+ * @param drillConfig   drill config
+ * @return true if check for temporary table should be done, false 
otherwise
+ */
+private boolean checkForTemporaryTable(List names, String 
defaultSchemaPath, DrillConfig drillConfig) {
+  if (names.size() == 1) {
+return true;
+  }
--- End diff --

A single name might be a temporary table, or it might not be. Hence the 
name suggestion above. Is the intent to simply apply a filter so that we 
separate out those that are not temporary tables from those that may or may not 
be?

If we want a positive check that this is, in fact, a temp table, wouldn't 
we have to resolve the name against known temp tables rather than just infer 
based on path length?

And, if we are just filtering out known non-temp tables, is that good 
enough for the use case? That is, can the case we are fixing handle the 
ambiguous "may or may not be temporary" case?


> CTTAS: unable to resolve temporary table if workspace is indicated without 
> schema
> -
>
> Key: DRILL-5238
> URL: https://issues.apache.org/jira/browse/DRILL-5238
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.10.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>  Labels: ready-to-commit
> Fix For: 1.10.0
>
>
> Drill is unable to resolve temporary table if default temporary workspace is 
> partially indicated (schema was set using USE command and only workspace name 
> is used in query).
> {noformat}
> 0: jdbc:drill:zk=local> use dfs;
> +---+--+
> |  ok   | summary  |
> +---+--+
> | true  | Default schema changed to [dfs]  |
> +---+--+
> 1 row selected (0.156 seconds)
> 0: jdbc:drill:zk=local> create temporary table tmp.t as select 'A' from 
> (values(1));
> +---++
> | Fragment  | Number of records written  |
> +---++
> | 0_0   | 1  |
> +---++
> 1 row selected (1.525 seconds)
> 0: jdbc:drill:zk=local> select * from tmp.t;
> Feb 02, 2017 11:24:09 AM 
> org.apache.calcite.sql.validate.SqlValidatorException 
> SEVERE: org.apache.calcite.sql.validate.SqlValidatorException: Table 'tmp.t' 
> not found
> Feb 02, 2017 11:24:09 AM org.apache.calcite.runtime.CalciteException 
> SEVERE: org.apache.calcite.runtime.CalciteContextException: From line 1, 
> column 15 to line 1, column 17: Table 'tmp.t' not found
> Error: VALIDATION ERROR: From line 1, column 15 to line 1, column 17: Table 
> 'tmp.t' not found
> SQL Query null
> [Error Id: 5266cb67-9d37-4a94-9a4e-28a4a2f94be5 on localhost:31010] 
> (state=,code=0)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5238) CTTAS: unable to resolve temporary table if workspace is indicated without schema

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15851839#comment-15851839
 ] 

ASF GitHub Bot commented on DRILL-5238:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/736#discussion_r99387926
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/SqlConverter.java
 ---
@@ -527,5 +529,31 @@ public RelOptTableImpl getTable(final List 
names) {
   }
   return super.getTable(names);
 }
+
+/**
+ * Check if passed table is temporary or not should be done if:
+ * schema is not indicated (only one element in the names 
list)
+ * current schema or indicated schema is default temporary 
workspace
+ *
+ * Examples (where dfs.tmp is default temporary workspace):
+ * select * from t
+ * select * from dfs.tmp.t
+ * use dfs; select * from tmp.t
+ *
+ * @param names list of schema and table names, table name 
is always the last element
+ * @param defaultSchemaPath current schema path set using USE command
+ * @param drillConfig   drill config
+ * @return true if check for temporary table should be done, false 
otherwise
+ */
+private boolean checkForTemporaryTable(List names, String 
defaultSchemaPath, DrillConfig drillConfig) {
--- End diff --

`checkForTemporaryTable` --> `isTemporaryTable` ?

Actually, given the first condition, is this `mightBeTemporaryTable`?


> CTTAS: unable to resolve temporary table if workspace is indicated without 
> schema
> -
>
> Key: DRILL-5238
> URL: https://issues.apache.org/jira/browse/DRILL-5238
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.10.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>  Labels: ready-to-commit
> Fix For: 1.10.0
>
>
> Drill is unable to resolve temporary table if default temporary workspace is 
> partially indicated (schema was set using USE command and only workspace name 
> is used in query).
> {noformat}
> 0: jdbc:drill:zk=local> use dfs;
> +---+--+
> |  ok   | summary  |
> +---+--+
> | true  | Default schema changed to [dfs]  |
> +---+--+
> 1 row selected (0.156 seconds)
> 0: jdbc:drill:zk=local> create temporary table tmp.t as select 'A' from 
> (values(1));
> +---++
> | Fragment  | Number of records written  |
> +---++
> | 0_0   | 1  |
> +---++
> 1 row selected (1.525 seconds)
> 0: jdbc:drill:zk=local> select * from tmp.t;
> Feb 02, 2017 11:24:09 AM 
> org.apache.calcite.sql.validate.SqlValidatorException 
> SEVERE: org.apache.calcite.sql.validate.SqlValidatorException: Table 'tmp.t' 
> not found
> Feb 02, 2017 11:24:09 AM org.apache.calcite.runtime.CalciteException 
> SEVERE: org.apache.calcite.runtime.CalciteContextException: From line 1, 
> column 15 to line 1, column 17: Table 'tmp.t' not found
> Error: VALIDATION ERROR: From line 1, column 15 to line 1, column 17: Table 
> 'tmp.t' not found
> SQL Query null
> [Error Id: 5266cb67-9d37-4a94-9a4e-28a4a2f94be5 on localhost:31010] 
> (state=,code=0)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (DRILL-5190) Display planning time for a query in its profile page

[Sudheesh Katkam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sudheesh Katkam updated DRILL-5190:
---
Labels:   (was: ready-to-commit)

> Display planning time for a query in its profile page
> -
>
> Key: DRILL-5190
> URL: https://issues.apache.org/jira/browse/DRILL-5190
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Web Server
>Affects Versions: 1.9.0
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>
> Currently, the Web UI does not display the time spent in planning for a query 
> in its profile page. The estimate needs to be done by seeing how late did the 
> earliest major fragment start. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5190) Display planning time for a query in its profile page

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15851819#comment-15851819
 ] 

ASF GitHub Bot commented on DRILL-5190:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/738#discussion_r99386366
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/profile/ProfileResources.java
 ---
@@ -77,12 +77,14 @@
*/
   public static String getPrettyDuration(long startTimeMillis, long 
endTimeMillis) {
 long durationInMillis = (startTimeMillis > endTimeMillis ? 
System.currentTimeMillis() : endTimeMillis) - startTimeMillis;
-long hours = TimeUnit.MILLISECONDS.toHours(durationInMillis);
+long days = TimeUnit.MILLISECONDS.toDays(durationInMillis);
--- End diff --

This logic more-or-less duplicates that in DRILL-5230. Please pull this 
logic out into a common utility, and adopt a common display format.


> Display planning time for a query in its profile page
> -
>
> Key: DRILL-5190
> URL: https://issues.apache.org/jira/browse/DRILL-5190
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Web Server
>Affects Versions: 1.9.0
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>  Labels: ready-to-commit
>
> Currently, the Web UI does not display the time spent in planning for a query 
> in its profile page. The estimate needs to be done by seeing how late did the 
> earliest major fragment start. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (DRILL-5230) Translation of millisecond duration into hours is incorrect

[Sudheesh Katkam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sudheesh Katkam updated DRILL-5230:
---
Labels: easyfix  (was: easyfix ready-to-commit)

> Translation of millisecond duration into hours is incorrect
> ---
>
> Key: DRILL-5230
> URL: https://issues.apache.org/jira/browse/DRILL-5230
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Web Server
>Affects Versions: 1.9.0
>Reporter: Kunal Khatua
>  Labels: easyfix
> Fix For: 1.10.0
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> The method 
> {code:JAVA}org.apache.drill.exec.server.rest.profile.TableBuilder.appendMillis(long,
>  String){code}
> has a bug where the human readable translation of a 1+ hr duration in 
> milliseconds is reported incorrectly. 
> This has to do with the {code:JAVA}SimpleDateFormat.format() {code} method 
> incorectly translating it. 
> For e.g.
> {code:JAVA}
> long x = 4545342L; //1 hour 15 min 45.342 sec
> public void appendMillis(x, null);
> {code}
> This formats the value as {noformat}17h15m{noformat} instead of 
> {noformat}1h15m{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5230) Translation of millisecond duration into hours is incorrect

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15851805#comment-15851805
 ] 

ASF GitHub Bot commented on DRILL-5230:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/739#discussion_r99385089
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/profile/TableBuilder.java
 ---
@@ -56,6 +47,30 @@ public TableBuilder(final String[] columns) {
 sb.append("\n");
   }
 
+  /**
+   * Representation of a millisecond duration in a short readable text
+   * @param millis  Duration in milliseconds
+   * @returnHuman-Readable Duration Time
+   */
+  public String shortDurationFormat(long millis) {
+long days = TimeUnit.MILLISECONDS.toDays(millis);
+long hours = TimeUnit.MILLISECONDS.toHours(millis) - 
TimeUnit.DAYS.toHours(TimeUnit.MILLISECONDS.toDays(millis));
+long minutes = TimeUnit.MILLISECONDS.toMinutes(millis) - 
TimeUnit.HOURS.toMinutes(TimeUnit.MILLISECONDS.toHours(millis));
+long seconds = TimeUnit.MILLISECONDS.toSeconds(millis) - 
TimeUnit.MINUTES.toSeconds(TimeUnit.MILLISECONDS.toMinutes(millis));
+long milliSeconds = millis - 
TimeUnit.SECONDS.toMillis(TimeUnit.MILLISECONDS.toSeconds(millis));
+String formattedDuration = "";
+if (days >= 1) {
+  formattedDuration = days + "d" + hours + "h" + minutes + "m";
+} else if (hours >= 1) {
+  formattedDuration = hours + "h" + minutes + "m";
+} else if (minutes >= 1) {
+  formattedDuration = minutes + "m" + seconds + "s";
+} else {
+  formattedDuration = seconds + "." + milliSeconds + "s";
--- End diff --

The above won't work in general. If the time is 2 ms, you'll get an output 
of 0.2s, which is not quite right...

Here, consider using the very handy `String.format` method:
```
String.format("%.3f", seconds + milliSeconds / 1000.0 );
```


> Translation of millisecond duration into hours is incorrect
> ---
>
> Key: DRILL-5230
> URL: https://issues.apache.org/jira/browse/DRILL-5230
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Web Server
>Affects Versions: 1.9.0
>Reporter: Kunal Khatua
>  Labels: easyfix, ready-to-commit
> Fix For: 1.10.0
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> The method 
> {code:JAVA}org.apache.drill.exec.server.rest.profile.TableBuilder.appendMillis(long,
>  String){code}
> has a bug where the human readable translation of a 1+ hr duration in 
> milliseconds is reported incorrectly. 
> This has to do with the {code:JAVA}SimpleDateFormat.format() {code} method 
> incorectly translating it. 
> For e.g.
> {code:JAVA}
> long x = 4545342L; //1 hour 15 min 45.342 sec
> public void appendMillis(x, null);
> {code}
> This formats the value as {noformat}17h15m{noformat} instead of 
> {noformat}1h15m{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5230) Translation of millisecond duration into hours is incorrect

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15851804#comment-15851804
 ] 

ASF GitHub Bot commented on DRILL-5230:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/739#discussion_r99385541
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/profile/TableBuilder.java
 ---
@@ -56,6 +47,30 @@ public TableBuilder(final String[] columns) {
 sb.append("\n");
   }
 
+  /**
+   * Representation of a millisecond duration in a short readable text
+   * @param millis  Duration in milliseconds
+   * @returnHuman-Readable Duration Time
+   */
+  public String shortDurationFormat(long millis) {
--- End diff --

Used elsewhere? If so, consider making static (so we don't need an 
instance.) If only used here, consider making private.

In fact, maybe move this out to a utility class. This logic can be used 
anywhere we want to display a duration such as in logs, in tests, etc.


> Translation of millisecond duration into hours is incorrect
> ---
>
> Key: DRILL-5230
> URL: https://issues.apache.org/jira/browse/DRILL-5230
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Web Server
>Affects Versions: 1.9.0
>Reporter: Kunal Khatua
>  Labels: easyfix, ready-to-commit
> Fix For: 1.10.0
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> The method 
> {code:JAVA}org.apache.drill.exec.server.rest.profile.TableBuilder.appendMillis(long,
>  String){code}
> has a bug where the human readable translation of a 1+ hr duration in 
> milliseconds is reported incorrectly. 
> This has to do with the {code:JAVA}SimpleDateFormat.format() {code} method 
> incorectly translating it. 
> For e.g.
> {code:JAVA}
> long x = 4545342L; //1 hour 15 min 45.342 sec
> public void appendMillis(x, null);
> {code}
> This formats the value as {noformat}17h15m{noformat} instead of 
> {noformat}1h15m{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5230) Translation of millisecond duration into hours is incorrect

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15851803#comment-15851803
 ] 

ASF GitHub Bot commented on DRILL-5230:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/739#discussion_r99384748
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/profile/TableBuilder.java
 ---
@@ -56,6 +47,30 @@ public TableBuilder(final String[] columns) {
 sb.append("\n");
   }
 
+  /**
+   * Representation of a millisecond duration in a short readable text
+   * @param millis  Duration in milliseconds
+   * @returnHuman-Readable Duration Time
+   */
+  public String shortDurationFormat(long millis) {
+long days = TimeUnit.MILLISECONDS.toDays(millis);
+long hours = TimeUnit.MILLISECONDS.toHours(millis) - 
TimeUnit.DAYS.toHours(TimeUnit.MILLISECONDS.toDays(millis));
+long minutes = TimeUnit.MILLISECONDS.toMinutes(millis) - 
TimeUnit.HOURS.toMinutes(TimeUnit.MILLISECONDS.toHours(millis));
+long seconds = TimeUnit.MILLISECONDS.toSeconds(millis) - 
TimeUnit.MINUTES.toSeconds(TimeUnit.MILLISECONDS.toMinutes(millis));
+long milliSeconds = millis - 
TimeUnit.SECONDS.toMillis(TimeUnit.MILLISECONDS.toSeconds(millis));
+String formattedDuration = "";
+if (days >= 1) {
+  formattedDuration = days + "d" + hours + "h" + minutes + "m";
--- End diff --

Is this the display format? Maybe add some space:

```
5d 14h 2m
```

Instead of:
```
5d14h2m
```


> Translation of millisecond duration into hours is incorrect
> ---
>
> Key: DRILL-5230
> URL: https://issues.apache.org/jira/browse/DRILL-5230
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Web Server
>Affects Versions: 1.9.0
>Reporter: Kunal Khatua
>  Labels: easyfix, ready-to-commit
> Fix For: 1.10.0
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> The method 
> {code:JAVA}org.apache.drill.exec.server.rest.profile.TableBuilder.appendMillis(long,
>  String){code}
> has a bug where the human readable translation of a 1+ hr duration in 
> milliseconds is reported incorrectly. 
> This has to do with the {code:JAVA}SimpleDateFormat.format() {code} method 
> incorectly translating it. 
> For e.g.
> {code:JAVA}
> long x = 4545342L; //1 hour 15 min 45.342 sec
> public void appendMillis(x, null);
> {code}
> This formats the value as {noformat}17h15m{noformat} instead of 
> {noformat}1h15m{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5230) Translation of millisecond duration into hours is incorrect

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15851806#comment-15851806
 ] 

ASF GitHub Bot commented on DRILL-5230:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/739#discussion_r99385729
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/profile/TableBuilder.java
 ---
@@ -78,29 +93,8 @@ public void appendTime(final long d, final String link) {
   }
 
   public void appendMillis(final long p, final String link) {
-final double secs = p/1000.0;
-final double mins = secs/60;
-final double hours = mins/60;
-final double days = hours / 24;
-SimpleDateFormat timeFormat = null;
-if (days >= 10) {
-  timeFormat = this.days;
-} else if (days >= 1) {
-  timeFormat = this.sdays;
-} else if (hours >= 10) {
-  timeFormat = this.hours;
-}else if(hours >= 1){
-  timeFormat = this.shours;
-}else if (mins >= 10){
-  timeFormat = this.mins;
-}else if (mins >= 1){
-  timeFormat = this.smins;
-}else if (secs >= 10){
-  timeFormat = this.secs;
-}else {
-  timeFormat = this.ssecs;
-}
-appendCell(timeFormat.format(new Date(p)), null);
+String shortReadableDuration = shortDurationFormat(p);
+appendCell(shortReadableDuration, link);
--- End diff --

In Drill, we like brevity:
```
appendCell(shortDurationFormat(p), link);
```


> Translation of millisecond duration into hours is incorrect
> ---
>
> Key: DRILL-5230
> URL: https://issues.apache.org/jira/browse/DRILL-5230
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Web Server
>Affects Versions: 1.9.0
>Reporter: Kunal Khatua
>  Labels: easyfix, ready-to-commit
> Fix For: 1.10.0
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> The method 
> {code:JAVA}org.apache.drill.exec.server.rest.profile.TableBuilder.appendMillis(long,
>  String){code}
> has a bug where the human readable translation of a 1+ hr duration in 
> milliseconds is reported incorrectly. 
> This has to do with the {code:JAVA}SimpleDateFormat.format() {code} method 
> incorectly translating it. 
> For e.g.
> {code:JAVA}
> long x = 4545342L; //1 hour 15 min 45.342 sec
> public void appendMillis(x, null);
> {code}
> This formats the value as {noformat}17h15m{noformat} instead of 
> {noformat}1h15m{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5230) Translation of millisecond duration into hours is incorrect

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15851807#comment-15851807
 ] 

ASF GitHub Bot commented on DRILL-5230:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/739#discussion_r99385390
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/profile/TableBuilder.java
 ---
@@ -56,6 +47,30 @@ public TableBuilder(final String[] columns) {
 sb.append("\n");
   }
 
+  /**
+   * Representation of a millisecond duration in a short readable text
+   * @param millis  Duration in milliseconds
+   * @returnHuman-Readable Duration Time
+   */
+  public String shortDurationFormat(long millis) {
+long days = TimeUnit.MILLISECONDS.toDays(millis);
+long hours = TimeUnit.MILLISECONDS.toHours(millis) - 
TimeUnit.DAYS.toHours(TimeUnit.MILLISECONDS.toDays(millis));
+long minutes = TimeUnit.MILLISECONDS.toMinutes(millis) - 
TimeUnit.HOURS.toMinutes(TimeUnit.MILLISECONDS.toHours(millis));
+long seconds = TimeUnit.MILLISECONDS.toSeconds(millis) - 
TimeUnit.MINUTES.toSeconds(TimeUnit.MILLISECONDS.toMinutes(millis));
+long milliSeconds = millis - 
TimeUnit.SECONDS.toMillis(TimeUnit.MILLISECONDS.toSeconds(millis));
+String formattedDuration = "";
+if (days >= 1) {
+  formattedDuration = days + "d" + hours + "h" + minutes + "m";
+} else if (hours >= 1) {
+  formattedDuration = hours + "h" + minutes + "m";
+} else if (minutes >= 1) {
+  formattedDuration = minutes + "m" + seconds + "s";
+} else {
+  formattedDuration = seconds + "." + milliSeconds + "s";
+}
+return formattedDuration;
--- End diff --

A nit, but the code is clearer if you do:

```
if (cond1) {
return expr1;
} else if (cond2) {
return expr2;
} else {
return expr3;
}
```

The above eliminates the temp variable and makes it very clear what is 
happening.


> Translation of millisecond duration into hours is incorrect
> ---
>
> Key: DRILL-5230
> URL: https://issues.apache.org/jira/browse/DRILL-5230
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Web Server
>Affects Versions: 1.9.0
>Reporter: Kunal Khatua
>  Labels: easyfix, ready-to-commit
> Fix For: 1.10.0
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> The method 
> {code:JAVA}org.apache.drill.exec.server.rest.profile.TableBuilder.appendMillis(long,
>  String){code}
> has a bug where the human readable translation of a 1+ hr duration in 
> milliseconds is reported incorrectly. 
> This has to do with the {code:JAVA}SimpleDateFormat.format() {code} method 
> incorectly translating it. 
> For e.g.
> {code:JAVA}
> long x = 4545342L; //1 hour 15 min 45.342 sec
> public void appendMillis(x, null);
> {code}
> This formats the value as {noformat}17h15m{noformat} instead of 
> {noformat}1h15m{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-4864) Add ANSI format for date/time functions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15851361#comment-15851361
 ] 

ASF GitHub Bot commented on DRILL-4864:
---

Github user Serhii-Harnyk commented on the issue:

https://github.com/apache/drill/pull/581
  
@paul-rogers could you please review new changes?


> Add ANSI format for date/time functions
> ---
>
> Key: DRILL-4864
> URL: https://issues.apache.org/jira/browse/DRILL-4864
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.8.0
>Reporter: Serhii Harnyk
>Assignee: Serhii Harnyk
>  Labels: doc-impacting
> Fix For: Future
>
>
> The TO_DATE() is exposing the Joda string formatting conventions into the SQL 
> layer. This is not following SQL conventions used by ANSI and many other 
> database engines on the market.
> Add new UDFs: 
> * sql_to_date(String, Format), 
> * sql_to_time(String, Format), 
> * sql_to_timestamp(String, Format)
> that requires Postgres datetime format.
> Table of supported Postgres patterns
> ||Pattern name||Postgres format   
> |Full name of day|day   
> |Day of year|ddd   
> |Day of month|dd
> |Day of week|d   
> |Name of month|month
> |Abr name of month|mon
> |Full era name|ee
> |Name of day|dy   
> |Time zone|tz   
> |Hour 12 |hh   
> |Hour 12 |hh12   
> |Hour 24|hh24
> |Minute of hour|mi  
> |Second of minute|ss   
> |Millisecond of minute|ms
> |Week of year|ww   
> |Month|mm   
> |Halfday am|am
> |Year   |   y   
> |ref.|
> https://www.postgresql.org/docs/8.2/static/functions-formatting.html   |
> Table of acceptable Postgres pattern modifiers, which may be used in Format 
> string
> ||Description||Pattern||
> |fill mode (suppress padding blanks and zeroes)|fm |
> |fixed format global option (see usage notes)|fx |
> |translation mode (print localized day and month names based on 
> lc_messages)|tm |
> |spell mode (not yet implemented)|sp|
> |ref.|
> https://www.postgresql.org/docs/8.2/static/functions-formatting.html|



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-4864) Add ANSI format for date/time functions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15851349#comment-15851349
 ] 

ASF GitHub Bot commented on DRILL-4864:
---

Github user Serhii-Harnyk commented on a diff in the pull request:

https://github.com/apache/drill/pull/581#discussion_r99317199
  
--- Diff: 
logical/src/test/java/org/apache/drill/common/expression/fn/JodaDateValidatorTest.java
 ---
@@ -0,0 +1,176 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements.  See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to you under the Apache License, Version 2.0
+* (the "License"); you may not use this file except in compliance with
+* the License.  You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+package org.apache.drill.common.expression.fn;
+
+import org.joda.time.DateTime;
+import org.joda.time.DateTimeZone;
+import org.joda.time.format.DateTimeFormatter;
+import org.junit.Assert;
+import org.junit.Test;
+
+import static 
org.apache.drill.common.expression.fn.JodaDateValidator.toJodaFormat;
+import static org.joda.time.DateTime.parse;
+import static org.joda.time.format.DateTimeFormat.forPattern;
+
+public class JodaDateValidatorTest {
--- End diff --

Done


> Add ANSI format for date/time functions
> ---
>
> Key: DRILL-4864
> URL: https://issues.apache.org/jira/browse/DRILL-4864
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.8.0
>Reporter: Serhii Harnyk
>Assignee: Serhii Harnyk
>  Labels: doc-impacting
> Fix For: Future
>
>
> The TO_DATE() is exposing the Joda string formatting conventions into the SQL 
> layer. This is not following SQL conventions used by ANSI and many other 
> database engines on the market.
> Add new UDFs: 
> * sql_to_date(String, Format), 
> * sql_to_time(String, Format), 
> * sql_to_timestamp(String, Format)
> that requires Postgres datetime format.
> Table of supported Postgres patterns
> ||Pattern name||Postgres format   
> |Full name of day|day   
> |Day of year|ddd   
> |Day of month|dd
> |Day of week|d   
> |Name of month|month
> |Abr name of month|mon
> |Full era name|ee
> |Name of day|dy   
> |Time zone|tz   
> |Hour 12 |hh   
> |Hour 12 |hh12   
> |Hour 24|hh24
> |Minute of hour|mi  
> |Second of minute|ss   
> |Millisecond of minute|ms
> |Week of year|ww   
> |Month|mm   
> |Halfday am|am
> |Year   |   y   
> |ref.|
> https://www.postgresql.org/docs/8.2/static/functions-formatting.html   |
> Table of acceptable Postgres pattern modifiers, which may be used in Format 
> string
> ||Description||Pattern||
> |fill mode (suppress padding blanks and zeroes)|fm |
> |fixed format global option (see usage notes)|fx |
> |translation mode (print localized day and month names based on 
> lc_messages)|tm |
> |spell mode (not yet implemented)|sp|
> |ref.|
> https://www.postgresql.org/docs/8.2/static/functions-formatting.html|



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-4864) Add ANSI format for date/time functions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15851345#comment-15851345
 ] 

ASF GitHub Bot commented on DRILL-4864:
---

Github user Serhii-Harnyk commented on a diff in the pull request:

https://github.com/apache/drill/pull/581#discussion_r99313550
  
--- Diff: 
logical/src/main/java/org/apache/drill/common/expression/fn/JodaDateValidator.java
 ---
@@ -0,0 +1,216 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements.  See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to you under the Apache License, Version 2.0
+* (the "License"); you may not use this file except in compliance with
+* the License.  You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+package org.apache.drill.common.expression.fn;
+
+import com.google.common.collect.Sets;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.drill.common.map.CaseInsensitiveMap;
+
+import java.util.Comparator;
+import java.util.Set;
+
+public class JodaDateValidator {
+
+  private static final Set postgresValuesForDeleting = 
Sets.newTreeSet(new LengthDescComparator());
+  private static final CaseInsensitiveMap postgresToJodaMap = 
CaseInsensitiveMap.newTreeMap(new LengthDescComparator());
+
+  // tokens for deleting
+  public static final String SUFFIX_SP = "sp";
+  public static final String PREFIX_FM = "fm";
+  public static final String PREFIX_FX = "fx";
+  public static final String PREFIX_TM = "tm";
+
+  // postgres patterns
+  public static final String POSTGRES_FULL_NAME_OF_DAY = "day";
+  public static final String POSTGRES_DAY_OF_YEAR = "ddd";
+  public static final String POSTGRES_DAY_OF_MONTH = "dd";
+  public static final String POSTGRES_DAY_OF_WEEK = "d";
+  public static final String POSTGRES_NAME_OF_MONTH = "month";
+  public static final String POSTGRES_ABR_NAME_OF_MONTH = "mon";
+  public static final String POSTGRES_FULL_ERA_NAME = "ee";
+  public static final String POSTGRES_NAME_OF_DAY = "dy";
+  public static final String POSTGRES_TIME_ZONE_NAME = "tz";
+  public static final String POSTGRES_HOUR_12_NAME = "hh";
+  public static final String POSTGRES_HOUR_12_OTHER_NAME = "hh12";
+  public static final String POSTGRES_HOUR_24_NAME = "hh24";
+  public static final String POSTGRES_MINUTE_OF_HOUR_NAME = "mi";
+  public static final String POSTGRES_SECOND_OF_MINUTE_NAME = "ss";
+  public static final String POSTGRES_MILLISECOND_OF_MINUTE_NAME = "ms";
+  public static final String POSTGRES_WEEK_OF_YEAR = "ww";
+  public static final String POSTGRES_MONTH = "mm";
+  public static final String POSTGRES_HALFDAY_AM = "am";
+  public static final String POSTGRES_HALFDAY_PM = "pm";
+
+  // jodaTime patterns
+  public static final String JODA_FULL_NAME_OF_DAY = "";
+  public static final String JODA_DAY_OF_YEAR = "D";
+  public static final String JODA_DAY_OF_MONTH = "d";
+  public static final String JODA_DAY_OF_WEEK = "e";
+  public static final String JODA_NAME_OF_MONTH = "";
+  public static final String JODA_ABR_NAME_OF_MONTH = "MMM";
+  public static final String JODA_FULL_ERA_NAME = "G";
+  public static final String JODA_NAME_OF_DAY = "E";
+  public static final String JODA_TIME_ZONE_NAME = "TZ";
+  public static final String JODA_HOUR_12_NAME = "h";
+  public static final String JODA_HOUR_12_OTHER_NAME = "h";
+  public static final String JODA_HOUR_24_NAME = "H";
+  public static final String JODA_MINUTE_OF_HOUR_NAME = "m";
+  public static final String JODA_SECOND_OF_MINUTE_NAME = "s";
+  public static final String JODA_MILLISECOND_OF_MINUTE_NAME = "SSS";
+  public static final String JODA_WEEK_OF_YEAR = "w";
+  public static final String JODA_MONTH = "MM";
+  public static final String JODA_HALFDAY = "aa";
+
+  static {
+postgresToJodaMap.put(POSTGRES_FULL_NAME_OF_DAY, 
JODA_FULL_NAME_OF_DAY);
+postgresToJodaMap.put(POSTGRES_DAY_OF_YEAR, JODA_DAY_OF_YEAR);
+postgresToJodaMap.put(POSTGRES_DAY_OF_MONTH, JODA_DAY_OF_MONTH);
+postgresToJodaMap.put(POSTGRES_DAY_OF_WEEK, JODA_DAY_OF_WEEK);
+postgresToJodaMap.put(POSTGRES_NAME_OF_MONTH, 

[jira] [Commented] (DRILL-4864) Add ANSI format for date/time functions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15851348#comment-15851348
 ] 

ASF GitHub Bot commented on DRILL-4864:
---

Github user Serhii-Harnyk commented on a diff in the pull request:

https://github.com/apache/drill/pull/581#discussion_r99315665
  
--- Diff: 
logical/src/main/java/org/apache/drill/common/expression/fn/JodaDateValidator.java
 ---
@@ -0,0 +1,216 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements.  See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to you under the Apache License, Version 2.0
+* (the "License"); you may not use this file except in compliance with
+* the License.  You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+package org.apache.drill.common.expression.fn;
+
+import com.google.common.collect.Sets;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.drill.common.map.CaseInsensitiveMap;
+
+import java.util.Comparator;
+import java.util.Set;
+
+public class JodaDateValidator {
+
+  private static final Set postgresValuesForDeleting = 
Sets.newTreeSet(new LengthDescComparator());
+  private static final CaseInsensitiveMap postgresToJodaMap = 
CaseInsensitiveMap.newTreeMap(new LengthDescComparator());
+
+  // tokens for deleting
+  public static final String SUFFIX_SP = "sp";
+  public static final String PREFIX_FM = "fm";
+  public static final String PREFIX_FX = "fx";
+  public static final String PREFIX_TM = "tm";
+
+  // postgres patterns
+  public static final String POSTGRES_FULL_NAME_OF_DAY = "day";
+  public static final String POSTGRES_DAY_OF_YEAR = "ddd";
+  public static final String POSTGRES_DAY_OF_MONTH = "dd";
+  public static final String POSTGRES_DAY_OF_WEEK = "d";
+  public static final String POSTGRES_NAME_OF_MONTH = "month";
+  public static final String POSTGRES_ABR_NAME_OF_MONTH = "mon";
+  public static final String POSTGRES_FULL_ERA_NAME = "ee";
+  public static final String POSTGRES_NAME_OF_DAY = "dy";
+  public static final String POSTGRES_TIME_ZONE_NAME = "tz";
+  public static final String POSTGRES_HOUR_12_NAME = "hh";
+  public static final String POSTGRES_HOUR_12_OTHER_NAME = "hh12";
+  public static final String POSTGRES_HOUR_24_NAME = "hh24";
+  public static final String POSTGRES_MINUTE_OF_HOUR_NAME = "mi";
+  public static final String POSTGRES_SECOND_OF_MINUTE_NAME = "ss";
+  public static final String POSTGRES_MILLISECOND_OF_MINUTE_NAME = "ms";
+  public static final String POSTGRES_WEEK_OF_YEAR = "ww";
+  public static final String POSTGRES_MONTH = "mm";
+  public static final String POSTGRES_HALFDAY_AM = "am";
+  public static final String POSTGRES_HALFDAY_PM = "pm";
+
+  // jodaTime patterns
+  public static final String JODA_FULL_NAME_OF_DAY = "";
+  public static final String JODA_DAY_OF_YEAR = "D";
+  public static final String JODA_DAY_OF_MONTH = "d";
+  public static final String JODA_DAY_OF_WEEK = "e";
+  public static final String JODA_NAME_OF_MONTH = "";
+  public static final String JODA_ABR_NAME_OF_MONTH = "MMM";
+  public static final String JODA_FULL_ERA_NAME = "G";
+  public static final String JODA_NAME_OF_DAY = "E";
+  public static final String JODA_TIME_ZONE_NAME = "TZ";
+  public static final String JODA_HOUR_12_NAME = "h";
+  public static final String JODA_HOUR_12_OTHER_NAME = "h";
+  public static final String JODA_HOUR_24_NAME = "H";
+  public static final String JODA_MINUTE_OF_HOUR_NAME = "m";
+  public static final String JODA_SECOND_OF_MINUTE_NAME = "s";
+  public static final String JODA_MILLISECOND_OF_MINUTE_NAME = "SSS";
+  public static final String JODA_WEEK_OF_YEAR = "w";
+  public static final String JODA_MONTH = "MM";
+  public static final String JODA_HALFDAY = "aa";
+
+  static {
+postgresToJodaMap.put(POSTGRES_FULL_NAME_OF_DAY, 
JODA_FULL_NAME_OF_DAY);
+postgresToJodaMap.put(POSTGRES_DAY_OF_YEAR, JODA_DAY_OF_YEAR);
+postgresToJodaMap.put(POSTGRES_DAY_OF_MONTH, JODA_DAY_OF_MONTH);
+postgresToJodaMap.put(POSTGRES_DAY_OF_WEEK, JODA_DAY_OF_WEEK);
+postgresToJodaMap.put(POSTGRES_NAME_OF_MONTH, 

[jira] [Commented] (DRILL-4864) Add ANSI format for date/time functions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15851347#comment-15851347
 ] 

ASF GitHub Bot commented on DRILL-4864:
---

Github user Serhii-Harnyk commented on a diff in the pull request:

https://github.com/apache/drill/pull/581#discussion_r99316546
  
--- Diff: 
logical/src/main/java/org/apache/drill/common/expression/fn/JodaDateValidator.java
 ---
@@ -0,0 +1,216 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements.  See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to you under the Apache License, Version 2.0
+* (the "License"); you may not use this file except in compliance with
+* the License.  You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+package org.apache.drill.common.expression.fn;
+
+import com.google.common.collect.Sets;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.drill.common.map.CaseInsensitiveMap;
+
+import java.util.Comparator;
+import java.util.Set;
+
+public class JodaDateValidator {
+
+  private static final Set postgresValuesForDeleting = 
Sets.newTreeSet(new LengthDescComparator());
+  private static final CaseInsensitiveMap postgresToJodaMap = 
CaseInsensitiveMap.newTreeMap(new LengthDescComparator());
+
+  // tokens for deleting
+  public static final String SUFFIX_SP = "sp";
+  public static final String PREFIX_FM = "fm";
+  public static final String PREFIX_FX = "fx";
+  public static final String PREFIX_TM = "tm";
+
+  // postgres patterns
+  public static final String POSTGRES_FULL_NAME_OF_DAY = "day";
+  public static final String POSTGRES_DAY_OF_YEAR = "ddd";
+  public static final String POSTGRES_DAY_OF_MONTH = "dd";
+  public static final String POSTGRES_DAY_OF_WEEK = "d";
+  public static final String POSTGRES_NAME_OF_MONTH = "month";
+  public static final String POSTGRES_ABR_NAME_OF_MONTH = "mon";
+  public static final String POSTGRES_FULL_ERA_NAME = "ee";
+  public static final String POSTGRES_NAME_OF_DAY = "dy";
+  public static final String POSTGRES_TIME_ZONE_NAME = "tz";
+  public static final String POSTGRES_HOUR_12_NAME = "hh";
+  public static final String POSTGRES_HOUR_12_OTHER_NAME = "hh12";
+  public static final String POSTGRES_HOUR_24_NAME = "hh24";
+  public static final String POSTGRES_MINUTE_OF_HOUR_NAME = "mi";
+  public static final String POSTGRES_SECOND_OF_MINUTE_NAME = "ss";
+  public static final String POSTGRES_MILLISECOND_OF_MINUTE_NAME = "ms";
+  public static final String POSTGRES_WEEK_OF_YEAR = "ww";
+  public static final String POSTGRES_MONTH = "mm";
+  public static final String POSTGRES_HALFDAY_AM = "am";
+  public static final String POSTGRES_HALFDAY_PM = "pm";
+
+  // jodaTime patterns
+  public static final String JODA_FULL_NAME_OF_DAY = "";
+  public static final String JODA_DAY_OF_YEAR = "D";
+  public static final String JODA_DAY_OF_MONTH = "d";
+  public static final String JODA_DAY_OF_WEEK = "e";
+  public static final String JODA_NAME_OF_MONTH = "";
+  public static final String JODA_ABR_NAME_OF_MONTH = "MMM";
+  public static final String JODA_FULL_ERA_NAME = "G";
+  public static final String JODA_NAME_OF_DAY = "E";
+  public static final String JODA_TIME_ZONE_NAME = "TZ";
+  public static final String JODA_HOUR_12_NAME = "h";
+  public static final String JODA_HOUR_12_OTHER_NAME = "h";
+  public static final String JODA_HOUR_24_NAME = "H";
+  public static final String JODA_MINUTE_OF_HOUR_NAME = "m";
+  public static final String JODA_SECOND_OF_MINUTE_NAME = "s";
+  public static final String JODA_MILLISECOND_OF_MINUTE_NAME = "SSS";
+  public static final String JODA_WEEK_OF_YEAR = "w";
+  public static final String JODA_MONTH = "MM";
+  public static final String JODA_HALFDAY = "aa";
+
+  static {
+postgresToJodaMap.put(POSTGRES_FULL_NAME_OF_DAY, 
JODA_FULL_NAME_OF_DAY);
+postgresToJodaMap.put(POSTGRES_DAY_OF_YEAR, JODA_DAY_OF_YEAR);
+postgresToJodaMap.put(POSTGRES_DAY_OF_MONTH, JODA_DAY_OF_MONTH);
+postgresToJodaMap.put(POSTGRES_DAY_OF_WEEK, JODA_DAY_OF_WEEK);
+postgresToJodaMap.put(POSTGRES_NAME_OF_MONTH, 

[jira] [Commented] (DRILL-4864) Add ANSI format for date/time functions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15851346#comment-15851346
 ] 

ASF GitHub Bot commented on DRILL-4864:
---

Github user Serhii-Harnyk commented on a diff in the pull request:

https://github.com/apache/drill/pull/581#discussion_r99315319
  
--- Diff: 
logical/src/main/java/org/apache/drill/common/expression/fn/JodaDateValidator.java
 ---
@@ -0,0 +1,216 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements.  See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to you under the Apache License, Version 2.0
+* (the "License"); you may not use this file except in compliance with
+* the License.  You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+package org.apache.drill.common.expression.fn;
+
+import com.google.common.collect.Sets;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.drill.common.map.CaseInsensitiveMap;
+
+import java.util.Comparator;
+import java.util.Set;
+
+public class JodaDateValidator {
+
+  private static final Set postgresValuesForDeleting = 
Sets.newTreeSet(new LengthDescComparator());
+  private static final CaseInsensitiveMap postgresToJodaMap = 
CaseInsensitiveMap.newTreeMap(new LengthDescComparator());
+
+  // tokens for deleting
+  public static final String SUFFIX_SP = "sp";
+  public static final String PREFIX_FM = "fm";
+  public static final String PREFIX_FX = "fx";
+  public static final String PREFIX_TM = "tm";
+
+  // postgres patterns
+  public static final String POSTGRES_FULL_NAME_OF_DAY = "day";
+  public static final String POSTGRES_DAY_OF_YEAR = "ddd";
+  public static final String POSTGRES_DAY_OF_MONTH = "dd";
+  public static final String POSTGRES_DAY_OF_WEEK = "d";
+  public static final String POSTGRES_NAME_OF_MONTH = "month";
+  public static final String POSTGRES_ABR_NAME_OF_MONTH = "mon";
+  public static final String POSTGRES_FULL_ERA_NAME = "ee";
+  public static final String POSTGRES_NAME_OF_DAY = "dy";
+  public static final String POSTGRES_TIME_ZONE_NAME = "tz";
+  public static final String POSTGRES_HOUR_12_NAME = "hh";
+  public static final String POSTGRES_HOUR_12_OTHER_NAME = "hh12";
+  public static final String POSTGRES_HOUR_24_NAME = "hh24";
+  public static final String POSTGRES_MINUTE_OF_HOUR_NAME = "mi";
+  public static final String POSTGRES_SECOND_OF_MINUTE_NAME = "ss";
+  public static final String POSTGRES_MILLISECOND_OF_MINUTE_NAME = "ms";
+  public static final String POSTGRES_WEEK_OF_YEAR = "ww";
+  public static final String POSTGRES_MONTH = "mm";
+  public static final String POSTGRES_HALFDAY_AM = "am";
+  public static final String POSTGRES_HALFDAY_PM = "pm";
+
+  // jodaTime patterns
+  public static final String JODA_FULL_NAME_OF_DAY = "";
+  public static final String JODA_DAY_OF_YEAR = "D";
+  public static final String JODA_DAY_OF_MONTH = "d";
+  public static final String JODA_DAY_OF_WEEK = "e";
+  public static final String JODA_NAME_OF_MONTH = "";
+  public static final String JODA_ABR_NAME_OF_MONTH = "MMM";
+  public static final String JODA_FULL_ERA_NAME = "G";
+  public static final String JODA_NAME_OF_DAY = "E";
+  public static final String JODA_TIME_ZONE_NAME = "TZ";
+  public static final String JODA_HOUR_12_NAME = "h";
+  public static final String JODA_HOUR_12_OTHER_NAME = "h";
+  public static final String JODA_HOUR_24_NAME = "H";
+  public static final String JODA_MINUTE_OF_HOUR_NAME = "m";
+  public static final String JODA_SECOND_OF_MINUTE_NAME = "s";
+  public static final String JODA_MILLISECOND_OF_MINUTE_NAME = "SSS";
+  public static final String JODA_WEEK_OF_YEAR = "w";
+  public static final String JODA_MONTH = "MM";
+  public static final String JODA_HALFDAY = "aa";
+
+  static {
+postgresToJodaMap.put(POSTGRES_FULL_NAME_OF_DAY, 
JODA_FULL_NAME_OF_DAY);
+postgresToJodaMap.put(POSTGRES_DAY_OF_YEAR, JODA_DAY_OF_YEAR);
+postgresToJodaMap.put(POSTGRES_DAY_OF_MONTH, JODA_DAY_OF_MONTH);
+postgresToJodaMap.put(POSTGRES_DAY_OF_WEEK, JODA_DAY_OF_WEEK);
+postgresToJodaMap.put(POSTGRES_NAME_OF_MONTH, 

[jira] [Commented] (DRILL-4864) Add ANSI format for date/time functions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15851319#comment-15851319
 ] 

ASF GitHub Bot commented on DRILL-4864:
---

Github user Serhii-Harnyk commented on a diff in the pull request:

https://github.com/apache/drill/pull/581#discussion_r99312091
  
--- Diff: 
exec/java-exec/src/main/codegen/templates/DateIntervalFunctionTemplates/SqlToDateTypeFunctions.java
 ---
@@ -0,0 +1,77 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import org.apache.drill.exec.expr.annotations.Workspace;
+
+<@pp.dropOutputFile/>
+
+<#list dateIntervalFunc.dates as type>
+
+<@pp.changeOutputFile name = 
"/org/apache/drill/exec/expr/fn/impl/SqlTo${type}.java"/>
+
+<#include "/@includes/license.ftl"/>
+
+package org.apache.drill.exec.expr.fn.impl;
+
+import org.apache.drill.exec.expr.DrillSimpleFunc;
+import org.apache.drill.exec.expr.annotations.FunctionTemplate;
+import 
org.apache.drill.exec.expr.annotations.FunctionTemplate.NullHandling;
+import org.apache.drill.exec.expr.annotations.Output;
+import org.apache.drill.exec.expr.annotations.Workspace;
+import org.apache.drill.exec.expr.annotations.Param;
+import org.apache.drill.exec.expr.holders.*;
+
+/*
+ * This class is generated using freemarker and the ${.template_name} 
template.
+ */
+
+@FunctionTemplate(name = "sql_to_${type?lower_case}",
+  scope = FunctionTemplate.FunctionScope.SIMPLE,
+  nulls = NullHandling.NULL_IF_NULL)
+public class SqlTo${type} implements DrillSimpleFunc {
+
+@Param  VarCharHolder left;
+@Param  VarCharHolder right;
+@Workspace org.joda.time.format.DateTimeFormatter format;
+@Output ${type}Holder out;
+
+public void setup() {
+// Get the desired output format
+byte[] buf = new byte[right.end - right.start];
+right.buffer.getBytes(right.start, buf, 0, right.end - 
right.start);
+String formatString = new String(buf, 
com.google.common.base.Charsets.UTF_8);
+String pattern = 
org.apache.drill.common.expression.fn.JodaDateValidator.toJodaFormat(formatString);
+format = org.joda.time.format.DateTimeFormat.forPattern(pattern);
+}
+
+public void eval() {
+// Get the input
+byte[] buf1 = new byte[left.end - left.start];
+left.buffer.getBytes(left.start, buf1, 0, left.end - left.start);
+String input = new String(buf1, 
com.google.common.base.Charsets.UTF_8);
+
+<#if type == "Date">
+out.value = (org.joda.time.DateMidnight.parse(input, 
format).withZoneRetainFields(org.joda.time.DateTimeZone.UTC)).getMillis();
+<#elseif type == "TimeStamp">
+out.value = org.joda.time.DateTime.parse(input, 
format).withZoneRetainFields(org.joda.time.DateTimeZone.UTC).getMillis();
+<#elseif type == "Time">
+out.value = (int) 
((format.parseDateTime(input)).withZoneRetainFields(org.joda.time.DateTimeZone.UTC).getMillis());
+
--- End diff --

Yes, we should. Fixed, thanks.


> Add ANSI format for date/time functions
> ---
>
> Key: DRILL-4864
> URL: https://issues.apache.org/jira/browse/DRILL-4864
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.8.0
>Reporter: Serhii Harnyk
>Assignee: Serhii Harnyk
>  Labels: doc-impacting
> Fix For: Future
>
>
> The TO_DATE() is exposing the Joda string formatting conventions into the SQL 
> layer. This is not following SQL conventions used by ANSI and many other 
> database engines on the market.
> Add new UDFs: 
> * sql_to_date(String, Format), 
> * sql_to_time(String, Format), 
> * sql_to_timestamp(String, Format)
> that requires Postgres datetime format.
> Table of supported Postgres patterns

[jira] [Commented] (DRILL-4864) Add ANSI format for date/time functions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15851320#comment-15851320
 ] 

ASF GitHub Bot commented on DRILL-4864:
---

Github user Serhii-Harnyk commented on a diff in the pull request:

https://github.com/apache/drill/pull/581#discussion_r99312234
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/fn/impl/testing/TestDateConversions.java
 ---
@@ -0,0 +1,200 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements.  See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to you under the Apache License, Version 2.0
+* (the "License"); you may not use this file except in compliance with
+* the License.  You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+package org.apache.drill.exec.fn.impl.testing;
+
+import org.apache.drill.BaseTestQuery;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import java.io.BufferedWriter;
+import java.io.File;
+import java.io.FileWriter;
+import java.io.IOException;
+
+public class TestDateConversions extends BaseTestQuery {
--- End diff --

Done


> Add ANSI format for date/time functions
> ---
>
> Key: DRILL-4864
> URL: https://issues.apache.org/jira/browse/DRILL-4864
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.8.0
>Reporter: Serhii Harnyk
>Assignee: Serhii Harnyk
>  Labels: doc-impacting
> Fix For: Future
>
>
> The TO_DATE() is exposing the Joda string formatting conventions into the SQL 
> layer. This is not following SQL conventions used by ANSI and many other 
> database engines on the market.
> Add new UDFs: 
> * sql_to_date(String, Format), 
> * sql_to_time(String, Format), 
> * sql_to_timestamp(String, Format)
> that requires Postgres datetime format.
> Table of supported Postgres patterns
> ||Pattern name||Postgres format   
> |Full name of day|day   
> |Day of year|ddd   
> |Day of month|dd
> |Day of week|d   
> |Name of month|month
> |Abr name of month|mon
> |Full era name|ee
> |Name of day|dy   
> |Time zone|tz   
> |Hour 12 |hh   
> |Hour 12 |hh12   
> |Hour 24|hh24
> |Minute of hour|mi  
> |Second of minute|ss   
> |Millisecond of minute|ms
> |Week of year|ww   
> |Month|mm   
> |Halfday am|am
> |Year   |   y   
> |ref.|
> https://www.postgresql.org/docs/8.2/static/functions-formatting.html   |
> Table of acceptable Postgres pattern modifiers, which may be used in Format 
> string
> ||Description||Pattern||
> |fill mode (suppress padding blanks and zeroes)|fm |
> |fixed format global option (see usage notes)|fx |
> |translation mode (print localized day and month names based on 
> lc_messages)|tm |
> |spell mode (not yet implemented)|sp|
> |ref.|
> https://www.postgresql.org/docs/8.2/static/functions-formatting.html|



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-4864) Add ANSI format for date/time functions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15851316#comment-15851316
 ] 

ASF GitHub Bot commented on DRILL-4864:
---

Github user Serhii-Harnyk commented on a diff in the pull request:

https://github.com/apache/drill/pull/581#discussion_r99311783
  
--- Diff: 
exec/java-exec/src/main/codegen/templates/DateIntervalFunctionTemplates/SqlToDateTypeFunctions.java
 ---
@@ -0,0 +1,77 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import org.apache.drill.exec.expr.annotations.Workspace;
+
+<@pp.dropOutputFile/>
+
+<#list dateIntervalFunc.dates as type>
+
+<@pp.changeOutputFile name = 
"/org/apache/drill/exec/expr/fn/impl/SqlTo${type}.java"/>
+
+<#include "/@includes/license.ftl"/>
+
+package org.apache.drill.exec.expr.fn.impl;
+
+import org.apache.drill.exec.expr.DrillSimpleFunc;
+import org.apache.drill.exec.expr.annotations.FunctionTemplate;
+import 
org.apache.drill.exec.expr.annotations.FunctionTemplate.NullHandling;
+import org.apache.drill.exec.expr.annotations.Output;
+import org.apache.drill.exec.expr.annotations.Workspace;
+import org.apache.drill.exec.expr.annotations.Param;
+import org.apache.drill.exec.expr.holders.*;
+
+/*
+ * This class is generated using freemarker and the ${.template_name} 
template.
+ */
+
+@FunctionTemplate(name = "sql_to_${type?lower_case}",
+  scope = FunctionTemplate.FunctionScope.SIMPLE,
+  nulls = NullHandling.NULL_IF_NULL)
+public class SqlTo${type} implements DrillSimpleFunc {
+
+@Param  VarCharHolder left;
+@Param  VarCharHolder right;
+@Workspace org.joda.time.format.DateTimeFormatter format;
+@Output ${type}Holder out;
+
+public void setup() {
+// Get the desired output format
+byte[] buf = new byte[right.end - right.start];
+right.buffer.getBytes(right.start, buf, 0, right.end - 
right.start);
+String formatString = new String(buf, 
com.google.common.base.Charsets.UTF_8);
+String pattern = 
org.apache.drill.common.expression.fn.JodaDateValidator.toJodaFormat(formatString);
--- End diff --

Before the using Java 8 classes, we need change some patterns in 
JodaDateValidator.class, because there are some differences, but in general 
there shouldn't be user-visible change in behavior or semantics.


> Add ANSI format for date/time functions
> ---
>
> Key: DRILL-4864
> URL: https://issues.apache.org/jira/browse/DRILL-4864
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.8.0
>Reporter: Serhii Harnyk
>Assignee: Serhii Harnyk
>  Labels: doc-impacting
> Fix For: Future
>
>
> The TO_DATE() is exposing the Joda string formatting conventions into the SQL 
> layer. This is not following SQL conventions used by ANSI and many other 
> database engines on the market.
> Add new UDFs: 
> * sql_to_date(String, Format), 
> * sql_to_time(String, Format), 
> * sql_to_timestamp(String, Format)
> that requires Postgres datetime format.
> Table of supported Postgres patterns
> ||Pattern name||Postgres format   
> |Full name of day|day   
> |Day of year|ddd   
> |Day of month|dd
> |Day of week|d   
> |Name of month|month
> |Abr name of month|mon
> |Full era name|ee
> |Name of day|dy   
> |Time zone|tz   
> |Hour 12 |hh   
> |Hour 12 |hh12   
> |Hour 24|hh24
> |Minute of hour|mi  
> |Second of minute|ss   
> |Millisecond of minute|ms
> |Week of year|ww   
> |Month|mm   
> |Halfday am|am
> |Year   |   y   
> |ref.|
> 

[jira] [Commented] (DRILL-4864) Add ANSI format for date/time functions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15851309#comment-15851309
 ] 

ASF GitHub Bot commented on DRILL-4864:
---

Github user Serhii-Harnyk commented on a diff in the pull request:

https://github.com/apache/drill/pull/581#discussion_r99311060
  
--- Diff: 
exec/java-exec/src/main/codegen/templates/DateIntervalFunctionTemplates/SqlToDateTypeFunctions.java
 ---
@@ -0,0 +1,77 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import org.apache.drill.exec.expr.annotations.Workspace;
+
+<@pp.dropOutputFile/>
+
+<#list dateIntervalFunc.dates as type>
+
+<@pp.changeOutputFile name = 
"/org/apache/drill/exec/expr/fn/impl/SqlTo${type}.java"/>
+
+<#include "/@includes/license.ftl"/>
+
+package org.apache.drill.exec.expr.fn.impl;
+
+import org.apache.drill.exec.expr.DrillSimpleFunc;
+import org.apache.drill.exec.expr.annotations.FunctionTemplate;
+import 
org.apache.drill.exec.expr.annotations.FunctionTemplate.NullHandling;
+import org.apache.drill.exec.expr.annotations.Output;
+import org.apache.drill.exec.expr.annotations.Workspace;
+import org.apache.drill.exec.expr.annotations.Param;
+import org.apache.drill.exec.expr.holders.*;
+
+/*
+ * This class is generated using freemarker and the ${.template_name} 
template.
+ */
+
+@FunctionTemplate(name = "sql_to_${type?lower_case}",
+  scope = FunctionTemplate.FunctionScope.SIMPLE,
+  nulls = NullHandling.NULL_IF_NULL)
+public class SqlTo${type} implements DrillSimpleFunc {
+
+@Param  VarCharHolder left;
+@Param  VarCharHolder right;
+@Workspace org.joda.time.format.DateTimeFormatter format;
+@Output ${type}Holder out;
+
+public void setup() {
+// Get the desired output format
+byte[] buf = new byte[right.end - right.start];
+right.buffer.getBytes(right.start, buf, 0, right.end - 
right.start);
+String formatString = new String(buf, 
com.google.common.base.Charsets.UTF_8);
--- End diff --

I found the method that does the same. Fixed.


> Add ANSI format for date/time functions
> ---
>
> Key: DRILL-4864
> URL: https://issues.apache.org/jira/browse/DRILL-4864
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.8.0
>Reporter: Serhii Harnyk
>Assignee: Serhii Harnyk
>  Labels: doc-impacting
> Fix For: Future
>
>
> The TO_DATE() is exposing the Joda string formatting conventions into the SQL 
> layer. This is not following SQL conventions used by ANSI and many other 
> database engines on the market.
> Add new UDFs: 
> * sql_to_date(String, Format), 
> * sql_to_time(String, Format), 
> * sql_to_timestamp(String, Format)
> that requires Postgres datetime format.
> Table of supported Postgres patterns
> ||Pattern name||Postgres format   
> |Full name of day|day   
> |Day of year|ddd   
> |Day of month|dd
> |Day of week|d   
> |Name of month|month
> |Abr name of month|mon
> |Full era name|ee
> |Name of day|dy   
> |Time zone|tz   
> |Hour 12 |hh   
> |Hour 12 |hh12   
> |Hour 24|hh24
> |Minute of hour|mi  
> |Second of minute|ss   
> |Millisecond of minute|ms
> |Week of year|ww   
> |Month|mm   
> |Halfday am|am
> |Year   |   y   
> |ref.|
> https://www.postgresql.org/docs/8.2/static/functions-formatting.html   |
> Table of acceptable Postgres pattern modifiers, which may be used in Format 
> string
> ||Description||Pattern||
> |fill mode (suppress padding blanks and zeroes)|fm |
> |fixed format global