Re: Review Request 32800: C++ Client Authentication API passing std::string across DLL boundaries
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32800/ --- (Updated April 6, 2015, 6:30 p.m.) Review request for drill, Alexander zarei, Parth Chandra, and Xiao Meng. Bugs: DRILL-2671 https://issues.apache.org/jira/browse/DRILL-2671 Repository: drill-git Description --- DrillUserProperty::setProperty is taking std::string as parameters. Memory gets allocated in the client yet Drill Client tries to clean it up. Diffs (updated) - contrib/native/client/src/include/drill/drillClient.hpp 9289df3 Diff: https://reviews.apache.org/r/32800/diff/ Testing --- Thanks, Norris Lee
[jira] [Created] (DRILL-2697) Pause injections should pause indefinitely until signalled
Sudheesh Katkam created DRILL-2697: -- Summary: Pause injections should pause indefinitely until signalled Key: DRILL-2697 URL: https://issues.apache.org/jira/browse/DRILL-2697 Project: Apache Drill Issue Type: Improvement Components: Execution - Flow Affects Versions: 0.9.0 Reporter: Sudheesh Katkam Fix For: 1.1.0 Currently injected pauses make threads sleep for a specified time. This can be an enhanced to stop the thread indefinitely using a CountDownLatch. It is quite similar to how cancellation works. Tasks: (a) Add another message to RPC layer to signal paused remote threads to resume (through ControlHandler) by counting down. Complications if the thread has not reached the pause site yet. (b) Add resume signal (like ctrl-c) to sqlline (further enhancement: another signal to trigger pause from sqlline) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (DRILL-2637) Schema change reported incorrectly although both the input columns are of same datatype
[
https://issues.apache.org/jira/browse/DRILL-2637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Hsuan-Yi Chu resolved DRILL-2637.
--
Resolution: Fixed
Schema change reported incorrectly although both the input columns are of
same datatype
---
Key: DRILL-2637
URL: https://issues.apache.org/jira/browse/DRILL-2637
Project: Apache Drill
Issue Type: Bug
Affects Versions: 0.9.0
Reporter: Khurram Faraaz
Assignee: Sean Hsuan-Yi Chu
Schema change is reported incorrectly, although the two columns hold data of
same datatype, and those two columns are input to Union All and an aggregate
count is performed on the results returned by Union All.
Tests were run on 4 node cluster.
{code}
0: jdbc:drill: select c1 from (select columns[0] c1 from `testWindow.csv`)
union all (select columns[0] c2 from `testWindow.csv`);
++
| c1 |
++
| 100|
| 10 |
| 2 |
| 50 |
| 55 |
| 67 |
| 113|
| 119|
| 89 |
| 57 |
| 61 |
| 100|
| 10 |
| 2 |
| 50 |
| 55 |
| 67 |
| 113|
| 119|
| 89 |
| 57 |
| 61 |
++
22 rows selected (0.121 seconds)
{code}
{code}
0: jdbc:drill: select count(c1) from (select columns[0] c1 from
`testWindow.csv`) union all (select columns[0] c2 from `testWindow.csv`);
++
| EXPR$0 |
++
Query failed: Query stopped., Schema change detected in the left input of
Union-All. This is not currently supported [
57dd6384-fb23-4ab0-aee9-fb7def390788 on centos-04.qa.lab:31010 ]
java.lang.RuntimeException: java.sql.SQLException: Failure while executing
query.
at sqlline.SqlLine$IncrementalRows.hasNext(SqlLine.java:2514)
at sqlline.SqlLine$TableOutputFormat.print(SqlLine.java:2148)
at sqlline.SqlLine.print(SqlLine.java:1809)
at sqlline.SqlLine$Commands.execute(SqlLine.java:3766)
at sqlline.SqlLine$Commands.sql(SqlLine.java:3663)
at sqlline.SqlLine.dispatch(SqlLine.java:889)
at sqlline.SqlLine.begin(SqlLine.java:763)
at sqlline.SqlLine.start(SqlLine.java:498)
at sqlline.SqlLine.main(SqlLine.java:460)
{code}
Stack trace from drillbit.log
{code}
2015-03-31 20:10:07,825 [2ae500df-db85-2583-fa7f-b89beb7e5ac0:frag:0:0] ERROR
o.a.drill.exec.work.foreman.Foreman - Error
0b4d9b3a-d8af-4dc9-be47-46c4547a793a: RemoteRpcException: Failure while
running fragment., Schema change detected in the left input of Union-All.
This is not currently supported [ b9555eb8-c009-4e9c-b058-ffae3f015df7 on
centos-04.qa.lab:31010 ]
[ b9555eb8-c009-4e9c-b058-ffae3f015df7 on centos-04.qa.lab:31010 ]
org.apache.drill.exec.rpc.RemoteRpcException: Failure while running
fragment., Schema change detected in the left input of Union-All. This is not
currently supported [ b9555eb8-c009-4e9c-b058-ffae3f015df7 on
centos-04.qa.lab:31010 ]
[ b9555eb8-c009-4e9c-b058-ffae3f015df7 on centos-04.qa.lab:31010 ]
at
org.apache.drill.exec.work.foreman.QueryManager.statusUpdate(QueryManager.java:163)
[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.work.foreman.QueryManager$RootStatusReporter.statusChange(QueryManager.java:281)
[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.AbstractStatusReporter.fail(AbstractStatusReporter.java:114)
[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.AbstractStatusReporter.fail(AbstractStatusReporter.java:110)
[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.internalFail(FragmentExecutor.java:230)
[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:165)
[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_75]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_75]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75]
2015-03-31 20:10:07,825 [2ae500df-db85-2583-fa7f-b89beb7e5ac0:frag:0:0] WARN
o.a.d.e.w.fragment.FragmentExecutor - Error while initializing or executing
fragment
java.lang.RuntimeException: Error closing fragment
[jira] [Resolved] (DRILL-2591) Aggregate in left input to Union All does not work
[
https://issues.apache.org/jira/browse/DRILL-2591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Hsuan-Yi Chu resolved DRILL-2591.
--
Resolution: Fixed
Aggregate in left input to Union All does not work
--
Key: DRILL-2591
URL: https://issues.apache.org/jira/browse/DRILL-2591
Project: Apache Drill
Issue Type: Bug
Components: Query Planning Optimization
Affects Versions: 0.9.0
Environment: {code}
0: jdbc:drill: select * from sys.version;
+++-+-++
| commit_id | commit_message | commit_time | build_email | build_time |
+++-+-++
| 9d92b8e319f2d46e8659d903d355450e15946533 | DRILL-2580: Exit early from
HashJoinBatch if build side is empty | 26.03.2015 @ 16:13:53 EDT | Unknown
| 26.03.2015 @ 16:53:21 EDT |
+++-+-++
1 row selected (0.104 seconds)
{code}
Reporter: Khurram Faraaz
Assignee: Sean Hsuan-Yi Chu
Attachments: DRILL-2591.1.patch
If the left input to Union All has an aggregate function, the result is
SQLException. This was seen on a 4 node cluster.
{code}
0: jdbc:drill: select max(key) from `dateData.json` union all select key
from `timeStmpData.json`;
++
| EXPR$0 |
++
Query failed: Query stopped., Schema change detected in the left input of
Union-All. This is not currently supported [
441285d7-e4a5-46c8-ab11-a0332945e3fc on centos-04.qa.lab:31010 ]
java.lang.RuntimeException: java.sql.SQLException: Failure while executing
query.
at sqlline.SqlLine$IncrementalRows.hasNext(SqlLine.java:2514)
at sqlline.SqlLine$TableOutputFormat.print(SqlLine.java:2148)
at sqlline.SqlLine.print(SqlLine.java:1809)
at sqlline.SqlLine$Commands.execute(SqlLine.java:3766)
at sqlline.SqlLine$Commands.sql(SqlLine.java:3663)
at sqlline.SqlLine.dispatch(SqlLine.java:889)
at sqlline.SqlLine.begin(SqlLine.java:763)
at sqlline.SqlLine.start(SqlLine.java:498)
at sqlline.SqlLine.main(SqlLine.java:460)
{code}
Stack trace from drillbit.log
{code}
2015-03-27 00:29:09,795 [2aeb5baa-5af0-ac70-b49a-53e61c92be51:frag:0:0] ERROR
o.a.drill.exec.work.foreman.Foreman - Error
e3ad43f5-fda6-48e5-9e74-779c69bb3cb2: RemoteRpcException: Failure while
running fragment., Schema change detected in the left input of Union-All.
This is not currently supported [ c2c7add0-651b-44d8-9a7c-3218761098e4 on
centos-04.qa.lab:31010 ]
[ c2c7add0-651b-44d8-9a7c-3218761098e4 on centos-04.qa.lab:31010 ]
org.apache.drill.exec.rpc.RemoteRpcException: Failure while running
fragment., Schema change detected in the left input of Union-All. This is not
currently supported [ c2c7add0-651b-44d8-9a7c-3218761098e4 on
centos-04.qa.lab:31010 ]
[ c2c7add0-651b-44d8-9a7c-3218761098e4 on centos-04.qa.lab:31010 ]
at
org.apache.drill.exec.work.foreman.QueryManager.statusUpdate(QueryManager.java:163)
[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.work.foreman.QueryManager$RootStatusReporter.statusChange(QueryManager.java:281)
[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.AbstractStatusReporter.fail(AbstractStatusReporter.java:114)
[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.AbstractStatusReporter.fail(AbstractStatusReporter.java:110)
[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.internalFail(FragmentExecutor.java:230)
[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:165)
[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_75]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_75]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75]
2015-03-27 00:29:09,796 [2aeb5baa-5af0-ac70-b49a-53e61c92be51:frag:0:0] WARN
o.a.d.e.w.fragment.FragmentExecutor - Error while initializing or executing
fragment
java.lang.RuntimeException: Error closing fragment context.
at
org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources(FragmentExecutor.java:224)
[jira] [Resolved] (DRILL-708) TRUNC(n1) function returns a decimal instead of int
[ https://issues.apache.org/jira/browse/DRILL-708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Parth Chandra resolved DRILL-708. - Resolution: Won't Fix TRUNC(n1) function returns a decimal instead of int --- Key: DRILL-708 URL: https://issues.apache.org/jira/browse/DRILL-708 Project: Apache Drill Issue Type: Bug Components: Functions - Drill Affects Versions: m1 Reporter: Krystal Assignee: Yash Sharma Priority: Minor Fix For: 1.0.0 Attachments: DRILL-708.patch The TRUNC(n1) function should return the integer part of n1 as an int. In drill, however, the value returned is a decimal. For example, select trunc(15.79) from voter where voter_id=10; returns 15.0 The other database platforms return an int value of 15. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: drill-interpreter project errors
You can safely delete the module from your working directory. Thanks. Why didn't my rebasing of my branch on master remove the module? Daniel Jason Altekruse wrote: The interpreter module was removed in a recent commit to make the interpreter usable at planning time. The interpreter code was committed a few months ago, but only had unit tests written against it in its own module. To make the interpreter work with the other modules and hook it up to planning rules it needed to be refactored, the way we chose to refactor it actually removed the need for the separate module and build step. You can safely delete the module from your working directory. https://github.com/apache/drill/commit/0aa8b19d624d173da51de36aa164f3435d3366a4 -Jason Altekruse On Mon, Apr 6, 2015 at 1:20 PM, Daniel Barclay dbarc...@maprtech.com wrote: What's going on with the drill-interpreter module? For while now, in Eclipse it shows up as having errors (e.g., that org.apache.drill.BaseTestQuery is unknown), even after I issue the Maven - Update Project ... command on it. Daniel -- Daniel Barclay MapR Technologies -- Daniel Barclay MapR Technologies
[jira] [Resolved] (DRILL-2226) Create test utilities for checking plans for patterns
[ https://issues.apache.org/jira/browse/DRILL-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Altekruse resolved DRILL-2226. Resolution: Fixed Fix Version/s: (was: 1.0.0) 0.8.0 Fixed in ed397862eb9584572aa0fcb684dfc9554b00cf60 Create test utilities for checking plans for patterns - Key: DRILL-2226 URL: https://issues.apache.org/jira/browse/DRILL-2226 Project: Apache Drill Issue Type: Improvement Components: Tools, Build Test Reporter: Jason Altekruse Assignee: Jason Altekruse Fix For: 0.8.0 Attachments: DRILL-2226.patch Regex matching for calcite text format plans, includes expected and excluded pattern matching. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: drill-interpreter project errors
The interpreter module was removed in a recent commit to make the interpreter usable at planning time. The interpreter code was committed a few months ago, but only had unit tests written against it in its own module. To make the interpreter work with the other modules and hook it up to planning rules it needed to be refactored, the way we chose to refactor it actually removed the need for the separate module and build step. You can safely delete the module from your working directory. https://github.com/apache/drill/commit/0aa8b19d624d173da51de36aa164f3435d3366a4 -Jason Altekruse On Mon, Apr 6, 2015 at 1:20 PM, Daniel Barclay dbarc...@maprtech.com wrote: What's going on with the drill-interpreter module? For while now, in Eclipse it shows up as having errors (e.g., that org.apache.drill.BaseTestQuery is unknown), even after I issue the Maven - Update Project ... command on it. Daniel -- Daniel Barclay MapR Technologies
Review Request 32904: SQL scalar functions docs
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32904/ --- Review request for drill and Bridget Bridget. Bugs: DRILL-2681 https://issues.apache.org/jira/browse/DRILL-2681 Repository: drill-git Description --- SQL scalar functions docs, maprdb plugin doc fixes, time zone limitations section, Query HBase doc fixes, fix broken links in contribution guidelines and contribution ideas, prelim. overview of config options Diffs - _docs/connect/009-mapr-db-plugin.md a0582f3 _docs/develop/contribute/001-guidelines.md 7782f90 _docs/develop/contribute/002-ideas.md 052d3e7 _docs/develop/develop-drill/001-compile.md dea42e9 _docs/manage/conf/001-mem-alloc.md 4caf563 _docs/query/003-query-hbase.md e6e70fa _docs/sql-ref/001-data-types.md cf353c7 _docs/sql-ref/002-lexical-structure.md 731b2b3 _docs/sql-ref/003-operators.md 074375a _docs/sql-ref/004-functions.md e444268 _docs/sql-ref/005-nest-functions.md c6e7ff2 _docs/sql-ref/data-types/001-date.md e08873b _docs/sql-ref/data-types/002-diff-data-types.md 229fc85 _docs/sql-ref/functions/002-conversion.md PRE-CREATION _docs/sql-ref/functions/002-data-type-fmt.md 87f4a4c _docs/sql-ref/functions/003-date-time-fcns.md 6815484 _docs/sql-ref/functions/004-string.md PRE-CREATION _docs/sql-ref/functions/005-aggregate.md PRE-CREATION _docs/sql-ref/functions/006-nulls.md PRE-CREATION Diff: https://reviews.apache.org/r/32904/diff/ Testing --- Thanks, Kristine Hahn
[jira] [Created] (DRILL-2704) Schema change not reported in Union All query
Khurram Faraaz created DRILL-2704:
-
Summary: Schema change not reported in Union All query
Key: DRILL-2704
URL: https://issues.apache.org/jira/browse/DRILL-2704
Project: Apache Drill
Issue Type: Bug
Components: Query Planning Optimization
Affects Versions: 0.9.0
Environment: 4 node cluster on CentOS
Reporter: Khurram Faraaz
Assignee: Jinfeng Ni
Schema change is not reported in Union All query. Inputs to Union All read from
CSV file.
{code}
0: jdbc:drill: select columns[0] from `f1.csv` where columns[0] = 13 union all
select columns[0] from `f2.csv` where columns[0] = 10;
Query failed: RemoteRpcException: Failure while running fragment., 9.99 [
a80303f5-781c-4288-875e-90ad60c8caac on centos-02.qa.lab:31010 ]
[ a80303f5-781c-4288-875e-90ad60c8caac on centos-02.qa.lab:31010 ]
Error: exception while executing query: Failure while executing query.
(state=,code=0)
0: jdbc:drill: select * from `f1.csv`;
++
| columns |
++
| [1] |
| [2] |
| [55] |
| [77] |
| [-100] |
| [13] |
| [17] |
| [19] |
| [2] |
| [1000] |
| [99] |
| [9.99] |
| [12345.987] |
| [Hello World] |
| [this is a test string] |
| [foobar] |
| [] |
| [new string] |
| [34534534] |
| [345345] |
| [2321312] |
| [67867867] |
| [112319089] |
| [-14695967] |
| [100]|
| [99] |
| [7] |
| [5] |
| [13] |
| [] |
| [] |
| [] |
| [0] |
| [1] |
| [-1] |
| null |
| null |
| null |
| null |
++
39 rows selected (0.16 seconds)
0: jdbc:drill: select * from `f2.csv`;
++
| columns |
++
| [1] |
| [0] |
| [-1] |
| [99] |
| [1000] |
| [10] |
| [-13]|
| [-17]|
| [19] |
| [17] |
| [13] |
| [] |
| [no way] |
| [good quality] |
| [] |
| [done!] |
| [Yay!!!] |
| null |
++
18 rows selected (0.128 seconds)
Casting columns[0] to varchar gives correct results
0: jdbc:drill: select cast(columns[0] as varchar(100)) from `f1.csv` where
columns[0] = '13' union all select cast(columns[0] as varchar(100)) from
`f2.csv` where columns[0] = '10';
++
| EXPR$0 |
++
| 13 |
| 13 |
| 10 |
++
3 rows selected (0.474 seconds)
Stack trace from drillbit.log for failing query
2015-04-06 23:31:43,674 [2adce8a0-24c8-58ad-2f46-30c42a33ef2a:frag:0:0] ERROR
o.a.d.e.w.f.AbstractStatusReporter - Error
a4ec50e8-88ea-4474-9a36-b2250f34e340: Failure while running fragment.
java.lang.NumberFormatException: 9.99
at
org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.nfeI(StringFunctionHelpers.java:96)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.varCharToInt(StringFunctionHelpers.java:121)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.test.generated.FiltererGen37.doEval(FilterTemplate2.java:42)
~[na:na]
at
org.apache.drill.exec.test.generated.FiltererGen37.filterBatchNoSV(FilterTemplate2.java:98)
~[na:na]
at
org.apache.drill.exec.test.generated.FiltererGen37.filterBatch(FilterTemplate2.java:71)
~[na:na]
at
org.apache.drill.exec.physical.impl.filter.FilterRecordBatch.doWork(FilterRecordBatch.java:82)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:93)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:142)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:118)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:99)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:89)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext(RemovingRecordBatch.java:96)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:142)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
Re: unit test failure
@Jason and those who had successful runs, how many forks did you guys run with? -Hanifi On Mon, Apr 6, 2015 at 3:37 PM, Jason Altekruse altekruseja...@gmail.com wrote: I just had a successful run on my linux machine and two runs on my mac. The logs from my linux machine do not show the stacktrace or error message, but searching the output from my mac I do see it, on the same test. - Jason Altekruse On Mon, Apr 6, 2015 at 2:57 PM, Parth Chandra pchan...@maprtech.com wrote: I'm seeing the following in the Unit tests (the test seem to be passing though). Anyone else see these or is it just my branch? *Running org.apache.drill.TestUnionAll#testUnionAllImplicitCastingFailure* key 52459253098448904 1116675951 Exception (no rows returned): org.apache.drill.exec.rpc.RpcException: RemoteRpcException: Failure while running fragment., Failure while trying to materialize incoming schema. Errors: Error in expression at index -1. Error: Missing function implementation: [castBIGINT(BIT-OPTIONAL)]. Full expression: --UNKNOWN EXPRESSION--.. [ ac878d4f-942d-4597-bd01-d55623a0abe0 on localhost:31022 ] [ ac878d4f-942d-4597-bd01-d55623a0abe0 on localhost:31022 ] *Running org.apache.drill.TestUnionAll#testDistinctOverUnionAllwithFullyQualifiedColumnNames* 13:30:50.846 [WorkManager-2] ERROR o.a.z.server.NIOServerCnxnFactory - Thread Thread[WorkManager-2,10,main] died java.lang.IllegalStateException: null at com.google.common.base.Preconditions.checkState(Preconditions.java:133) ~[guava-14.0.1.jar:na] at org.apache.drill.common.DeferredException.addException(DeferredException.java:47) ~[classes/:na] at org.apache.drill.common.DeferredException.addThrowable(DeferredException.java:61) ~[classes/:na] at org.apache.drill.exec.ops.FragmentContext.fail(FragmentContext.java:135) ~[classes/:na] at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:181) ~[classes/:na] at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) ~[classes/:na] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_65] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_65] at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_65]
[jira] [Resolved] (DRILL-2681) add sql functions docs (string, convert_to/from)
[ https://issues.apache.org/jira/browse/DRILL-2681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kristine Hahn resolved DRILL-2681. -- Resolution: Fixed add sql functions docs (string, convert_to/from) Key: DRILL-2681 URL: https://issues.apache.org/jira/browse/DRILL-2681 Project: Apache Drill Issue Type: Task Components: Documentation Reporter: Kristine Hahn Assignee: Kristine Hahn Includes SQL scalar functions, maprdb plugin doc fixes, time zone limitations section, Query HBase doc fixes, fix broken links in contribution guidelines and contribution ideas, prelim. overview of config options -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (DRILL-1840) Cloning the source repository as documented in the wiki
[
https://issues.apache.org/jira/browse/DRILL-1840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kristine Hahn resolved DRILL-1840.
--
Resolution: Fixed
Sorry for the delay. Thanks for letting us know about the broken links.
We fixed the links in the docs at the new site http://drill.apache.org/docs/.
We're moving away from the wiki docs.
The corrected links are in these sections:
* http://drill.apache.org/docs/apache-drill-contribution-guidelines/
* http://drill.apache.org/docs/apache-drill-contribution-ideas/
Thanks!
Cloning the source repository as documented in the wiki
---
Key: DRILL-1840
URL: https://issues.apache.org/jira/browse/DRILL-1840
Project: Apache Drill
Issue Type: Improvement
Components: Documentation
Environment: wiki
Reporter: Karl Heinz Marbaise
Assignee: Kristine Hahn
Fix For: Future
On the following page
https://cwiki.apache.org/confluence/display/DRILL/Compiling+Drill+from+Source
it is mentioned to clone the following repository:
{{git clone https://git-wip-us.apache.org/repos/asf/incubator-drill.git}}
which is the meantime not possible cause the repository name has changed
based on the migration from incubator to tlp...
{{git clone https://git-wip-us.apache.org/repos/asf/drill.git}}
Unfortunately i haven't the karma to change the wiki nor to login into the
wicki so i couldn' fix this issue..
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: unit test failure
On both the linux VM and my mac I ran with a fork count of 2. On Mon, Apr 6, 2015 at 3:43 PM, Hanifi Gunes hgu...@maprtech.com wrote: @Jason and those who had successful runs, how many forks did you guys run with? -Hanifi On Mon, Apr 6, 2015 at 3:37 PM, Jason Altekruse altekruseja...@gmail.com wrote: I just had a successful run on my linux machine and two runs on my mac. The logs from my linux machine do not show the stacktrace or error message, but searching the output from my mac I do see it, on the same test. - Jason Altekruse On Mon, Apr 6, 2015 at 2:57 PM, Parth Chandra pchan...@maprtech.com wrote: I'm seeing the following in the Unit tests (the test seem to be passing though). Anyone else see these or is it just my branch? *Running org.apache.drill.TestUnionAll#testUnionAllImplicitCastingFailure* key 52459253098448904 1116675951 Exception (no rows returned): org.apache.drill.exec.rpc.RpcException: RemoteRpcException: Failure while running fragment., Failure while trying to materialize incoming schema. Errors: Error in expression at index -1. Error: Missing function implementation: [castBIGINT(BIT-OPTIONAL)]. Full expression: --UNKNOWN EXPRESSION--.. [ ac878d4f-942d-4597-bd01-d55623a0abe0 on localhost:31022 ] [ ac878d4f-942d-4597-bd01-d55623a0abe0 on localhost:31022 ] *Running org.apache.drill.TestUnionAll#testDistinctOverUnionAllwithFullyQualifiedColumnNames* 13:30:50.846 [WorkManager-2] ERROR o.a.z.server.NIOServerCnxnFactory - Thread Thread[WorkManager-2,10,main] died java.lang.IllegalStateException: null at com.google.common.base.Preconditions.checkState(Preconditions.java:133) ~[guava-14.0.1.jar:na] at org.apache.drill.common.DeferredException.addException(DeferredException.java:47) ~[classes/:na] at org.apache.drill.common.DeferredException.addThrowable(DeferredException.java:61) ~[classes/:na] at org.apache.drill.exec.ops.FragmentContext.fail(FragmentContext.java:135) ~[classes/:na] at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:181) ~[classes/:na] at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) ~[classes/:na] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_65] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_65] at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_65]
Regarding unit test failures -- patterns possible fixes
Hey devs, I have been dealing with sporadic unit test failures since last week. I'd like to share my findings on failing unit tests outlining the patterns causing these. The general advice I was given was to run tests lowering the fork count (the default is 4). This sounds ok to get around the problem. If this is your intent feel free to give a try. However, I wanted to investigate this further, went ahead and set fork count to 8, number of cores testing machine has. The first set of failures manifested around TestUnionAll TestExampleQueries. After Hakeem's pointer, I noticed that some test cases in TestUnionAll TestExampleQueries creates drops views with the same name. In a concurrent settings(given fork count 0) with no strict ordering among test cases, this sure creates a havoc throwing arbitrary errors at each test run. https://issues.apache.org/jira/browse/DRILL-2684 takes care of this. After applying this fix, now I can cleanly run java-exec tests, faster consistently. Then my hive tests started failing since multiple test classes use the same metadata folders and clean up after without caring whether other hive tests are still running. I got a patch to unique-ify metadata folders for test class. https://issues.apache.org/jira/browse/DRILL-2685 tracks this however the patch is not public yet. The last problem relates to hive related tests living under jdbc module. I know that there has been an ongoing effort to migrate these tests from jdbc module to where they belong. I am not sure about the timeframe for the migration but I would think the sooner is better. To avoid further failures, I would strongly recommend devs to use view names and external resources that are unique across test cases. One idea is to suffix view a view name with test case/class name for instance prefer using names_view_test_union_all rather than a more generic names_view. Also with increasing number of test cases checked in, it takes more and more time to complete a test run. We should consider concurrent test runs as a legitimate everyday use case and design tests accordingly. I would be interested in hearing other unit test failure patterns so as to be alert. Feel free to share if you discovered any. Regards. -Hanifi
Fwd: unit test failure
I'm seeing the following in the Unit tests (the test seem to be passing though). Anyone else see these or is it just my branch? *Running org.apache.drill.TestUnionAll#testUnionAllImplicitCastingFailure* key 52459253098448904 1116675951 Exception (no rows returned): org.apache.drill.exec.rpc.RpcException: RemoteRpcException: Failure while running fragment., Failure while trying to materialize incoming schema. Errors: Error in expression at index -1. Error: Missing function implementation: [castBIGINT(BIT-OPTIONAL)]. Full expression: --UNKNOWN EXPRESSION--.. [ ac878d4f-942d-4597-bd01-d55623a0abe0 on localhost:31022 ] [ ac878d4f-942d-4597-bd01-d55623a0abe0 on localhost:31022 ] *Running org.apache.drill.TestUnionAll#testDistinctOverUnionAllwithFullyQualifiedColumnNames* 13:30:50.846 [WorkManager-2] ERROR o.a.z.server.NIOServerCnxnFactory - Thread Thread[WorkManager-2,10,main] died java.lang.IllegalStateException: null at com.google.common.base.Preconditions.checkState(Preconditions.java:133) ~[guava-14.0.1.jar:na] at org.apache.drill.common.DeferredException.addException(DeferredException.java:47) ~[classes/:na] at org.apache.drill.common.DeferredException.addThrowable(DeferredException.java:61) ~[classes/:na] at org.apache.drill.exec.ops.FragmentContext.fail(FragmentContext.java:135) ~[classes/:na] at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:181) ~[classes/:na] at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) ~[classes/:na] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_65] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_65] at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_65]
[jira] [Created] (DRILL-2700) Missing/deleted drillbit.log files on cluster are not reported
Khurram Faraaz created DRILL-2700:
-
Summary: Missing/deleted drillbit.log files on cluster are not
reported
Key: DRILL-2700
URL: https://issues.apache.org/jira/browse/DRILL-2700
Project: Apache Drill
Issue Type: Bug
Components: Tools, Build Test
Affects Versions: 0.9.0
Environment: 4 node cluster on CentOS
Reporter: Khurram Faraaz
Assignee: Steven Phillips
Missing drillbit.log files on cluster, under /opt/mapr/drill/drill-0.9.0/logs/
are not reported as missing. Instead, user is able to issue and execute SQL
queries from sqlline prompt. Failing queries are also executed and
errors/Exceptions are reported on sqlline prompt, but there is no corresponding
drillbit.log file where details must be written to in the case of
passing/failing queries.
I have manually deleted the drillbit.log (users can accidentally delete
drillbit.log), and continue to execute SQL from sqlline prompt, and in the case
of failing SQL there will be no drillbit.log file since it was accidentally
deleted, and hence stacktrace and other diagnostic information is lost.
{code}
[root@centos-01 bin]# clush -g khurram service mapr-warden stop
10.10.100.201: stopping WARDEN
10.10.100.202: stopping WARDEN
10.10.100.203: stopping WARDEN
10.10.100.204: stopping WARDEN
10.10.100.202: looking to stop mapr-core processes not started by warden
10.10.100.203: looking to stop mapr-core processes not started by warden
10.10.100.204: looking to stop mapr-core processes not started by warden
10.10.100.201: looking to stop mapr-core processes not started by warden
[root@centos-01 bin]# clush -g khurram service mapr-warden start
10.10.100.204: Starting WARDEN, logging to /opt/mapr/logs/warden.log.
10.10.100.203: Starting WARDEN, logging to /opt/mapr/logs/warden.log.
10.10.100.202: Starting WARDEN, logging to /opt/mapr/logs/warden.log.
10.10.100.201: Starting WARDEN, logging to /opt/mapr/logs/warden.log.
10.10.100.204: .
10.10.100.203: .
10.10.100.204: For diagnostics look at /opt/mapr/logs/ for
createsystemvolumes.log, warden.log and configured services log files
10.10.100.202: .
10.10.100.201: .
10.10.100.203: For diagnostics look at /opt/mapr/logs/ for
createsystemvolumes.log, warden.log and configured services log files
10.10.100.202: For diagnostics look at /opt/mapr/logs/ for
createsystemvolumes.log, warden.log and configured services log files
10.10.100.201: For diagnostics look at /opt/mapr/logs/ for
createsystemvolumes.log, warden.log and configured services log files
[root@centos-01 bin]# clush -g khurram ls -lrt
/opt/mapr/drill/drill-0.9.0/logs/drillbit.log
10.10.100.201: -rw-r--r-- 1 mapr mapr 17522 Apr 6 20:49
/opt/mapr/drill/drill-0.9.0/logs/drillbit.log
10.10.100.204: -rw-r--r-- 1 mapr mapr 17522 Apr 6 20:49
/opt/mapr/drill/drill-0.9.0/logs/drillbit.log
10.10.100.202: -rw-r--r-- 1 mapr mapr 17522 Apr 6 20:49
/opt/mapr/drill/drill-0.9.0/logs/drillbit.log
10.10.100.203: -rw-r--r-- 1 mapr mapr 17522 Apr 6 20:49
/opt/mapr/drill/drill-0.9.0/logs/drillbit.log
Manually remove drillbit.log files from cluster nodes
[root@centos-01 bin]# clush -g khurram rm -rf
/opt/mapr/drill/drill-0.9.0/logs/drillbit.log
[root@centos-01 bin]# clush -g khurram ls -lrt
/opt/mapr/drill/drill-0.9.0/logs/drillbit.log
10.10.100.201: ls: cannot access /opt/mapr/drill/drill-0.9.0/logs/drillbit.log:
No such file or directory
clush: 10.10.100.201: exited with exit code 2
10.10.100.204: ls: cannot access /opt/mapr/drill/drill-0.9.0/logs/drillbit.log:
No such file or directory
clush: 10.10.100.204: exited with exit code 2
10.10.100.203: ls: cannot access /opt/mapr/drill/drill-0.9.0/logs/drillbit.log:
No such file or directory
clush: 10.10.100.203: exited with exit code 2
10.10.100.202: ls: cannot access /opt/mapr/drill/drill-0.9.0/logs/drillbit.log:
No such file or directory
clush: 10.10.100.202: exited with exit code 2
[root@centos-01 bin]# ./sqlline -u jdbc:drill:
Drill log directory: /opt/mapr/drill/drill-0.9.0/logs
sqlline version 1.1.6
0: jdbc:drill: use sys;
+++
| ok | summary |
+++
| true | Default schema changed to 'sys' |
+++
1 row selected (1.199 seconds)
0: jdbc:drill: select * from sys.version;
+++-+-++
| commit_id | commit_message | commit_time | build_email | build_time |
+++-+-++
| 9d92b8e319f2d46e8659d903d355450e15946533 | DRILL-2580: Exit early from
HashJoinBatch if build side is empty | 26.03.2015 @ 16:13:53 EDT | Unknown
| 26.03.2015 @ 16:59:31 EDT |
+++-+-++
1 row selected (0.608 seconds)
0: jdbc:drill: select * from sys.drillbits;
Re: unit test failure
I just had a successful run on my linux machine and two runs on my mac. The logs from my linux machine do not show the stacktrace or error message, but searching the output from my mac I do see it, on the same test. - Jason Altekruse On Mon, Apr 6, 2015 at 2:57 PM, Parth Chandra pchan...@maprtech.com wrote: I'm seeing the following in the Unit tests (the test seem to be passing though). Anyone else see these or is it just my branch? *Running org.apache.drill.TestUnionAll#testUnionAllImplicitCastingFailure* key 52459253098448904 1116675951 Exception (no rows returned): org.apache.drill.exec.rpc.RpcException: RemoteRpcException: Failure while running fragment., Failure while trying to materialize incoming schema. Errors: Error in expression at index -1. Error: Missing function implementation: [castBIGINT(BIT-OPTIONAL)]. Full expression: --UNKNOWN EXPRESSION--.. [ ac878d4f-942d-4597-bd01-d55623a0abe0 on localhost:31022 ] [ ac878d4f-942d-4597-bd01-d55623a0abe0 on localhost:31022 ] *Running org.apache.drill.TestUnionAll#testDistinctOverUnionAllwithFullyQualifiedColumnNames* 13:30:50.846 [WorkManager-2] ERROR o.a.z.server.NIOServerCnxnFactory - Thread Thread[WorkManager-2,10,main] died java.lang.IllegalStateException: null at com.google.common.base.Preconditions.checkState(Preconditions.java:133) ~[guava-14.0.1.jar:na] at org.apache.drill.common.DeferredException.addException(DeferredException.java:47) ~[classes/:na] at org.apache.drill.common.DeferredException.addThrowable(DeferredException.java:61) ~[classes/:na] at org.apache.drill.exec.ops.FragmentContext.fail(FragmentContext.java:135) ~[classes/:na] at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:181) ~[classes/:na] at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) ~[classes/:na] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_65] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_65] at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_65]
Re: unit test failure
Hi, Actually, that unit test was written by me. The failure was the expected behavior. The annotation = @Test(expected = RpcException.class) On Mon, Apr 6, 2015 at 3:37 PM, Jason Altekruse altekruseja...@gmail.com wrote: I just had a successful run on my linux machine and two runs on my mac. The logs from my linux machine do not show the stacktrace or error message, but searching the output from my mac I do see it, on the same test. - Jason Altekruse On Mon, Apr 6, 2015 at 2:57 PM, Parth Chandra pchan...@maprtech.com wrote: I'm seeing the following in the Unit tests (the test seem to be passing though). Anyone else see these or is it just my branch? *Running org.apache.drill.TestUnionAll#testUnionAllImplicitCastingFailure* key 52459253098448904 1116675951 Exception (no rows returned): org.apache.drill.exec.rpc.RpcException: RemoteRpcException: Failure while running fragment., Failure while trying to materialize incoming schema. Errors: Error in expression at index -1. Error: Missing function implementation: [castBIGINT(BIT-OPTIONAL)]. Full expression: --UNKNOWN EXPRESSION--.. [ ac878d4f-942d-4597-bd01-d55623a0abe0 on localhost:31022 ] [ ac878d4f-942d-4597-bd01-d55623a0abe0 on localhost:31022 ] *Running org.apache.drill.TestUnionAll#testDistinctOverUnionAllwithFullyQualifiedColumnNames* 13:30:50.846 [WorkManager-2] ERROR o.a.z.server.NIOServerCnxnFactory - Thread Thread[WorkManager-2,10,main] died java.lang.IllegalStateException: null at com.google.common.base.Preconditions.checkState(Preconditions.java:133) ~[guava-14.0.1.jar:na] at org.apache.drill.common.DeferredException.addException(DeferredException.java:47) ~[classes/:na] at org.apache.drill.common.DeferredException.addThrowable(DeferredException.java:61) ~[classes/:na] at org.apache.drill.exec.ops.FragmentContext.fail(FragmentContext.java:135) ~[classes/:na] at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:181) ~[classes/:na] at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) ~[classes/:na] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_65] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_65] at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_65]
Review Request 32910: minor doc changes
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32910/ --- Review request for drill and Bridget Bridget. Bugs: DRILL-2701 https://issues.apache.org/jira/browse/DRILL-2701 Repository: drill-git Description --- minor doc changes Diffs - _docs/connect/009-mapr-db-plugin.md 62aa068 _docs/query/003-query-hbase.md bdfbb8a _docs/sql-ref/001-data-types.md 5607c85 _docs/sql-ref/003-operators.md b9a95e7 _docs/sql-ref/data-types/001-date.md ec97e3b _docs/sql-ref/functions/002-conversion.md 64fc25c _docs/sql-ref/functions/005-aggregate.md 250527f _docs/sql-ref/functions/006-nulls.md 9041e49 Diff: https://reviews.apache.org/r/32910/diff/ Testing --- Thanks, Kristine Hahn
[jira] [Created] (DRILL-2701) configuration options
Kristine Hahn created DRILL-2701: Summary: configuration options Key: DRILL-2701 URL: https://issues.apache.org/jira/browse/DRILL-2701 Project: Apache Drill Issue Type: Task Components: Documentation Reporter: Kristine Hahn Assignee: Kristine Hahn -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Regarding unit test failures -- patterns possible fixes
From my personal experience, using a forkCount of 4 makes some tests to timeout, not because there are some concurrent problems but just because my machine is not powerful enough to run as many tests in parallel. On Mon, Apr 6, 2015 at 5:06 PM, Hanifi Gunes hgu...@maprtech.com wrote: Hey devs, I have been dealing with sporadic unit test failures since last week. I'd like to share my findings on failing unit tests outlining the patterns causing these. The general advice I was given was to run tests lowering the fork count (the default is 4). This sounds ok to get around the problem. If this is your intent feel free to give a try. However, I wanted to investigate this further, went ahead and set fork count to 8, number of cores testing machine has. The first set of failures manifested around TestUnionAll TestExampleQueries. After Hakeem's pointer, I noticed that some test cases in TestUnionAll TestExampleQueries creates drops views with the same name. In a concurrent settings(given fork count 0) with no strict ordering among test cases, this sure creates a havoc throwing arbitrary errors at each test run. https://issues.apache.org/jira/browse/DRILL-2684 takes care of this. After applying this fix, now I can cleanly run java-exec tests, faster consistently. Then my hive tests started failing since multiple test classes use the same metadata folders and clean up after without caring whether other hive tests are still running. I got a patch to unique-ify metadata folders for test class. https://issues.apache.org/jira/browse/DRILL-2685 tracks this however the patch is not public yet. The last problem relates to hive related tests living under jdbc module. I know that there has been an ongoing effort to migrate these tests from jdbc module to where they belong. I am not sure about the timeframe for the migration but I would think the sooner is better. To avoid further failures, I would strongly recommend devs to use view names and external resources that are unique across test cases. One idea is to suffix view a view name with test case/class name for instance prefer using names_view_test_union_all rather than a more generic names_view. Also with increasing number of test cases checked in, it takes more and more time to complete a test run. We should consider concurrent test runs as a legitimate everyday use case and design tests accordingly. I would be interested in hearing other unit test failure patterns so as to be alert. Feel free to share if you discovered any. Regards. -Hanifi -- Abdelhakim Deneche Software Engineer http://www.mapr.com/ Now Available - Free Hadoop On-Demand Training http://www.mapr.com/training?utm_source=Emailutm_medium=Signatureutm_campaign=Free%20available
Re: Review Request 32494: DRILL-2565: Add AlreadyClosedSqlException, some key checks, and test for future DRILL-2489 fix.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32494/#review79105 --- Ship it! Sqlline does not seem to be an issue. - Parth Chandra On March 31, 2015, 9:32 p.m., Daniel Barclay wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32494/ --- (Updated March 31, 2015, 9:32 p.m.) Review request for drill, Mehant Baid and Parth Chandra. Bugs: DRILL-2565 https://issues.apache.org/jira/browse/DRILL-2565 Repository: drill-git Description --- - Created AlreadyClosedSqlException. - (Moved JdbcApiSqlException to be subclass of SQLNonTransientException.) - Created test Drill2489CallsAfterCloseThrowExceptionsTest for eventual fixing of DRILL-2489. - (Is partial: Covers Connection, Statement, and ResultSet.) - (Is interim: Most methods' test disabled with @Ignore.) - Added already-closed checking in key places, especially those that involve communication and could hang for a while rather than dying quickly (e.g., Statement.execute...(...)). Diffs - exec/jdbc/src/main/java/org/apache/drill/jdbc/AlreadyClosedSqlException.java PRE-CREATION exec/jdbc/src/main/java/org/apache/drill/jdbc/DrillConnectionImpl.java e590778 exec/jdbc/src/main/java/org/apache/drill/jdbc/DrillDatabaseMetaData.java ca1648d exec/jdbc/src/main/java/org/apache/drill/jdbc/DrillResultSet.java 0ce33f4 exec/jdbc/src/main/java/org/apache/drill/jdbc/DrillStatement.java d934c7c exec/jdbc/src/main/java/org/apache/drill/jdbc/InvalidCursorStateSqlException.java 7b04371 exec/jdbc/src/main/java/org/apache/drill/jdbc/JdbcApiSqlException.java d6b05fb exec/jdbc/src/test/java/org/apache/drill/jdbc/test/Drill2489CallsAfterCloseThrowExceptionsTest.java PRE-CREATION Diff: https://reviews.apache.org/r/32494/diff/ Testing --- Ran new specific tests. Run existing tests; no new problems. Thanks, Daniel Barclay
Re: Review Request 32800: C++ Client Authentication API passing std::string across DLL boundaries
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32800/#review79116 --- Ship it! Ship It! - Parth Chandra On April 6, 2015, 6:30 p.m., Norris Lee wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32800/ --- (Updated April 6, 2015, 6:30 p.m.) Review request for drill, Alexander zarei, Parth Chandra, and Xiao Meng. Bugs: DRILL-2671 https://issues.apache.org/jira/browse/DRILL-2671 Repository: drill-git Description --- DrillUserProperty::setProperty is taking std::string as parameters. Memory gets allocated in the client yet Drill Client tries to clean it up. Diffs - contrib/native/client/src/include/drill/drillClient.hpp 9289df3 Diff: https://reviews.apache.org/r/32800/diff/ Testing --- Thanks, Norris Lee
Re: Regarding unit test failures -- patterns possible fixes
I have just seen Jacques' comment under the patch. I am not sure whether we should set forks to a fraction of available cores. Any ideas? On Mon, Apr 6, 2015 at 5:52 PM, Hanifi Gunes hgu...@maprtech.com wrote: @Ted, I actually have a patch pending review for this at https://issues.apache.org/jira/browse/DRILL-2039. However, @Hakeem you may still end up in a troubling situation as your mac might have 4 logical cores =) *sysctl hw | grep **cpu* should tell you. -Hanifi On Mon, Apr 6, 2015 at 5:22 PM, Ted Dunning ted.dunn...@gmail.com wrote: Maven allows the forkCount to be set in terms of the number of cores on the machine by appending a C to the number. (I think) On Mon, Apr 6, 2015 at 5:15 PM, Abdel Hakim Deneche adene...@maprtech.com wrote: From my personal experience, using a forkCount of 4 makes some tests to timeout, not because there are some concurrent problems but just because my machine is not powerful enough to run as many tests in parallel. On Mon, Apr 6, 2015 at 5:06 PM, Hanifi Gunes hgu...@maprtech.com wrote: Hey devs, I have been dealing with sporadic unit test failures since last week. I'd like to share my findings on failing unit tests outlining the patterns causing these. The general advice I was given was to run tests lowering the fork count (the default is 4). This sounds ok to get around the problem. If this is your intent feel free to give a try. However, I wanted to investigate this further, went ahead and set fork count to 8, number of cores testing machine has. The first set of failures manifested around TestUnionAll TestExampleQueries. After Hakeem's pointer, I noticed that some test cases in TestUnionAll TestExampleQueries creates drops views with the same name. In a concurrent settings(given fork count 0) with no strict ordering among test cases, this sure creates a havoc throwing arbitrary errors at each test run. https://issues.apache.org/jira/browse/DRILL-2684 takes care of this. After applying this fix, now I can cleanly run java-exec tests, faster consistently. Then my hive tests started failing since multiple test classes use the same metadata folders and clean up after without caring whether other hive tests are still running. I got a patch to unique-ify metadata folders for test class. https://issues.apache.org/jira/browse/DRILL-2685 tracks this however the patch is not public yet. The last problem relates to hive related tests living under jdbc module. I know that there has been an ongoing effort to migrate these tests from jdbc module to where they belong. I am not sure about the timeframe for the migration but I would think the sooner is better. To avoid further failures, I would strongly recommend devs to use view names and external resources that are unique across test cases. One idea is to suffix view a view name with test case/class name for instance prefer using names_view_test_union_all rather than a more generic names_view. Also with increasing number of test cases checked in, it takes more and more time to complete a test run. We should consider concurrent test runs as a legitimate everyday use case and design tests accordingly. I would be interested in hearing other unit test failure patterns so as to be alert. Feel free to share if you discovered any. Regards. -Hanifi -- Abdelhakim Deneche Software Engineer http://www.mapr.com/ Now Available - Free Hadoop On-Demand Training http://www.mapr.com/training?utm_source=Emailutm_medium=Signatureutm_campaign=Free%20available
[jira] [Resolved] (DRILL-2646) CONVERT_TO/CONVERT_FROM syntax (arguments) not specified
[ https://issues.apache.org/jira/browse/DRILL-2646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kristine Hahn resolved DRILL-2646. -- Resolution: Fixed CONVERT_TO/CONVERT_FROM syntax (arguments) not specified Key: DRILL-2646 URL: https://issues.apache.org/jira/browse/DRILL-2646 Project: Apache Drill Issue Type: Bug Components: Documentation Reporter: Daniel Barclay (Drill) Assignee: Kristine Hahn On the SQL Functions page at http://drill.apache.org/docs/sql-functions/, the section on the CONVERT_TO and CONVERT_FROM functions never specifies the syntax (doesn't specify how many parameters each takes, which parameter is which) or semantics (including what the parameters are). Additionally, the name Type for the first column is confusing, since the values in that column do not seem to be Drill SQL types. Would something roughly like type specifier or conversion specifier be more accurate? Also, the connection between the values in the Type column and the functions isn't clear. Besides their being no specification of the parameters (to say where a value from the Type would go), the example at the end of the section doesn't use any of the values in the Type column (or, in fact, any value in the whole table). (If the string literal 'JSON' in that CONVERT_FROM is an example of the kind of thing listed in the Type column, then why isn't JSON listed in that column (and what else is missing)? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (DRILL-2707) Projecting a varchar column from a parquet file after a full outer join with a json file results in IOOBException
Rahul Challapalli created DRILL-2707:
Summary: Projecting a varchar column from a parquet file after a
full outer join with a json file results in IOOBException
Key: DRILL-2707
URL: https://issues.apache.org/jira/browse/DRILL-2707
Project: Apache Drill
Issue Type: Bug
Components: Execution - Relational Operators
Reporter: Rahul Challapalli
Assignee: Chris Westin
Priority: Critical
git.commit.id.abbrev=a53e123
I did a full outer join between a parquet file and a json file and projected a
varchar column. Below is what I see
{code}
0: jdbc:drill:schema=dfs_eea select
. . . . . . . . . . . . . . p.varchar_col
. . . . . . . . . . . . . . from dfs.`cross-sources`.`fewtypes.parquet` p
. . . . . . . . . . . . . . full outer join
dfs.`cross-sources`.`fewtypes_null.json` o
. . . . . . . . . . . . . . on p.int_col=o.int_col;
+-+
| varchar_col |
+-+
java.lang.IndexOutOfBoundsException: index: 180, length: 10 (expected: range(0,
180))
at io.netty.buffer.AbstractByteBuf.checkIndex(AbstractByteBuf.java:1143)
at
io.netty.buffer.PooledUnsafeDirectByteBuf.getBytes(PooledUnsafeDirectByteBuf.java:136)
at io.netty.buffer.WrappedByteBuf.getBytes(WrappedByteBuf.java:289)
at
io.netty.buffer.UnsafeDirectLittleEndian.getBytes(UnsafeDirectLittleEndian.java:25)
at io.netty.buffer.DrillBuf.getBytes(DrillBuf.java:596)
at io.netty.buffer.DrillBuf.getBytes(DrillBuf.java:596)
at io.netty.buffer.DrillBuf.getBytes(DrillBuf.java:596)
at io.netty.buffer.DrillBuf.getBytes(DrillBuf.java:596)
at
org.apache.drill.exec.vector.VarCharVector$Accessor.get(VarCharVector.java:387)
at
org.apache.drill.exec.vector.VarCharVector$Accessor.getObject(VarCharVector.java:411)
at
org.apache.drill.exec.vector.accessor.VarCharAccessor.getObject(VarCharAccessor.java:108)
at
org.apache.drill.exec.vector.accessor.BoundCheckingAccessor.getObject(BoundCheckingAccessor.java:137)
at
org.apache.drill.jdbc.AvaticaDrillSqlAccessor.getObject(AvaticaDrillSqlAccessor.java:165)
at
net.hydromatic.avatica.AvaticaResultSet.getObject(AvaticaResultSet.java:351)
at sqlline.SqlLine$Rows$Row.init(SqlLine.java:2388)
at sqlline.SqlLine$IncrementalRows.hasNext(SqlLine.java:2504)
at sqlline.SqlLine$TableOutputFormat.print(SqlLine.java:2148)
at sqlline.SqlLine.print(SqlLine.java:1809)
at sqlline.SqlLine$Commands.execute(SqlLine.java:3766)
at sqlline.SqlLine$Commands.sql(SqlLine.java:3663)
at sqlline.SqlLine.dispatch(SqlLine.java:889)
at sqlline.SqlLine.begin(SqlLine.java:763)
at sqlline.SqlLine.start(SqlLine.java:498)
at sqlline.SqlLine.main(SqlLine.java:460)
{code}
Not sure if this is a client-specific issue as there is no exception from the
drillbit log files
However if I project a varchar column from a json file after a FOJ, there seems
to be no issues
{code}
0: jdbc:drill:schema=dfs_eea select
. . . . . . . . . . . . . . o.varchar_col
. . . . . . . . . . . . . . from dfs.`cross-sources`.`fewtypes.parquet` p
. . . . . . . . . . . . . . full outer join
dfs.`cross-sources`.`fewtypes_null.json` o
. . . . . . . . . . . . . . on p.int_col=o.int_col;
+-+
| varchar_col |
+-+
| jllkjsdhfg |
| null|
| gfdstweopiu |
| gjklhsdfgkjhkASDF |
| oieoiutriotureWERTgwgEWRg |
| gjkdfkjglfd |
| ioerutklsdfASDgerGWEr |
| lkjgfiurtoUYFHfahui |
| IOUfiuodsfIUfjkh |
| iweuoHUIhUwer |
| null|
| dfgoiuert |
| uitreo |
| uigoMnvjjkdf |
| NvvdfHVG|
| null|
| null|
| uiuikjk |
| null|
| hjiwgh |
| null|
| jhgduitweriuoert |
| KfijUIwre |
| Nhkhuivb|
| null|
| null|
+-+
26 rows selected (0.212 seconds)
{code}
I attached the parquet and json files used. Let me know if you need anything
more.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Regarding unit test failures -- patterns possible fixes
My mac knows it has 2 physical and 4 logical cores: ted:spy$ sysctl hw | grep cpu hw.ncpu: 4 hw.activecpu: 4 hw.physicalcpu: 2 hw.physicalcpu_max: 2 hw.logicalcpu: 4 hw.logicalcpu_max: 4 hw.cputype: 7 hw.cpusubtype: 4 hw.cpu64bit_capable: 1 hw.cpufamily: 526772277 hw.cpufrequency: 29 hw.cpufrequency_min: 29 hw.cpufrequency_max: 29 hw.cputhreadtype: 1 hw.ncpu = 4 hw.cpufrequency = 29 hw.availcpu = 4 On Mon, Apr 6, 2015 at 5:52 PM, Hanifi Gunes hgu...@maprtech.com wrote: @Ted, I actually have a patch pending review for this at https://issues.apache.org/jira/browse/DRILL-2039. However, @Hakeem you may still end up in a troubling situation as your mac might have 4 logical cores =) *sysctl hw | grep **cpu* should tell you. -Hanifi On Mon, Apr 6, 2015 at 5:22 PM, Ted Dunning ted.dunn...@gmail.com wrote: Maven allows the forkCount to be set in terms of the number of cores on the machine by appending a C to the number. (I think) On Mon, Apr 6, 2015 at 5:15 PM, Abdel Hakim Deneche adene...@maprtech.com wrote: From my personal experience, using a forkCount of 4 makes some tests to timeout, not because there are some concurrent problems but just because my machine is not powerful enough to run as many tests in parallel. On Mon, Apr 6, 2015 at 5:06 PM, Hanifi Gunes hgu...@maprtech.com wrote: Hey devs, I have been dealing with sporadic unit test failures since last week. I'd like to share my findings on failing unit tests outlining the patterns causing these. The general advice I was given was to run tests lowering the fork count (the default is 4). This sounds ok to get around the problem. If this is your intent feel free to give a try. However, I wanted to investigate this further, went ahead and set fork count to 8, number of cores testing machine has. The first set of failures manifested around TestUnionAll TestExampleQueries. After Hakeem's pointer, I noticed that some test cases in TestUnionAll TestExampleQueries creates drops views with the same name. In a concurrent settings(given fork count 0) with no strict ordering among test cases, this sure creates a havoc throwing arbitrary errors at each test run. https://issues.apache.org/jira/browse/DRILL-2684 takes care of this. After applying this fix, now I can cleanly run java-exec tests, faster consistently. Then my hive tests started failing since multiple test classes use the same metadata folders and clean up after without caring whether other hive tests are still running. I got a patch to unique-ify metadata folders for test class. https://issues.apache.org/jira/browse/DRILL-2685 tracks this however the patch is not public yet. The last problem relates to hive related tests living under jdbc module. I know that there has been an ongoing effort to migrate these tests from jdbc module to where they belong. I am not sure about the timeframe for the migration but I would think the sooner is better. To avoid further failures, I would strongly recommend devs to use view names and external resources that are unique across test cases. One idea is to suffix view a view name with test case/class name for instance prefer using names_view_test_union_all rather than a more generic names_view. Also with increasing number of test cases checked in, it takes more and more time to complete a test run. We should consider concurrent test runs as a legitimate everyday use case and design tests accordingly. I would be interested in hearing other unit test failure patterns so as to be alert. Feel free to share if you discovered any. Regards. -Hanifi -- Abdelhakim Deneche Software Engineer http://www.mapr.com/ Now Available - Free Hadoop On-Demand Training http://www.mapr.com/training?utm_source=Emailutm_medium=Signatureutm_campaign=Free%20available
Re: Review Request 32867: DRILL-2695: Add support for large in lists via values operator
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32867/#review79089 --- exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/fn/JsonReader.java https://reviews.apache.org/r/32867/#comment128187 what is the utility of this loop here? exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/fn/JsonReader.java https://reviews.apache.org/r/32867/#comment128192 I think we should set inOuterList above the if statement if an empty array is a valid record. exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/fn/VectorOutput.java https://reviews.apache.org/r/32867/#comment128216 Why is this an IOException? This seems to have nothing with I/O. exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/fn/VectorOutput.java https://reviews.apache.org/r/32867/#comment128220 The same. - Hanifi Gunes On April 5, 2015, 11:49 p.m., Jacques Nadeau wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32867/ --- (Updated April 5, 2015, 11:49 p.m.) Review request for drill, Aman Sinha and Hanifi Gunes. Repository: drill-git Description --- Add Support for large in conditions through the use of the Values operator. Update JSON reader to support reading Extended JSON. Update JSON writer to support writing extended JSON data. Update JSON reader to automatically unwrap a file that includes a single top-level array. Update Options manager to use getOption(TypeValidator) to directly retrieve typed value. Remove JSON rewinding Diffs - common/src/main/java/org/apache/drill/common/JSONOptions.java 945cd92 common/src/main/java/org/apache/drill/common/logical/data/Constant.java 460803d common/src/main/java/org/apache/drill/common/logical/data/Values.java PRE-CREATION common/src/main/java/org/apache/drill/common/logical/data/visitors/AbstractLogicalVisitor.java 92e370f common/src/main/java/org/apache/drill/common/logical/data/visitors/LogicalVisitor.java 3a426bf contrib/storage-mongo/src/main/java/org/apache/drill/exec/store/mongo/MongoRecordReader.java 15ef197 exec/java-exec/src/main/codegen/templates/JsonOutputRecordWriter.java 1d0dc9d exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java bd93206 exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/DateUtility.java a031bee exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/conv/JsonConvertFrom.java c828cf4 exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/AbstractPhysicalVisitor.java 27b0ecb exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/PhysicalVisitor.java e6a89d0 exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/Values.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/values/ValuesBatchCreator.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillOptiq.java 796f0f7 exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillRuleSets.java b1a7189 exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillValuesRel.java a3551e7 exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/ScanFieldDeterminer.java 59c65f9 exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/ValuesPrel.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/ValuesPrule.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/server/options/BaseOptionManager.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/server/options/FallbackOptionManager.java 45d393c exec/java-exec/src/main/java/org/apache/drill/exec/server/options/OptionManager.java 4ffe9a3 exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SystemOptionManager.java 608fac7 exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/json/JSONFormatPlugin.java 6cf1ce5 exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/json/JSONRecordReader.java cc7cb83 exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/json/JsonProcessor.java ce6017b exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/json/JsonRecordWriter.java 76c4ace exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/json/RewindableUtf8Reader.java b9075de exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/json/reader/BaseJsonProcessor.java 509798a
Re: Review Request 32713: DRILL-2638: Regularize startup/shutdown logging a bit in/near Drillbit.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32713/#review79115 --- exec/java-exec/src/main/java/org/apache/drill/exec/server/Drillbit.java https://reviews.apache.org/r/32713/#comment128227 In future, please use the Guava stopwatch for this time of thing. - Jacques Nadeau On April 1, 2015, 8:55 p.m., Daniel Barclay wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32713/ --- (Updated April 1, 2015, 8:55 p.m.) Review request for drill, Jacques Nadeau and Parth Chandra. Bugs: DRILL-2638 https://issues.apache.org/jira/browse/DRILL-2638 Repository: drill-git Description --- In Drillbit: - Added shutdown time (ms). - Regularized to have bracketing starting and finished messages (when DEBUG enabled; include only messages with times at INFO level). (See examples below.) - Edited text. In FunctionImplementationRegistry - DEBUG - INFO for message with startup time. - Edited text. (Examples: At DEBUG level: ... DEBUG o.apache.drill.exec.server.Drillbit - Starting new Drillbit. ... DEBUG o.apache.drill.exec.server.Drillbit - Construction started. ... INFO o.apache.drill.exec.server.Drillbit - Construction completed (1259 ms). ... DEBUG o.apache.drill.exec.server.Drillbit - Startup begun. ... INFO o.apache.drill.exec.server.Drillbit - Startup completed (5359 ms). ... DEBUG o.apache.drill.exec.server.Drillbit - Started new Drillbit. ... ... DEBUG o.apache.drill.exec.server.Drillbit - Shutdown begun. ... INFO o.apache.drill.exec.server.Drillbit - Shutdown completed (6346 ms). At INFO level: ... INFO o.apache.drill.exec.server.Drillbit - Construction completed (1259 ms). ... INFO o.apache.drill.exec.server.Drillbit - Startup completed (5359 ms). ... ... INFO o.apache.drill.exec.server.Drillbit - Shutdown completed (6346 ms). ) Diffs - exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionImplementationRegistry.java e96fa60 exec/java-exec/src/main/java/org/apache/drill/exec/server/Drillbit.java 958f2dc Diff: https://reviews.apache.org/r/32713/diff/ Testing --- Ran at INFO vs. DEBUG levels to confirm (manually). Ran existing tests; no new errors. Thanks, Daniel Barclay
Re: Review Request 32795: DRILL-2383: Add exception and pause injections for testing drillbit stability
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32795/#review79118
---
exec/java-exec/src/main/java/org/apache/drill/exec/rpc/user/QueryResultHandler.java
https://reviews.apache.org/r/32795/#comment128228
you can get rid of finished and just use queryState to check if the
listener has finished
- abdelhakim deneche
On April 6, 2015, 10:53 p.m., Sudheesh Katkam wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32795/
---
(Updated April 6, 2015, 10:53 p.m.)
Review request for drill, abdelhakim deneche, Chris Westin, and Jacques
Nadeau.
Repository: drill-git
Description
---
[DRILL-2383](https://issues.apache.org/jira/browse/DRILL-2383): Inject
exceptions and pauses in various components of Drill
+ Controls can be introduced in any class that has access to FragmentContext
or QueryContext
+ Controls can be fired by altering the DRILLBIT_CONTROL_INJECTIONS session
option
+ Renames: SimulatedExceptions = ExecutionControls, ExceptionInjector =
ExecutionControlsInjector
+ Instructions to add other types of injections are in Injection
+ ExecutionControls are added as a side effect of altering
DRILLBIT_CONTROL_INJECTIONS session option
+ Added injection sites in Foreman, DrillSqlWorker, FragmentExecutor
+ Unit tests in TestDrillbitResilience, TestExceptionInjection and
TestPauseInjection
Other commits included:
+ [DRILL-2437](https://issues.apache.org/jira/browse/DRILL-2437): Moved
ExecutionControls from DrillbitContext to UserSession
+ [DRILL-2382](https://issues.apache.org/jira/browse/DRILL-2382): Added
address and port to InjectionOption to specify drillbit
Other bugs fixed:
+ BUG: Only one SystemRecord (static instance) exists per node. This causes
race conditions when multiple
drillbits run on the same node. FIX: create a new instance per request.
Other edits:
+ Added QueryId back to QueryContext
+ Log query id as string in DrillClient, WorkEventBus, QueryResultHandler
+ Use try..catch block only around else clause for OptionList in
FragmentContext
+ Fixed drillbitContext spelling error in QueryContext
+ Show explicitly that submitWork() returns queryId in UserServer
+ Added an alternative way to generate sources in protocol/readme.txt
+ Added JSONStringValidator to TypeValidators
=
USAGE:
Current checked exception sites:
+ Foreman: run-try-beginning (ForemanException), run-try-end
(ForemanException), send-fragments (ForemanException)
+ DrillSqlWorker: sql-parsing (ForemanSetupException)
+ FragmentExecutor: fragment-execution (IOException)
Current pause sites:
+ Foreman: pause-run-plan
To set controls:
```
ALTER SESSION SET `drill.exec.testing.controls`='{
injections:[
{
type:exception,
siteClass: org.apache.drill.exec.work.fragment.FragmentExecutor,
desc: fragment-execution,
nSkip: 0,
nFire: 1,
exceptionClass: java.io.IOException,
drillbitAddress: 10.10.10.10,
userPort: 31019
},
{
type:pause,
siteClass: org.apache.drill.exec.work.foreman.Foreman,
desc: pause-run-plan,
millis: 5000
}
] }';
```
NOTE:
(1) If controls are specified, they are passed to every fragment as part of
PlanFragment. Then onwards, ExecutionControls live in FragmentContext. And
since FragmentContext is created for every fragment, injections will be fired
on ALL fragments.
(2) drillbitAddress and userPort are optional. If they are set, that
injection will be fired ONLY on specified drillbit. If they are not set, the
injection will be fired on EVERY drillbit.
Diffs
-
exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java
bd93206
exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java
9a948fb
exec/java-exec/src/main/java/org/apache/drill/exec/client/PrintingResultsListener.java
98948af
exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java
a4ac724
exec/java-exec/src/main/java/org/apache/drill/exec/ops/QueryContext.java
3b51a69
exec/java-exec/src/main/java/org/apache/drill/exec/planner/PhysicalPlanReader.java
8d77136
exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/SimpleParallelizer.java
66ba229
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DrillSqlWorker.java
d6f25fb
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/SetOptionHandler.java
dc63ef9
exec/java-exec/src/main/java/org/apache/drill/exec/rpc/control/WorkEventBus.java
a5a5441
Re: Review Request 32910: minor doc changes
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32910/#review79098 --- Ship it! Ship It! - Bridget Bridget On April 6, 2015, 11:48 p.m., Kristine Hahn wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32910/ --- (Updated April 6, 2015, 11:48 p.m.) Review request for drill and Bridget Bridget. Bugs: DRILL-2701 https://issues.apache.org/jira/browse/DRILL-2701 Repository: drill-git Description --- minor doc changes Diffs - _docs/connect/009-mapr-db-plugin.md 62aa068 _docs/query/003-query-hbase.md bdfbb8a _docs/sql-ref/001-data-types.md 5607c85 _docs/sql-ref/003-operators.md b9a95e7 _docs/sql-ref/data-types/001-date.md ec97e3b _docs/sql-ref/functions/002-conversion.md 64fc25c _docs/sql-ref/functions/005-aggregate.md 250527f _docs/sql-ref/functions/006-nulls.md 9041e49 Diff: https://reviews.apache.org/r/32910/diff/ Testing --- Thanks, Kristine Hahn
Re: drill-interpreter project errors
Chris Westin wrote: That's not going to remove the project from your IDE. Of course, but shouldn't it have deleted all the files from Git? Daniel When I saw Jason's commit went through, I removed the project, updated maven modules metadata, and rebuilt after the next time I rebased. Then all was well. On Mon, Apr 6, 2015 at 1:52 PM, Daniel Barclay dbarc...@maprtech.com wrote: You can safely delete the module from your working directory. Thanks. Why didn't my rebasing of my branch on master remove the module? Daniel Jason Altekruse wrote: The interpreter module was removed in a recent commit to make the interpreter usable at planning time. The interpreter code was committed a few months ago, but only had unit tests written against it in its own module. To make the interpreter work with the other modules and hook it up to planning rules it needed to be refactored, the way we chose to refactor it actually removed the need for the separate module and build step. You can safely delete the module from your working directory. https://github.com/apache/drill/commit/0aa8b19d624d173da51de36aa164f3 435d3366a4 -Jason Altekruse On Mon, Apr 6, 2015 at 1:20 PM, Daniel Barclay dbarc...@maprtech.com wrote: What's going on with the drill-interpreter module? For while now, in Eclipse it shows up as having errors (e.g., that org.apache.drill.BaseTestQuery is unknown), even after I issue the Maven - Update Project ... command on it. Daniel -- Daniel Barclay MapR Technologies -- Daniel Barclay MapR Technologies -- Daniel Barclay MapR Technologies
[jira] [Resolved] (DRILL-2214) Aggregate function documentation needs to be corrected
[
https://issues.apache.org/jira/browse/DRILL-2214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kristine Hahn resolved DRILL-2214.
--
Resolution: Fixed
The info about DISTINCT and ALL has been added to the doc. Included your
examples, also:
http://drill.apache.org/docs/aggregate-and-aggregate-statistical/
Aggregate function documentation needs to be corrected
--
Key: DRILL-2214
URL: https://issues.apache.org/jira/browse/DRILL-2214
Project: Apache Drill
Issue Type: Bug
Components: Documentation
Affects Versions: 0.8.0
Reporter: Victoria Markman
Assignee: Kristine Hahn
It turns out that all of our basic aggregate functions (MIN, MAX, COUNT, AVG,
SUM) accept ALL and DISTINCT keywords.
If none specified, ALL is the default.
For example,
{code}
0: jdbc:drill:schema=dfs select a2 from t2;
++
| a2 |
++
| 0 |
| 1 |
| 2 |
| 2 |
| 2 |
| 3 |
| 4 |
| 5 |
| 6 |
| 7 |
| 7 |
| 8 |
| 9 |
++
13 rows selected (0.056 seconds)
0: jdbc:drill:schema=dfs select avg(all a2) from t2;
++
| EXPR$0 |
++
| 4.3076923076923075 |
++
1 row selected (0.084 seconds)
0: jdbc:drill:schema=dfs select avg(distinct a2) from t2;
++
| EXPR$0 |
++
| 4.5|
++
1 row selected (0.079 seconds)
0: jdbc:drill:schema=dfs select sum(all a2) from t2;
++
| EXPR$0 |
++
| 56 |
++
1 row selected (0.086 seconds)
0: jdbc:drill:schema=dfs select sum(distinct a2) from t2;
++
| EXPR$0 |
++
| 45 |
++
1 row selected (0.078 seconds)
++
| EXPR$0 |
++
| 13 |
++
1 row selected (0.056 seconds)
0: jdbc:drill:schema=dfs select count(all a2) from t2;
++
| EXPR$0 |
++
| 13 |
++
1 row selected (0.056 seconds)
0: jdbc:drill:schema=dfs select count(distinct a2) from t2;
++
| EXPR$0 |
++
| 10 |
++
1 row selected (0.074 seconds)
{code}
Including MIN/MAX:
{code}
0: jdbc:drill:schema=dfs select min(distinct a2) from t2;
++
| EXPR$0 |
++
| 0 |
++
1 row selected (0.095 seconds)
0: jdbc:drill:schema=dfs select min(all a2) from t2;
++
| EXPR$0 |
++
| 0 |
++
1 row selected (0.065 seconds)
0: jdbc:drill:schema=dfs select max(all a2) from t2;
++
| EXPR$0 |
++
| 9 |
++
1 row selected (0.094 seconds)
0: jdbc:drill:schema=dfs select max(distinct a2) from t2;
++
| EXPR$0 |
++
| 9 |
++
1 row selected (0.067 seconds)
{code}
I don't know if we should mention MIN/MAX, but SUM and AVG we should.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (DRILL-2215) COALESCE function needs to be documented
[ https://issues.apache.org/jira/browse/DRILL-2215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kristine Hahn resolved DRILL-2215. -- Resolution: Fixed Now covered on http://drill.apache.org/docs/functions-for-handling-nulls/ and ready for review. Thanks. COALESCE function needs to be documented Key: DRILL-2215 URL: https://issues.apache.org/jira/browse/DRILL-2215 Project: Apache Drill Issue Type: Bug Components: Documentation Affects Versions: 0.8.0 Reporter: Victoria Markman Assignee: Kristine Hahn COALESCE function returns the first non-null expression in the list. If all expressions evaluate to null, then the COALESCE function will return null. COALESCE( expr1, expr2, ... expr_n ) expr1 to expr_n are the expressions to test for non-null values. Expressions have to be of the same type. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (DRILL-2706) doc configuration options
Kristine Hahn created DRILL-2706: Summary: doc configuration options Key: DRILL-2706 URL: https://issues.apache.org/jira/browse/DRILL-2706 Project: Apache Drill Issue Type: Task Components: Documentation Reporter: Kristine Hahn Assignee: Kristine Hahn -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 32795: DRILL-2383: Add exception and pause injections for testing drillbit stability
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32795/#review79108
---
Some initial thoughts. Still working on second half of patch.
exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java
https://reviews.apache.org/r/32795/#comment128202
use {} syntax so we aren't always calling to string (even when not logging).
exec/java-exec/src/main/java/org/apache/drill/exec/planner/PhysicalPlanReader.java
https://reviews.apache.org/r/32795/#comment128203
This shouldn't be in this class. This has nothing to do with reading
physical plans. If you want to get hold of a object mapper, go through config.
exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/SimpleParallelizer.java
https://reviews.apache.org/r/32795/#comment128204
don't remove. this guarantees builder formatting.
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/SetOptionHandler.java
https://reviews.apache.org/r/32795/#comment128205
Please move all this functionality to the option managers and option
validators. For example, have a hook in the validator to execute a command
against the option manager. Add which types of options are allowed at which
levels, etc. (I think the framing is already there for this.)
exec/java-exec/src/main/java/org/apache/drill/exec/rpc/control/WorkEventBus.java
https://reviews.apache.org/r/32795/#comment128208
wrap this and below with if(logger.isDebugEnabled()){}
exec/java-exec/src/main/java/org/apache/drill/exec/rpc/user/UserResultsListener.java
https://reviews.apache.org/r/32795/#comment128207
Why is this change part of this patch?
exec/java-exec/src/main/java/org/apache/drill/exec/rpc/user/UserServer.java
https://reviews.apache.org/r/32795/#comment128209
why the change?
exec/java-exec/src/main/java/org/apache/drill/exec/rpc/user/UserServer.java
https://reviews.apache.org/r/32795/#comment128210
why the change?
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/MemoryRecord.java
https://reviews.apache.org/r/32795/#comment128212
why is this called dummy instance?
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/MemoryRecord.java
https://reviews.apache.org/r/32795/#comment128211
why remove? people shouldn't be creating a memory record, right?
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/ThreadsRecord.java
https://reviews.apache.org/r/32795/#comment128213
same, why remove?
exec/java-exec/src/main/java/org/apache/drill/exec/testing/ExceptionInjection.java
https://reviews.apache.org/r/32795/#comment128214
Can you please move to using JsonCreator and doing all validation at
deserialization time? This should also remove the need for two separate
classes.
exec/java-exec/src/main/java/org/apache/drill/exec/testing/ExceptionInjection.java
https://reviews.apache.org/r/32795/#comment128217
We should probably check the method constructor at exception identification
time, not at fire time.
exec/java-exec/src/main/java/org/apache/drill/exec/testing/ExceptionInjection.java
https://reviews.apache.org/r/32795/#comment128218
Why do we need a custom serializer here?
- Jacques Nadeau
On April 6, 2015, 10:53 p.m., Sudheesh Katkam wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32795/
---
(Updated April 6, 2015, 10:53 p.m.)
Review request for drill, abdelhakim deneche, Chris Westin, and Jacques
Nadeau.
Repository: drill-git
Description
---
[DRILL-2383](https://issues.apache.org/jira/browse/DRILL-2383): Inject
exceptions and pauses in various components of Drill
+ Controls can be introduced in any class that has access to FragmentContext
or QueryContext
+ Controls can be fired by altering the DRILLBIT_CONTROL_INJECTIONS session
option
+ Renames: SimulatedExceptions = ExecutionControls, ExceptionInjector =
ExecutionControlsInjector
+ Instructions to add other types of injections are in Injection
+ ExecutionControls are added as a side effect of altering
DRILLBIT_CONTROL_INJECTIONS session option
+ Added injection sites in Foreman, DrillSqlWorker, FragmentExecutor
+ Unit tests in TestDrillbitResilience, TestExceptionInjection and
TestPauseInjection
Other commits included:
+ [DRILL-2437](https://issues.apache.org/jira/browse/DRILL-2437): Moved
ExecutionControls from DrillbitContext to UserSession
+ [DRILL-2382](https://issues.apache.org/jira/browse/DRILL-2382): Added
address and port to InjectionOption to specify drillbit
Other bugs fixed:
+ BUG: Only one SystemRecord (static instance) exists per node. This causes
race conditions when
Review Request 32801: DRILL-2668: Fix: CAST(1.1 AS FLOAT) was yielding DOUBLE.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32801/ --- Review request for drill, Mehant Baid and Parth Chandra. Bugs: DRILL-2668 https://issues.apache.org/jira/browse/DRILL-2668 Repository: drill-git Description --- Added test method (and others for parallel cases) to TestFunctionsQuery. Fixed use of DOUBLE methods/classes for FLOAT: - Changed a FLOAT - DOUBLE mapping in ValueExpressions. - Added parallel FLOAT methods/classes in EvaluationVisitor. Diffs - common/src/main/java/org/apache/drill/common/expression/ValueExpressions.java e095e7e exec/java-exec/src/main/java/org/apache/drill/exec/expr/EvaluationVisitor.java 386ab79 exec/java-exec/src/test/java/org/apache/drill/TestFunctionsQuery.java f1005ab Diff: https://reviews.apache.org/r/32801/diff/ Testing --- Ran new specific test and existing tests. Thanks, Daniel Barclay
[jira] [Created] (DRILL-2708) Query failed: AssertionError: VARCHAR
Daniel Barclay (Drill) created DRILL-2708:
-
Summary: Query failed: AssertionError: VARCHAR
Key: DRILL-2708
URL: https://issues.apache.org/jira/browse/DRILL-2708
Project: Apache Drill
Issue Type: Bug
Reporter: Daniel Barclay (Drill)
Here's another case of assertion message that is bad (not quite saying what the
problem is) and that shows up at the user interface (SQLLine):
{noformat}
0: jdbc:drill:zk=local EXPLAIN PLAN FOR SELECT catalog_name['asd'] FROM
INFORMATION_SCHEMA.CATALOGS;
Query failed: AssertionError: VARCHAR
Error: exception while executing query: Failure while executing query.
(state=,code=0)
0: jdbc:drill:zk=local SELECT catalog_name['asd'] FROM
INFORMATION_SCHEMA.CATALOGS;
Query failed: AssertionError: VARCHAR
Error: exception while executing query: Failure while executing query.
(state=,code=0)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: User Authentication patch: question with regards to UserCredentials versus properties.
Currently we add UserCredentials to every PlanFragment we generate as part of the query. PlanFragments are distributed across the Drillbits in cluster. Adding password to UserCredentials will expose the password to all Drillbits beyond the Drillbit where the UserSession is. This is main reason I didn't want to add password to UserCredentials for now. In future if we want to securely identify PlanFragment, we can make use of other methods such as delegation token. Adding the password to Properties list is not something Drill introduced. JDBC already gets the username and password are through Properties which become part of UserSession at connected Drillbit. We currently get username property in UserCredentials object from Properties we got from JDBC. Thanks Venki On Mon, Apr 6, 2015 at 10:02 PM, Jacques Nadeau jacq...@apache.org wrote: Hey Venki et al, I just saw that DRILL-2674 went in. It looks like we're handling the password field as a general property. The intention of the UserCredentials object is that it would be expanded to support the other fields that were required as part of user credentials (beyond just username). Was there a reason that you decided to use a general property rather than expanding the credential object to specifically hold the password? thx Jacques
Review Request 32886: DRILL-2639: Planner bug - RelOptPlanner.CannotPlanExceptio
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32886/ --- Review request for drill and Aman Sinha. Bugs: DRILL-2639 https://issues.apache.org/jira/browse/DRILL-2639 Repository: drill-git Description --- At planning for Union-All, block only the cases where implicit casting cannot resolve an output type Diffs - exec/java-exec/src/main/java/org/apache/drill/exec/planner/common/DrillRelOptUtil.java aba6022 exec/java-exec/src/main/java/org/apache/drill/exec/planner/common/DrillUnionRelBase.java 932aa76 exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillOptiq.java 796f0f7 exec/java-exec/src/test/java/org/apache/drill/TestUnionAll.java fee1d6a Diff: https://reviews.apache.org/r/32886/diff/ Testing --- QA, unit Thanks, Sean Hsuan-Yi Chu
