Re: Review Request 32795: DRILL-2383: Add exception and pause injections for testing drillbit stability
On April 14, 2015, 4:28 p.m., Chris Westin wrote:
exec/java-exec/src/main/java/org/apache/drill/exec/rpc/user/QueryResultHandler.java,
line 237
https://reviews.apache.org/r/32795/diff/3-4/?file=918272#file918272line237
What's going on here? Can we get some comments explaining this? I would
have expected the state to be FINISHED or CANCELLED.
state should be FINISHED or CANCELLED indeed. The assertion was added to make
sure this.queryState was not set already by a previous message from the server
- abdelhakim
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32795/#review80037
---
On April 14, 2015, 4:45 a.m., Sudheesh Katkam wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32795/
---
(Updated April 14, 2015, 4:45 a.m.)
Review request for drill, abdelhakim deneche, Chris Westin, and Jacques
Nadeau.
Repository: drill-git
Description
---
[DRILL-2383](https://issues.apache.org/jira/browse/DRILL-2383): Support to
inject exceptions and pauses in various components of Drill
+ Controls can be introduced in any class that has access to
FragmentContext/QueryContext
+ Controls are fired only if assertions are enabled
+ Controls can be fired by altering the DRILLBIT_CONTROL_INJECTIONS session
option
+ Renames: SimulatedExceptions = ExecutionControls, ExceptionInjector =
ExecutionControlsInjector
+ Added injection sites in Foreman, DrillSqlWorker, FragmentExecutor
+ Unit tests in TestDrillbitResilience, TestExceptionInjection and
TestPauseInjection
Other commits included:
+ [DRILL-2437](https://issues.apache.org/jira/browse/DRILL-2437): Moved
ExecutionControls from DrillbitContext to FragmentContext/QueryContext
+ [DRILL-2382](https://issues.apache.org/jira/browse/DRILL-2382): Added
address and port to Injection to specify drillbit
+ [DRILL-2384](https://issues.apache.org/jira/browse/DRILL-2384): Added
QueryState to SingleRowListener and assert that state is COMPLETED while
testing
Other edits:
+ Support for short lived session options in SessionOptionManager (using TTL
in OptionValidator)
+ Introduced query count in UserSession
+ Added QueryState to queryCompleted() in UserResultsListener to check if
COMPLETED/CANCELED
+ Added JSONStringValidator to TypeValidators
+ Log query id as string in DrillClient, WorkEventBus, QueryResultHandler
+ Use try..catch block only around else clause for OptionList in
FragmentContext
+ Fixed drillbitContext spelling error in QueryContext
+ Do not call setLocalOption twice in FallbackOptionManager
+ Show explicitly that submitWork() returns queryId in UserServer
+ Updated protocol/readme.txt to include an alternative way to generate
sources
=
USAGE:
Current checked exception sites:
+ Foreman: run-try-beginning (ForemanException), run-try-end
(ForemanException), send-fragments (ForemanException)
+ DrillSqlWorker: sql-parsing (ForemanSetupException)
+ FragmentExecutor: fragment-execution (IOException)
Current pause sites:
+ Foreman: pause-run-plan
To set controls:
```
ALTER SESSION SET `drill.exec.testing.controls`='{
injections:[
{
type:exception,
siteClass: org.apache.drill.exec.work.fragment.FragmentExecutor,
desc: fragment-execution,
nSkip: 0,
nFire: 1,
exceptionClass: java.io.IOException,
address: 10.10.10.10,
port: 31019
},
{
type:pause,
siteClass: org.apache.drill.exec.work.foreman.Foreman,
desc: pause-run-plan,
nSkip: 0,
nFire: 1,
millis: 5000
}
] }';
```
NOTE:
(1) If controls are specified, they are passed to every fragment as part of
PlanFragment. Then onwards, ExecutionControls live in FragmentContext. And
since FragmentContext is created for every fragment, injections will be fired
on ALL fragments.
(2) address and port are optional. If they are set, that injection will be
fired ONLY on specified drillbit. If they are not set, the injection will be
fired on EVERY drillbit.
Diffs
-
exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java
bd93206
exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java
579cf7d
exec/java-exec/src/main/java/org/apache/drill/exec/client/PrintingResultsListener.java
98948af
exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java
da2229c
exec/java-exec/src/main/java/org/apache/drill/exec/ops/QueryContext.java
2fa0b18
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DrillSqlWorker.java
b98778d
Re: Community Hangout happening now
Can we start recording these for later consumption? On Tue, Apr 14, 2015 at 10:00 AM, Abdel Hakim Deneche adene...@maprtech.com wrote: Feel free to join the hangout: https://plus.google.com/hangouts/_/event/ci4rdiju8bv04a64efj5fedd0lc -- Abdelhakim Deneche Software Engineer http://www.mapr.com/ Now Available - Free Hadoop On-Demand Training http://www.mapr.com/training?utm_source=Emailutm_medium=Signatureutm_campaign=Free%20available
Review Request 33180: DRILL-2181 Throw proper error message when flatten is used within an 'order by' or 'group by'
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33180/ --- Review request for drill and Jason Altekruse. Bugs: DRILL-2181 https://issues.apache.org/jira/browse/DRILL-2181 Repository: drill-git Description --- Flatten function in order by, group by, distinct, aggregate functions is disabled Diffs - exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/parser/UnsupportedOperatorsVisitor.java 4830fe1 exec/java-exec/src/test/java/org/apache/drill/TestDisabledFunctionality.java f62f060 Diff: https://reviews.apache.org/r/33180/diff/ Testing --- tpch, functional (except for the cases which are meant to be removed) Thanks, Sean Hsuan-Yi Chu
[jira] [Created] (DRILL-2781) Protobuf changes for nested loop join
Mehant Baid created DRILL-2781: -- Summary: Protobuf changes for nested loop join Key: DRILL-2781 URL: https://issues.apache.org/jira/browse/DRILL-2781 Project: Apache Drill Issue Type: Bug Reporter: Mehant Baid Assignee: Mehant Baid Priority: Minor Fix For: 0.9.0 A couple of the protobuf files were not regenerated as part of the nested loop join change. Will regenerate the couple of files and merge as part of this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 32795: DRILL-2383: Add exception and pause injections for testing drillbit stability
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32795/#review80037 --- exec/java-exec/src/main/java/org/apache/drill/exec/ops/QueryContext.java https://reviews.apache.org/r/32795/#comment129788 Why do FragmentContext and QueryContext both need their own set of ExecutionControls? exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/SimpleParallelizer.java https://reviews.apache.org/r/32795/#comment129789 newBuilder() goes on the previous line, since it's not actually setting an attribute in the builder. Do we really need these empty end-of-line comments? exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/SimpleParallelizer.java https://reviews.apache.org/r/32795/#comment129790 If you are going to use these end-of-line comments, then always space them the same way, and do it on all lines exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DrillSqlWorker.java https://reviews.apache.org/r/32795/#comment129791 This doesn't look like an exact substitution; what's this change about? exec/java-exec/src/main/java/org/apache/drill/exec/rpc/user/QueryResultHandler.java https://reviews.apache.org/r/32795/#comment129792 What's going on here? Can we get some comments explaining this? I would have expected the state to be FINISHED or CANCELLED. exec/java-exec/src/main/java/org/apache/drill/exec/server/options/OptionValidator.java https://reviews.apache.org/r/32795/#comment129793 Can we get a comment describing what this is? exec/java-exec/src/main/java/org/apache/drill/exec/server/options/OptionValidator.java https://reviews.apache.org/r/32795/#comment129794 Comments for the arguments? exec/java-exec/src/main/java/org/apache/drill/exec/server/options/OptionValidator.java https://reviews.apache.org/r/32795/#comment129795 Should this be validated? Is any value ok? exec/java-exec/src/main/java/org/apache/drill/exec/server/options/OptionValidator.java https://reviews.apache.org/r/32795/#comment129796 -1 has a special meaning? exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SessionOptionManager.java https://reviews.apache.org/r/32795/#comment129798 This looks stateless -- could it be static? - Chris Westin On April 13, 2015, 9:45 p.m., Sudheesh Katkam wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32795/ --- (Updated April 13, 2015, 9:45 p.m.) Review request for drill, abdelhakim deneche, Chris Westin, and Jacques Nadeau. Repository: drill-git Description --- [DRILL-2383](https://issues.apache.org/jira/browse/DRILL-2383): Support to inject exceptions and pauses in various components of Drill + Controls can be introduced in any class that has access to FragmentContext/QueryContext + Controls are fired only if assertions are enabled + Controls can be fired by altering the DRILLBIT_CONTROL_INJECTIONS session option + Renames: SimulatedExceptions = ExecutionControls, ExceptionInjector = ExecutionControlsInjector + Added injection sites in Foreman, DrillSqlWorker, FragmentExecutor + Unit tests in TestDrillbitResilience, TestExceptionInjection and TestPauseInjection Other commits included: + [DRILL-2437](https://issues.apache.org/jira/browse/DRILL-2437): Moved ExecutionControls from DrillbitContext to FragmentContext/QueryContext + [DRILL-2382](https://issues.apache.org/jira/browse/DRILL-2382): Added address and port to Injection to specify drillbit + [DRILL-2384](https://issues.apache.org/jira/browse/DRILL-2384): Added QueryState to SingleRowListener and assert that state is COMPLETED while testing Other edits: + Support for short lived session options in SessionOptionManager (using TTL in OptionValidator) + Introduced query count in UserSession + Added QueryState to queryCompleted() in UserResultsListener to check if COMPLETED/CANCELED + Added JSONStringValidator to TypeValidators + Log query id as string in DrillClient, WorkEventBus, QueryResultHandler + Use try..catch block only around else clause for OptionList in FragmentContext + Fixed drillbitContext spelling error in QueryContext + Do not call setLocalOption twice in FallbackOptionManager + Show explicitly that submitWork() returns queryId in UserServer + Updated protocol/readme.txt to include an alternative way to generate sources = USAGE: Current checked exception sites: + Foreman: run-try-beginning (ForemanException), run-try-end (ForemanException), send-fragments (ForemanException) + DrillSqlWorker: sql-parsing (ForemanSetupException) + FragmentExecutor: fragment-execution (IOException)
Re: Review Request 33052: DRILL-2611: value vectors should report valid value count
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33052/#review80041 --- Ship it! Ship It! - Mehant Baid On April 13, 2015, 11:01 p.m., Hanifi Gunes wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33052/ --- (Updated April 13, 2015, 11:01 p.m.) Review request for drill, Mehant Baid and Parth Chandra. Repository: drill-git Description --- DRILL-2611: value vectors should report valid value count Changes - unify the behavior of value count interfaces across VVs -- get/setters - ensure value count reported reflects underlying state of the buffer - enforce consumers to use getAccessor().get/setValueCount - ensure metadata created based on getAccessor().getValueCount Diffs - exec/java-exec/src/main/codegen/templates/ComplexWriters.java 576fd8352197ba950be7d7e661fb52dd92b52f2a exec/java-exec/src/main/codegen/templates/FixedValueVectors.java e9ec220dc653db1e1acb0538bbbc1207fb4ee194 exec/java-exec/src/main/codegen/templates/NullableValueVectors.java 075316e4f3ac5327e7893688c5e88cfee98e50bc exec/java-exec/src/main/codegen/templates/RepeatedValueVectors.java c7cf8e6fe18f1b9813ae22495ac79a447f61cfff exec/java-exec/src/main/codegen/templates/VariableLengthVectors.java edb851eb10be43d889ce5fd98d9bde036707870a exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/ColumnReader.java 759327a307aefd51dc69ea4282a7d58d6309e142 exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/FixedByteAlignedReader.java c2af964fd606924587fe2093b3ccb1ec1de922af exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/FixedWidthRepeatedReader.java f20d7655c76237fc5d3a95760f00ca2400a8df07 exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/NullableColumnReader.java 16519a851a18924fb59753c456a2da4076d5d245 exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/NullableFixedByteAlignedReaders.java 8087118e1de8ef1b043c80ab4fc85215284670dc exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/VarLengthColumnReaders.java 7464f30179059a728f5c30f37c17adf0f332604c exec/java-exec/src/main/java/org/apache/drill/exec/vector/BaseDataValueVector.java d48ea99237bb822cafc8b835c3af0f4789c6eb29 exec/java-exec/src/main/java/org/apache/drill/exec/vector/BaseValueVector.java 81d3a8623fb86068d8c81f08e1d38d37b856e26c exec/java-exec/src/main/java/org/apache/drill/exec/vector/BitVector.java d8bd9723db9f2ecd1466b1144345ca371f68a3bb Diff: https://reviews.apache.org/r/33052/diff/ Testing --- unit, reg, sf100 Thanks, Hanifi Gunes
Re: Review Request 33035: DRILL-2719: ValueVector#getBuffers(clear) must consistently clear vectors retain buffers
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33035/#review80039 --- Ship it! Ship It! - Mehant Baid On April 13, 2015, 10:07 p.m., Hanifi Gunes wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33035/ --- (Updated April 13, 2015, 10:07 p.m.) Review request for drill, Mehant Baid and Parth Chandra. Repository: drill-git Description --- DRILL-2719: ValueVector#getBuffers(clear) must consistently clear vectors retain buffers BaseDataValueVector - getBuffers now rely on getBufferSize while determining buffers to return - getBuffers maintains reference count to underlying buffers while clearing the vector - getBufferSize relies on value count reported by accessor while determining buffer size - replaced DeadBuf references with an empty buffer. underlying buffer now should never be *null*. Templates VV subtypes - ensure getBuffers conforms to VV#getBuffers TestEmptyPopulator - make mock allocator return an empty buffer when requested Diffs - exec/java-exec/src/main/codegen/templates/NullableValueVectors.java 075316e4f3ac5327e7893688c5e88cfee98e50bc exec/java-exec/src/main/codegen/templates/RepeatedValueVectors.java c7cf8e6fe18f1b9813ae22495ac79a447f61cfff exec/java-exec/src/main/codegen/templates/VariableLengthVectors.java edb851eb10be43d889ce5fd98d9bde036707870a exec/java-exec/src/main/java/org/apache/drill/exec/vector/BaseDataValueVector.java d48ea99237bb822cafc8b835c3af0f4789c6eb29 exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/AbstractMapVector.java b0783afe57317dcd8dc7a2a8d967dcdb1f305edb exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/RepeatedListVector.java c0f529961343145d67f835a95f58c4eaf2fae2a4 exec/java-exec/src/test/java/org/apache/drill/exec/vector/complex/TestEmptyPopulator.java 8426a6abbf0c20be6f81bc82521f31ed7cde2557 Diff: https://reviews.apache.org/r/33035/diff/ Testing --- unit and beyond Thanks, Hanifi Gunes
Re: Review Request 32987: DRILL-2675: Implement a subset of User Exceptions to improve how errors are reported to the user
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32987/#review80026 --- General comment on improving pattern. I propose the calling pattern to be as follows: (1) throw UserException.wrap(e) .type(ErrorType.DATA_READ) .message(%s - %s, suffix) .addContext(File Name, /myfile) .addContext(Line Number, 4) .build(); or (2) throw UserException.create() .type(ErrorType.DATA_READ) .message(%s - %s, suffix) .addContext(File Name, /myfile) .addContext(Line Number, 4) .build(); For (1), this will generate a new message if the exception tree doesn't have a UserException. If it does have a UserException, the type and message will be dropped but the context will be added. For (2), we're creating a new exception explicitly because we don't have a causing exception to pass in. Do you think this will cover the key cases? I think we don't need Drill as part of UserException. I also think that all constructors for UserException and its children should be hidden and all entry should be through create() or wrap(). common/src/main/java/org/apache/drill/common/exceptions/DrillUserException.java https://reviews.apache.org/r/32987/#comment129781 common/src/main/java/org/apache/drill/common/exceptions/DrillUserException.java https://reviews.apache.org/r/32987/#comment129778 common/src/main/java/org/apache/drill/common/exceptions/UserExceptionContext.java https://reviews.apache.org/r/32987/#comment129777 comment out of date since we can no longer specify verbose properties common/src/main/java/org/apache/drill/common/exceptions/UserExceptionContext.java https://reviews.apache.org/r/32987/#comment129779 says int but is long. - Jacques Nadeau On April 13, 2015, 5:55 p.m., abdelhakim deneche wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32987/ --- (Updated April 13, 2015, 5:55 p.m.) Review request for drill, Jacques Nadeau, Jason Altekruse, and Parth Chandra. Bugs: DRILL-2675 https://issues.apache.org/jira/browse/DRILL-2675 Repository: drill-git Description --- **INITIAL PATCH** with a working solution. This patch cleans the path for errors, especially user errors with meaningful messages, to be propagated properly to the client. The patch includes changes to 2 existing use cases where the error message was successfully improved. The general idea is: if a code wants to throw an exception that contains a meaningful error message, it throws a DrillUserException. The propagation code will make sure this exception is propagated to the client. The user exception object doesn't contain the final error message, but enough information about the error, the client will use this information to display a better error message. Any exception that is not a DrillUserException (or one of it's subclasses) will be considered as a system exception. For those exceptions the client will only display the error id and drillbit identity in case the user wants to check the logs for more informations about the error. Error objects sent to the client will still contain a stack trace that can be used to display more information if the client has enabled the error verbose mode. Diffs - common/src/main/java/org/apache/drill/common/exceptions/DrillRemoteException.java PRE-CREATION common/src/main/java/org/apache/drill/common/exceptions/DrillUserException.java PRE-CREATION common/src/main/java/org/apache/drill/common/exceptions/ErrorHelper.java PRE-CREATION common/src/main/java/org/apache/drill/common/exceptions/UserExceptionContext.java PRE-CREATION common/src/test/java/org/apache/drill/common/exceptions/TestDrillUserException.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java 9a948fb exec/java-exec/src/main/java/org/apache/drill/exec/client/PrintingResultsListener.java 98948af exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java da2229c exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/ScreenCreator.java 6b3caf4 exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DrillSqlWorker.java b98778d exec/java-exec/src/main/java/org/apache/drill/exec/rpc/CoordinationQueue.java 0016d6a exec/java-exec/src/main/java/org/apache/drill/exec/rpc/RemoteRpcException.java 14ea873 exec/java-exec/src/main/java/org/apache/drill/exec/rpc/RpcBus.java b974963 exec/java-exec/src/main/java/org/apache/drill/exec/rpc/user/QueryResultHandler.java a1be83b
Re: Review Request 33180: DRILL-2181 Throw proper error message when flatten is used within an 'order by' or 'group by'
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33180/ --- (Updated April 15, 2015, 12:42 a.m.) Review request for drill and Jason Altekruse. Changes --- update description Bugs: DRILL-2181 https://issues.apache.org/jira/browse/DRILL-2181 Repository: drill-git Description --- Flatten function in order by, group by, distinct, aggregate functions is disabled Diffs - exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/parser/UnsupportedOperatorsVisitor.java 4830fe1 exec/java-exec/src/test/java/org/apache/drill/TestDisabledFunctionality.java f62f060 Diff: https://reviews.apache.org/r/33180/diff/ Testing (updated) --- tpch, functional (except for the cases which are meant to be removed), unit Thanks, Sean Hsuan-Yi Chu
[jira] [Created] (DRILL-2796) Select keys from JSON file where column-name not in null results in RelOptPlanner.CannotPlanException
Khurram Faraaz created DRILL-2796:
-
Summary: Select keys from JSON file where column-name not in
null results in RelOptPlanner.CannotPlanException
Key: DRILL-2796
URL: https://issues.apache.org/jira/browse/DRILL-2796
Project: Apache Drill
Issue Type: Bug
Components: Query Planning Optimization
Affects Versions: 0.9.0
Environment: 9d92b8e319f2d46e8659d903d355450e15946533 | DRILL-2580:
Exit early from HashJoinBatch if build side is empty | 26.03.2015 @ 16:13:53 EDT
Reporter: Khurram Faraaz
Assignee: Jinfeng Ni
Query that has, column-name not in (null) in its predicate, results in
RelOptPlanner.CannotPlanException. Tests were run on 4 node cluster and on
CentOS.
Data is being selected from a JSON data file.
{code}
0: jdbc:drill: select * from `mKeyJSN.json`;
+++++++++
|key1|key2|key3|key4|key5|key6|
key7|key8|
+++++++++
| 1234 | null | null | null | null | null |
null | null |
| null | 1245685| null | null | null | null |
null | null |
| null | null | hello world! | null | null | null
| null | null |
| null | null | null | true | null | null |
null | null |
| null | null | null | null | 2000-03-10 | null |
null | null |
| null | null | null | null | null | 2012-01-21
15:19:12.123 | null | null |
| null | null | null | null | null | null |
21:34:32.321 | null |
| null | null | null | null | null | null |
null | 9789.99|
+++++++++
8 rows selected (0.1 seconds)
store.json.all_text_mode was set to false
0: jdbc:drill: select * from sys.options where name like
'%json.all_text_mode%';
++++++++
|name|kind|type| num_val | string_val | bool_val |
float_val |
++++++++
| store.json.all_text_mode | BOOLEAN| SYSTEM | null | null
| false | null |
++++++++
1 row selected (0.135 seconds)
Failing query
0: jdbc:drill: select key1,key2,key3,key4,key5,key6,key7,key8 from
`mKeyJSN.json` where key5 not in (null);
Query failed: RelOptPlanner.CannotPlanException: Node
[rel#14924:Subset#7.LOGICAL.ANY([]).[]] could not be implemented; planner state:
Root: rel#14924:Subset#7.LOGICAL.ANY([]).[]
Original rel:
AbstractConverter(subset=[rel#14924:Subset#7.LOGICAL.ANY([]).[]],
convention=[LOGICAL], DrillDistributionTraitDef=[ANY([])], sort=[[]]): rowcount
= 1.7976931348623157E308, cumulative cost = {inf}, id = 14925
ProjectRel(subset=[rel#14923:Subset#7.NONE.ANY([]).[]], key1=[$2], key2=[$3],
key3=[$4], key4=[$5], key5=[$1], key6=[$6], key7=[$7], key8=[$8]): rowcount =
1.7976931348623157E308, cumulative cost = {1.7976931348623157E308 rows,
Infinity cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 14922
FilterRel(subset=[rel#14921:Subset#6.NONE.ANY([]).[]],
condition=[AND(NOT(IS TRUE($11)), IS NOT NULL($9))]): rowcount =
4.0448095534402104E307, cumulative cost = {4.0448095534402104E307 rows,
1.7976931348623157E308 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 14920
JoinRel(subset=[rel#14919:Subset#5.NONE.ANY([]).[]], condition=[=($9,
$10)], joinType=[left]): rowcount = 1.7976931348623157E308, cumulative cost =
{1.7976931348623157E308 rows, 0.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id =
14918
ProjectRel(subset=[rel#14912:Subset#1.NONE.ANY([]).[]], $f0=[$0],
$f1=[$1], $f2=[$2], $f3=[$3], $f4=[$4], $f5=[$5], $f6=[$6], $f7=[$7], $f8=[$8],
$f9=[$1]): rowcount = 100.0, cumulative cost = {100.0 rows, 1000.0 cpu, 0.0 io,
0.0 network, 0.0 memory}, id = 14911
EnumerableTableAccessRel(subset=[rel#14910:Subset#0.ENUMERABLE.ANY([]).[]],
table=[[dfs, tmp, mKeyJSN.json]]): rowcount = 100.0, cumulative cost = {100.0
rows, 101.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 14888
AggregateRel(subset=[rel#14917:Subset#4.NONE.ANY([]).[]], group=[{0}],
agg#0=[MIN($1)]): rowcount = 1.7976931348623158E307, cumulative cost =
{1.7976931348623158E307 rows, 0.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id =
14916
Re: Review Request 33209: DRILL-2341: Type information is lost if list of fields is specified during create view
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33209/#review80159 --- exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/ViewHandler.java https://reviews.apache.org/r/33209/#comment129925 This part of logic is quite similar in ViewHandler and CreateTableHandler. Is it possible that we move the common code to a helper method (check the field count, *, create the rename project, etc), so that both of the Handlers could call this method? - Jinfeng Ni On April 14, 2015, 10:28 p.m., Venki Korukanti wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33209/ --- (Updated April 14, 2015, 10:28 p.m.) Review request for drill and Jinfeng Ni. Repository: drill-git Description --- Create a cast to rename the view definition query row type to view field list specified in CREATE VIEW statement. Diffs - exec/java-exec/src/main/java/org/apache/drill/exec/planner/common/DrillRelOptUtil.java cacf26b exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/ViewHandler.java 4347249 exec/java-exec/src/test/java/org/apache/drill/exec/sql/TestViewSupport.java 578eace Diff: https://reviews.apache.org/r/33209/diff/ Testing --- Includes unittests. Thanks, Venki Korukanti
Re: Review Request 32987: DRILL-2675: Implement a subset of User Exceptions to improve how errors are reported to the user
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32987/ --- (Updated April 15, 2015, 4:09 a.m.) Review request for drill, Jacques Nadeau, Jason Altekruse, and Parth Chandra. Changes --- - removed public methods from ErrorHelper. - ErrorHelper and UserExceptionContext are package protected Bugs: DRILL-2675 https://issues.apache.org/jira/browse/DRILL-2675 Repository: drill-git Description (updated) --- The general idea is: if a code wants to throw an exception that contains a meaningful error message, it throws a UserException. The propagation code will make sure this exception is propagated to the client. The user exception object doesn't contain the final error message, but enough information about the error, the client will use this information to display a better error message. Any exception that is not a UserException (or one of it's subclasses) will be considered as a system exception. For those exceptions the client will display the error id and drillbit identity and the root exception's message. Error objects sent to the client will still contain a stack trace that can be used to display more information if the client has enabled the error verbose mode. **List of changes** - added UserException - _RemoteUserException_ replaces _RemoteRpcException_ - moved _ErrorHelper_ from exec/work to common/exceptions - Root executors don't log transmission errors anymore, they will be logged by the _FragmentContext_ - added 2 examples of how to use _UserException.Builder_ . _DrillSqlWorker_ throws a parse error when it catches a _ValidationException_ . _JsonRecordReader_ throws a data read error - made changes to _DrillPBError_ . enum _ErrorType_ error_type - we make sure to wrap exceptions into _UserException_ before building a _DrillPBError_ in _FragmentContext_, _FragmentExecutor_ and _Foreman_ - _AbstractStatusReporter_ doesn't log errors anymore, it just builds the corresponding _DrillPBError_ - _UserException_ has a context that can be used to add context information to existing user errors. The context is treated as an array of strings that will be displayed along the error message. The context object offers both add and push methods to add the context information to the bottom or top of the context list . context automatically generates a unique error id that can be used to track errors in the logs . context accepts _DrillbitEndpoint_ information that is added and displayed along the generated message - system errors will display the root exception message. This will allow Drill to provide a consistent experience even for exceptions not yet converted into proper user errors - _UserException.Builder_ can be used to create user exceptions. It's the only way to generate user/system exceptions - _UserResultListener.submissionFailed()_ expects a _UserException_ instead of _RemoteRpcException_ . fixed multiple unit tests that were expecting _RemoteRpcException_ . improved unit tests that were extracting the exception class and message from the error message itself - added proper documentation for _UserException_, _UserException.Builder_, _UserExceptionContext_ and _ErrorHelper_ - _ErrorHelper_ and _UserExceptionContext_ are package protected classes Diffs (updated) - common/src/main/java/org/apache/drill/common/exceptions/ErrorHelper.java PRE-CREATION common/src/main/java/org/apache/drill/common/exceptions/UserException.java PRE-CREATION common/src/main/java/org/apache/drill/common/exceptions/UserExceptionContext.java PRE-CREATION common/src/main/java/org/apache/drill/common/exceptions/UserRemoteException.java PRE-CREATION common/src/test/java/org/apache/drill/common/exceptions/TestUserException.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java 579cf7d exec/java-exec/src/main/java/org/apache/drill/exec/client/PrintingResultsListener.java 98948af exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java da2229c exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DrillSqlWorker.java b98778d exec/java-exec/src/main/java/org/apache/drill/exec/rpc/CoordinationQueue.java 0016d6a exec/java-exec/src/main/java/org/apache/drill/exec/rpc/RemoteRpcException.java 14ea873 exec/java-exec/src/main/java/org/apache/drill/exec/rpc/RpcBus.java b974963 exec/java-exec/src/main/java/org/apache/drill/exec/rpc/user/QueryResultHandler.java a1be83b exec/java-exec/src/main/java/org/apache/drill/exec/rpc/user/UserResultsListener.java 934a094 exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/QueryWrapper.java fbbf0b8 exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/json/JSONRecordReader.java cc7cb83
Re: Review Request 33209: DRILL-2341: Type information is lost if list of fields is specified during create view
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33209/#review80158 --- Ship it! One small note. Otherwise, lgtm exec/java-exec/src/main/java/org/apache/drill/exec/planner/common/DrillRelOptUtil.java https://reviews.apache.org/r/33209/#comment129924 can you create a jira (and not it here) which tracks removing this and is dependent on Jinfeng's calcite rebase? - Jacques Nadeau On April 15, 2015, 5:28 a.m., Venki Korukanti wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33209/ --- (Updated April 15, 2015, 5:28 a.m.) Review request for drill and Jinfeng Ni. Repository: drill-git Description --- Create a cast to rename the view definition query row type to view field list specified in CREATE VIEW statement. Diffs - exec/java-exec/src/main/java/org/apache/drill/exec/planner/common/DrillRelOptUtil.java cacf26b exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/ViewHandler.java 4347249 exec/java-exec/src/test/java/org/apache/drill/exec/sql/TestViewSupport.java 578eace Diff: https://reviews.apache.org/r/33209/diff/ Testing --- Includes unittests. Thanks, Venki Korukanti
[jira] [Created] (DRILL-2790) Implicit cast to numbers fails when Text data contains empty strings
Abhishek Girish created DRILL-2790:
--
Summary: Implicit cast to numbers fails when Text data contains
empty strings
Key: DRILL-2790
URL: https://issues.apache.org/jira/browse/DRILL-2790
Project: Apache Drill
Issue Type: Bug
Components: Query Planning Optimization
Affects Versions: 0.9.0
Reporter: Abhishek Girish
Assignee: Jinfeng Ni
When a column in a Text file (CSV) contains empty strings, any query which uses
valid implicit casts fail to execute with NumberFormatException. Drill should
ignore empty strings in such scenarios. If not, the error message must be
improved to clearly indicate the same.
*Text:*
{code:sql}
select columns[4] from `store.dat` limit 10;
++
| EXPR$0 |
++
| 2451189|
||
||
| 2451044|
| 2450910|
||
||
||
||
||
++
10 rows selected (0.154 seconds)
select * from `store.dat` where columns[4] = 2451189 limit 1;
Query failed: RemoteRpcException: Failure while running fragment., [
e5348e11-ec53-4332-981a-ff070253c19a on abhi6.qa.lab:31010 ]
[ e5348e11-ec53-4332-981a-ff070253c19a on abhi6.qa.lab:31010 ]
Error: exception while executing query: Failure while executing query.
(state=,code=0)
{code}
Log snippet:
{code}
2015-04-14 14:16:50,642 [2ad27c3c-8b36-a116-4d38-18410eff96d2:frag:0:0] ERROR
o.a.drill.exec.ops.FragmentContext - Fragment Context received failure --
Fragment: 0:0
java.lang.NumberFormatException:
at
org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.nfeI(StringFunctionHelpers.java:97)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.varCharToInt(StringFunctionHelpers.java:103)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.test.generated.FiltererGen97.doEval(FilterTemplate2.java:37)
~[na:na]
at
org.apache.drill.exec.test.generated.FiltererGen97.filterBatchNoSV(FilterTemplate2.java:98)
~[na:na]
at
org.apache.drill.exec.test.generated.FiltererGen97.filterBatch(FilterTemplate2.java:71)
~[na:na]
at
org.apache.drill.exec.physical.impl.filter.FilterRecordBatch.doWork(FilterRecordBatch.java:82)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:93)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:142)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:118)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
{code}
The following query works:
{code:sql}
select columns[4] from `store.dat` where columns[4] '' AND columns[4]=
2451189 limit 1;
++
| EXPR$0 |
++
| 2451189|
++
1 row selected (0.174 seconds)
{code}
*Parquet:*
{code:sql}
select s_closed_date_sk from store where s_closed_date_sk = 2451189 limit 1;
+--+
| s_closed_date_sk |
+--+
| 2451189 |
+--+
1 row selected (0.122 seconds)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (DRILL-2791) Improve error messages in C++ client for handshake failure
Norris Lee created DRILL-2791: - Summary: Improve error messages in C++ client for handshake failure Key: DRILL-2791 URL: https://issues.apache.org/jira/browse/DRILL-2791 Project: Apache Drill Issue Type: Bug Components: Client - C++ Reporter: Norris Lee Assignee: Norris Lee Priority: Minor Fix For: 0.9.0 Authentication and bad rpc errors only return the generic -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (DRILL-2792) Killing the drillbit which is the foreman results in direct memory being held on
Ramana Inukonda Nagaraj created DRILL-2792: -- Summary: Killing the drillbit which is the foreman results in direct memory being held on Key: DRILL-2792 URL: https://issues.apache.org/jira/browse/DRILL-2792 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Reporter: Ramana Inukonda Nagaraj Assignee: Chris Westin Killed one of the drillbits which is the foreman for the query- Profiles page reports that query has cancelled. Due to bug Drill-2778 sqlline hangs. However after killing sqlline the current direct memory used does not go down to pre query levels. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (DRILL-2784) handle filenames with slashes (/)
Will Ochandarena created DRILL-2784: --- Summary: handle filenames with slashes (/) Key: DRILL-2784 URL: https://issues.apache.org/jira/browse/DRILL-2784 Project: Apache Drill Issue Type: Bug Components: Client - CLI Affects Versions: 0.8.0 Reporter: Will Ochandarena Assignee: Daniel Barclay (Drill) Google App Engine writes JSON logs in the following directory/filename format - YY/MM/DD/HH/MM/SS_HH/MM/SS_S0.json Sadly, some of these forward slashes indicate new path (directory for year, month, day) - YY/MM/DD Others are embedded in the filename ( file per start_hour, end_hour) - HH/MM/SS_HH/MM/SS_S0.json Example string (spaces added to illustrate where filename starts) - 2015/04/14/ 17/00/00_17/59/59_S0.json Drill throws this error - 0: jdbc:drill:zk=local select * from dfs.`/.../2015/04/14/17/00/00_17/59/59_S0.json` LIMIT 5; Apr 14, 2015 11:39:05 AM org.eigenbase.sql.validate.SqlValidatorException init SEVERE: org.eigenbase.sql.validate.SqlValidatorException: Table 'dfs./.../2015/04/14/17/00/00_17/59/59_S0.json' not found Apr 14, 2015 11:39:05 AM org.eigenbase.util.EigenbaseException init SEVERE: org.eigenbase.util.EigenbaseContextException: From line 1, column 15 to line 1, column 17: Table 'dfs./Users/wochandarena/installerdialhomelogs/2015/04/14/17/00/00_17/59/59_S0.json' not found Query failed: SqlValidatorException: Table 'dfs./Users/wochandarena/installerdialhomelogs/2015/04/14/17/00/00_17/59/59_S0.json' not found If the file is renamed to not have slashes the query is successful. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (DRILL-2785) Aggregate MAX query does not return the maximum value
Khurram Faraaz created DRILL-2785:
-
Summary: Aggregate MAX query does not return the maximum value
Key: DRILL-2785
URL: https://issues.apache.org/jira/browse/DRILL-2785
Project: Apache Drill
Issue Type: Bug
Components: Query Planning Optimization
Affects Versions: 0.9.0
Environment: | 9d92b8e319f2d46e8659d903d355450e15946533 | DRILL-2580:
Exit early from HashJoinBatch if build side is empty | 26.03.2015 @ 16:13:53
EDT
Reporter: Khurram Faraaz
Assignee: Jinfeng Ni
Aggregate query that should return maximum value from over 63036308 records,
results in SQLException. Test was run on 4 node cluster on CentOS.
{code}
0: jdbc:drill: select max(cast(columns[0] as bigint)) from `deletions`;
++
| EXPR$0 |
++
Query failed: RemoteRpcException: Failure while running fragment.,
===Listeners will have opportunities to call in
to ask questions they have always wanted to ask about wealth [
67b3a1e6-0f1e-4bae-9e61-2e1d7fb7ba0d on centos-03.qa.lab:31010 ]
[ 67b3a1e6-0f1e-4bae-9e61-2e1d7fb7ba0d on centos-03.qa.lab:31010 ]
java.lang.RuntimeException: java.sql.SQLException: Failure while executing
query.
at sqlline.SqlLine$IncrementalRows.hasNext(SqlLine.java:2514)
at sqlline.SqlLine$TableOutputFormat.print(SqlLine.java:2148)
at sqlline.SqlLine.print(SqlLine.java:1809)
at sqlline.SqlLine$Commands.execute(SqlLine.java:3766)
at sqlline.SqlLine$Commands.sql(SqlLine.java:3663)
at sqlline.SqlLine.dispatch(SqlLine.java:889)
at sqlline.SqlLine.begin(SqlLine.java:763)
at sqlline.SqlLine.start(SqlLine.java:498)
at sqlline.SqlLine.main(SqlLine.java:460)
Physical plan for the failing query
0: jdbc:drill: explain plan for select max(cast(columns[0] as bigint)) from
`deletions`;
+++
|text|json|
+++
| 00-00Screen
00-01 StreamAgg(group=[{}], EXPR$0=[MAX($0)])
00-02UnionExchange
01-01 StreamAgg(group=[{}], EXPR$0=[MAX($0)])
01-02Project($f0=[CAST(ITEM($0, 0)):BIGINT])
01-03 Scan(groupscan=[EasyGroupScan [selectionRoot=/tmp/deletions,
numFiles=20, columns=[`columns`[0]],
files=[maprfs:/tmp/deletions/deletions-7-of-00020.csv,
maprfs:/tmp/deletions/deletions-00016-of-00020.csv,
maprfs:/tmp/deletions/deletions-00012-of-00020.csv,
maprfs:/tmp/deletions/deletions-8-of-00020.csv,
maprfs:/tmp/deletions/deletions-00019-of-00020.csv,
maprfs:/tmp/deletions/deletions-00015-of-00020.csv,
maprfs:/tmp/deletions/deletions-00018-of-00020.csv,
maprfs:/tmp/deletions/deletions-4-of-00020.csv,
maprfs:/tmp/deletions/deletions-0-of-00020.csv,
maprfs:/tmp/deletions/deletions-2-of-00020.csv,
maprfs:/tmp/deletions/deletions-5-of-00020.csv,
maprfs:/tmp/deletions/deletions-00014-of-00020.csv,
maprfs:/tmp/deletions/deletions-00017-of-00020.csv,
maprfs:/tmp/deletions/deletions-00010-of-00020.csv,
maprfs:/tmp/deletions/deletions-6-of-00020.csv,
maprfs:/tmp/deletions/deletions-3-of-00020.csv,
maprfs:/tmp/deletions/deletions-1-of-00020.csv,
maprfs:/tmp/deletions/deletions-00013-of-00020.csv,
maprfs:/tmp/deletions/deletions-9-of-00020.csv,
maprfs:/tmp/deletions/deletions-00011-of-00020.csv]]])
Note that count took close to 11 seconds, to count total number of records in
the files in that directory.
0: jdbc:drill: select count(columns[0]) from `deletions`;
++
| EXPR$0 |
++
| 63036308 |
++
1 row selected (10.669 seconds)
stack trace from drillbit.log
2015-04-14 18:36:56,513 [2ad2a1bd-cbb9-e1b9-d454-5f1882f6f427:frag:1:0] WARN
o.a.d.e.w.fragment.FragmentExecutor - Error while initializing or executing
fragment
java.lang.NumberFormatException: ,fr
at
org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.nfeL(StringFunctionHelpers.java:90)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.varCharToLong(StringFunctionHelpers.java:61)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.test.generated.ProjectorGen0.doEval(ProjectorTemplate.java:35)
~[na:na]
at
org.apache.drill.exec.test.generated.ProjectorGen0.projectRecords(ProjectorTemplate.java:62)
~[na:na]
at
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.doWork(ProjectRecordBatch.java:174)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:93)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:134)
[jira] [Created] (DRILL-2787) NPE when first argument to maxdir UDF is invalid
Victoria Markman created DRILL-2787:
---
Summary: NPE when first argument to maxdir UDF is invalid
Key: DRILL-2787
URL: https://issues.apache.org/jira/browse/DRILL-2787
Project: Apache Drill
Issue Type: Bug
Components: Functions - Drill
Affects Versions: 0.9.0
Reporter: Victoria Markman
Assignee: Jason Altekruse
dfs. is not a valid workspace
{code}
0: jdbc:drill:schema=dfs select * from bigtable where dir0 =
maxdir('dfs.','bigtable');
Query failed: NullPointerException:
Error: exception while executing query: Failure while executing query.
(state=,code=0)
{code}
dfs.test1 does not exist
{code}
0: jdbc:drill:schema=dfs select * from bigtable where dir0 =
maxdir('dfs.test1','bigtable');
Query failed: NullPointerException:
Error: exception while executing query: Failure while executing query.
(state=,code=0)
{code}
Empty string as a first argument:
{code}
0: jdbc:drill:schema=dfs select * from bigtable where dir0 =
maxdir('','bigtable');
Query failed: NullPointerException:
Error: exception while executing query: Failure while executing query.
(state=,code=0)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (DRILL-2788) Fix empty population logic in VVs
Hanifi Gunes created DRILL-2788: --- Summary: Fix empty population logic in VVs Key: DRILL-2788 URL: https://issues.apache.org/jira/browse/DRILL-2788 Project: Apache Drill Issue Type: Bug Components: Execution - Data Types Affects Versions: 0.8.0 Reporter: Hanifi Gunes Assignee: Hanifi Gunes The code block that populates empty values in NullableVVs does not handle the case where we populate a single value. The fix is often simple but debugging such failures is deadly. This issue proposes to verify existing empty population logic across VVs and fix them if found incorrect. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 33181: fix Hive version number in doc
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33181/#review80090 --- Ship it! Ship It! - Bridget Bridget On April 14, 2015, 6:15 p.m., Kristine Hahn wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33181/ --- (Updated April 14, 2015, 6:15 p.m.) Review request for drill and Bridget Bridget. Bugs: DRILL-2773 https://issues.apache.org/jira/browse/DRILL-2773 Repository: drill-git Description --- change from 1.0 to 0.13 Diffs - _docs/connect/006-reg-hive.md dfb03dc Diff: https://reviews.apache.org/r/33181/diff/ Testing --- Thanks, Kristine Hahn
Re: Review Request 32987: DRILL-2675: Implement a subset of User Exceptions to improve how errors are reported to the user
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32987/ --- (Updated April 14, 2015, 10:29 p.m.) Review request for drill, Jacques Nadeau, Jason Altekruse, and Parth Chandra. Changes --- - renamed DrillUserException to UserException - made the builder easier to use - added more documentation Bugs: DRILL-2675 https://issues.apache.org/jira/browse/DRILL-2675 Repository: drill-git Description --- **INITIAL PATCH** with a working solution. This patch cleans the path for errors, especially user errors with meaningful messages, to be propagated properly to the client. The patch includes changes to 2 existing use cases where the error message was successfully improved. The general idea is: if a code wants to throw an exception that contains a meaningful error message, it throws a DrillUserException. The propagation code will make sure this exception is propagated to the client. The user exception object doesn't contain the final error message, but enough information about the error, the client will use this information to display a better error message. Any exception that is not a DrillUserException (or one of it's subclasses) will be considered as a system exception. For those exceptions the client will only display the error id and drillbit identity in case the user wants to check the logs for more informations about the error. Error objects sent to the client will still contain a stack trace that can be used to display more information if the client has enabled the error verbose mode. Diffs (updated) - common/src/main/java/org/apache/drill/common/exceptions/ErrorHelper.java PRE-CREATION common/src/main/java/org/apache/drill/common/exceptions/UserException.java PRE-CREATION common/src/main/java/org/apache/drill/common/exceptions/UserExceptionContext.java PRE-CREATION common/src/main/java/org/apache/drill/common/exceptions/UserRemoteException.java PRE-CREATION common/src/test/java/org/apache/drill/common/exceptions/TestUserException.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java 579cf7d exec/java-exec/src/main/java/org/apache/drill/exec/client/PrintingResultsListener.java 98948af exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java da2229c exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DrillSqlWorker.java b98778d exec/java-exec/src/main/java/org/apache/drill/exec/rpc/CoordinationQueue.java 0016d6a exec/java-exec/src/main/java/org/apache/drill/exec/rpc/RemoteRpcException.java 14ea873 exec/java-exec/src/main/java/org/apache/drill/exec/rpc/RpcBus.java b974963 exec/java-exec/src/main/java/org/apache/drill/exec/rpc/user/QueryResultHandler.java a1be83b exec/java-exec/src/main/java/org/apache/drill/exec/rpc/user/UserResultsListener.java 934a094 exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/QueryWrapper.java fbbf0b8 exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/json/JSONRecordReader.java cc7cb83 exec/java-exec/src/main/java/org/apache/drill/exec/work/ErrorHelper.java 0773d6c exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java 23ef0d3 exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/QueryManager.java 8626d5b exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/SqlUnsupportedException.java 2299afa exec/java-exec/src/main/java/org/apache/drill/exec/work/fragment/AbstractStatusReporter.java 1b0885d exec/java-exec/src/main/java/org/apache/drill/exec/work/fragment/FragmentExecutor.java a7e6c46 exec/java-exec/src/main/java/org/apache/drill/exec/work/fragment/StatusReporter.java 26b5d68 exec/java-exec/src/test/java/org/apache/drill/BaseTestQuery.java 0c2f0e5 exec/java-exec/src/test/java/org/apache/drill/SingleRowListener.java 5703bf9 exec/java-exec/src/test/java/org/apache/drill/TestBugFixes.java 875fb25 exec/java-exec/src/test/java/org/apache/drill/TestDisabledFunctionality.java f62f060 exec/java-exec/src/test/java/org/apache/drill/TestStarQueries.java effef9b exec/java-exec/src/test/java/org/apache/drill/TestUnionAll.java 11d83f9 exec/java-exec/src/test/java/org/apache/drill/exec/server/TestDrillbitResilience.java e03098a exec/java-exec/src/test/java/org/apache/drill/exec/store/parquet/ParquetResultListener.java 55f0d75 exec/java-exec/src/test/java/org/apache/drill/exec/store/parquet/TestParquetPhysicalPlan.java 882cdbd exec/jdbc/src/main/java/org/apache/drill/jdbc/DrillCursor.java d666d06 exec/jdbc/src/main/java/org/apache/drill/jdbc/impl/DrillResultSetImpl.java 24ef62b protocol/src/main/java/org/apache/drill/exec/proto/SchemaUserBitShared.java f72d5e1
[jira] [Created] (DRILL-2793) Killing a non foreman node results in direct memory being held on
Ramana Inukonda Nagaraj created DRILL-2793:
--
Summary: Killing a non foreman node results in direct memory being
held on
Key: DRILL-2793
URL: https://issues.apache.org/jira/browse/DRILL-2793
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Reporter: Ramana Inukonda Nagaraj
Assignee: Chris Westin
Similar to DRILL-2792
Happens for non foreman nodes as well.
before:
{code}
1/6 select * from sys.memory;
+++--++++
| hostname | user_port | heap_current | heap_max | direct_current |
direct_max |
+++--++++
| atsqa6c62.qa.lab | 31010 | 72738752 | 4151836672 | 6048576|
34359738368 |
| atsqa6c59.qa.lab | 31010 | 177100136| 4151836672 | 200|
34359738368 |
| atsqa6c60.qa.lab | 31010 | 183895944| 4151836672 | 200|
34359738368 |
| atsqa6c57.qa.lab | 31010 | 312198128| 4151836672 | 200|
34359738368 |
| atsqa6c58.qa.lab | 31010 | 180850296| 4151836672 | 200|
34359738368 |
| atsqa6c61.qa.lab | 31010 | 307163256| 4151836672 | 200|
34359738368 |
| atsqa6c64.qa.lab | 31010 | 178883744| 4151836672 | 200|
34359738368 |
| atsqa6c63.qa.lab | 31010 | 326312736| 4151836672 | 200|
34359738368 |
+++--++++
8 rows selected (2.209 seconds)
{code}
After cancellation of non foreman node
{code}
0: jdbc:drill: select * from sys.memory;
+++--++++
| hostname | user_port | heap_current | heap_max | direct_current |
direct_max |
+++--++++
| atsqa6c62.qa.lab | 31010 | 395684912| 4151836672 | 1745146306 |
34359738368 |
| atsqa6c57.qa.lab | 31010 | 416717016| 4151836672 | 1751348355 |
34359738368 |
| atsqa6c58.qa.lab | 31010 | 365235768| 4151836672 | 1713761930 |
34359738368 |
| atsqa6c59.qa.lab | 31010 | 409859856| 4151836672 | 1763119827 |
34359738368 |
| atsqa6c60.qa.lab | 31010 | 369571576| 4151836672 | 1759217229 |
34359738368 |
| atsqa6c63.qa.lab | 31010 | 469310224| 4151836672 | 1725239747 |
34359738368 |
| atsqa6c64.qa.lab | 31010 | 471814416| 4151836672 | 1735044144 |
34359738368 |
+++--++++
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Review Request 32987: DRILL-2675: Implement a subset of User Exceptions to improve how errors are reported to the user
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32987/ --- (Updated April 14, 2015, 10:35 p.m.) Review request for drill, Jacques Nadeau, Jason Altekruse, and Parth Chandra. Bugs: DRILL-2675 https://issues.apache.org/jira/browse/DRILL-2675 Repository: drill-git Description --- **INITIAL PATCH** with a working solution. This patch cleans the path for errors, especially user errors with meaningful messages, to be propagated properly to the client. The patch includes changes to 2 existing use cases where the error message was successfully improved. The general idea is: if a code wants to throw an exception that contains a meaningful error message, it throws a DrillUserException. The propagation code will make sure this exception is propagated to the client. The user exception object doesn't contain the final error message, but enough information about the error, the client will use this information to display a better error message. Any exception that is not a DrillUserException (or one of it's subclasses) will be considered as a system exception. For those exceptions the client will only display the error id and drillbit identity in case the user wants to check the logs for more informations about the error. Error objects sent to the client will still contain a stack trace that can be used to display more information if the client has enabled the error verbose mode. Diffs - common/src/main/java/org/apache/drill/common/exceptions/ErrorHelper.java PRE-CREATION common/src/main/java/org/apache/drill/common/exceptions/UserException.java PRE-CREATION common/src/main/java/org/apache/drill/common/exceptions/UserExceptionContext.java PRE-CREATION common/src/main/java/org/apache/drill/common/exceptions/UserRemoteException.java PRE-CREATION common/src/test/java/org/apache/drill/common/exceptions/TestUserException.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java 579cf7d exec/java-exec/src/main/java/org/apache/drill/exec/client/PrintingResultsListener.java 98948af exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java da2229c exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DrillSqlWorker.java b98778d exec/java-exec/src/main/java/org/apache/drill/exec/rpc/CoordinationQueue.java 0016d6a exec/java-exec/src/main/java/org/apache/drill/exec/rpc/RemoteRpcException.java 14ea873 exec/java-exec/src/main/java/org/apache/drill/exec/rpc/RpcBus.java b974963 exec/java-exec/src/main/java/org/apache/drill/exec/rpc/user/QueryResultHandler.java a1be83b exec/java-exec/src/main/java/org/apache/drill/exec/rpc/user/UserResultsListener.java 934a094 exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/QueryWrapper.java fbbf0b8 exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/json/JSONRecordReader.java cc7cb83 exec/java-exec/src/main/java/org/apache/drill/exec/work/ErrorHelper.java 0773d6c exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java 23ef0d3 exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/QueryManager.java 8626d5b exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/SqlUnsupportedException.java 2299afa exec/java-exec/src/main/java/org/apache/drill/exec/work/fragment/AbstractStatusReporter.java 1b0885d exec/java-exec/src/main/java/org/apache/drill/exec/work/fragment/FragmentExecutor.java a7e6c46 exec/java-exec/src/main/java/org/apache/drill/exec/work/fragment/StatusReporter.java 26b5d68 exec/java-exec/src/test/java/org/apache/drill/BaseTestQuery.java 0c2f0e5 exec/java-exec/src/test/java/org/apache/drill/SingleRowListener.java 5703bf9 exec/java-exec/src/test/java/org/apache/drill/TestBugFixes.java 875fb25 exec/java-exec/src/test/java/org/apache/drill/TestDisabledFunctionality.java f62f060 exec/java-exec/src/test/java/org/apache/drill/TestStarQueries.java effef9b exec/java-exec/src/test/java/org/apache/drill/TestUnionAll.java 11d83f9 exec/java-exec/src/test/java/org/apache/drill/exec/server/TestDrillbitResilience.java e03098a exec/java-exec/src/test/java/org/apache/drill/exec/store/parquet/ParquetResultListener.java 55f0d75 exec/java-exec/src/test/java/org/apache/drill/exec/store/parquet/TestParquetPhysicalPlan.java 882cdbd exec/jdbc/src/main/java/org/apache/drill/jdbc/DrillCursor.java d666d06 exec/jdbc/src/main/java/org/apache/drill/jdbc/impl/DrillResultSetImpl.java 24ef62b protocol/src/main/java/org/apache/drill/exec/proto/SchemaUserBitShared.java f72d5e1 protocol/src/main/java/org/apache/drill/exec/proto/UserBitShared.java 96a921b protocol/src/main/java/org/apache/drill/exec/proto/beans/DrillPBError.java ac9cef5
[jira] [Resolved] (DRILL-2781) Protobuf changes for nested loop join
[ https://issues.apache.org/jira/browse/DRILL-2781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mehant Baid resolved DRILL-2781. Resolution: Fixed fixed in 5441e72c0d97e8ccd7c196f5a9f6f23fdc8d2b32 Protobuf changes for nested loop join - Key: DRILL-2781 URL: https://issues.apache.org/jira/browse/DRILL-2781 Project: Apache Drill Issue Type: Bug Reporter: Mehant Baid Assignee: Mehant Baid Priority: Minor Fix For: 0.9.0 Attachments: DRILL-2781.patch A couple of the protobuf files were not regenerated as part of the nested loop join change. Will regenerate the couple of files and merge as part of this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (DRILL-2789) Query fails when text-specific syntax is used on non-text formats.
Abhishek Girish created DRILL-2789:
--
Summary: Query fails when text-specific syntax is used on non-text
formats.
Key: DRILL-2789
URL: https://issues.apache.org/jira/browse/DRILL-2789
Project: Apache Drill
Issue Type: Bug
Components: Query Planning Optimization
Affects Versions: 0.9.0
Reporter: Abhishek Girish
Assignee: Jinfeng Ni
When a user specifies a query which is valid on Text formats, but invalid on
other formats, the query fails with an IllegalArgumentException, which is not
clear.
*Parquet:*
{code:sql}
select * from catalog_sales where columns[3] is not null limit 1;
Query failed: RemoteRpcException: Failure while running fragment., You tried to
read a [RepeatedInt] type when you are using a field reader of type
[NullableIntReaderImpl]. [ 8e105de0-8afa-47fb-a340-5a736ad92d80 on
abhi8.qa.lab:31010 ]
[ 8e105de0-8afa-47fb-a340-5a736ad92d80 on abhi8.qa.lab:31010 ]
Error: exception while executing query: Failure while executing query.
(state=,code=0)
{code}
Log:
{code}
2015-04-14 13:43:19,068 [2ad28418-b2fe-56f2-9c55-15fde67f8730:frag:1:0] ERROR
o.a.d.e.w.f.AbstractStatusReporter - Error
8e105de0-8afa-47fb-a340-5a736ad92d80: Failure while running fragment.
java.lang.IllegalArgumentException: You tried to read a [RepeatedInt] type when
you are using a field reader of type [NullableIntReaderImpl].
at
org.apache.drill.exec.vector.complex.impl.AbstractFieldReader.fail(AbstractFieldReader.java:787)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.vector.complex.impl.AbstractFieldReader.read(AbstractFieldReader.java:361)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.vector.complex.impl.NullableIntReaderImpl.read(NullableIntReaderImpl.java:87)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.test.generated.FiltererGen14.doEval(FilterTemplate2.java:23)
~[na:na]
at
org.apache.drill.exec.test.generated.FiltererGen14.filterBatchNoSV(FilterTemplate2.java:98)
~[na:na]
at
org.apache.drill.exec.test.generated.FiltererGen14.filterBatch(FilterTemplate2.java:71)
~[na:na]
at
org.apache.drill.exec.physical.impl.filter.FilterRecordBatch.doWork(FilterRecordBatch.java:82)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:93)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:142)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:118)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:99)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:89)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext(RemovingRecordBatch.java:96)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:142)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:118)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:99)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:89)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:135)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:142)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
[jira] [Created] (DRILL-2794) Partition pruning is not happening correctly when maxdir/mindir is used in the filter condition
Victoria Markman created DRILL-2794:
---
Summary: Partition pruning is not happening correctly when
maxdir/mindir is used in the filter condition
Key: DRILL-2794
URL: https://issues.apache.org/jira/browse/DRILL-2794
Project: Apache Drill
Issue Type: Bug
Components: Query Planning Optimization
Affects Versions: 0.9.0
Reporter: Victoria Markman
Assignee: Jinfeng Ni
Directory structure:
{code}
[Tue Apr 14 13:43:54 root@/mapr/vmarkman.cluster.com/test/smalltable ] # ls -R
.:
2014 2015 2016
./2014:
./2015:
01 02
./2015/01:
t1.csv
./2015/02:
t2.csv
./2016:
t1.csv
[Tue Apr 14 13:44:26 root@/mapr/vmarkman.cluster.com/test/bigtable ] # ls -R
.:
2015 2016
./2015:
01 02 03 04
./2015/01:
0_0_0.parquet 1_0_0.parquet 2_0_0.parquet 3_0_0.parquet 4_0_0.parquet
5_0_0.parquet
./2015/02:
0_0_0.parquet
./2015/03:
0_0_0.parquet
./2015/04:
0_0_0.parquet
./2016:
01 parquet.file
./2016/01:
0_0_0.parquet
{code}
Simple case, partition pruning is happening correctly: only 2016 directory is
scanned from 'smalltable'.
{code}
0: jdbc:drill:schema=dfs explain plan for select * from smalltable where dir0
= maxdir('dfs.test', 'bigtable');
+++
|text|json|
+++
| 00-00Screen
00-01 Project(*=[$0])
00-02Project(*=[$0])
00-03 Scan(groupscan=[EasyGroupScan [selectionRoot=/test/smalltable,
numFiles=1, columns=[`*`], files=[maprfs:/test/smalltable/2016/t1.csv]]])
| {
head : {
version : 1,
generator : {
type : ExplainHandler,
info :
},
type : APACHE_DRILL_PHYSICAL,
options : [ ],
queue : 0,
resultMode : EXEC
},
graph : [ {
pop : fs-scan,
@id : 3,
files : [ maprfs:/test/smalltable/2016/t1.csv ],
storage : {
type : file,
enabled : true,
connection : maprfs:///,
workspaces : {
root : {
location : /,
writable : false,
defaultInputFormat : null
},
...
...
{code}
With added second predicate (dir1 = mindir('dfs.test', 'bigtable/2016') which
evaluates to false (there is no directory '01' in smalltable)
we end up scanning everything in the smalltable. This does not look right to me
and I think this is a bug.
{code}
0: jdbc:drill:schema=dfs explain plan for select * from smalltable where dir0
= maxdir('dfs.test', 'bigtable') and dir1 = mindir('dfs.test', 'bigtable/2016');
+++
|text|json|
+++
| 00-00Screen
00-01 Project(*=[$0])
00-02Project(T15¦¦*=[$0])
00-03 SelectionVectorRemover
00-04Filter(condition=[AND(=($1, '2016'), =($2, '01'))])
00-05 Project(T15¦¦*=[$0], dir0=[$1], dir1=[$2])
00-06Scan(groupscan=[EasyGroupScan
[selectionRoot=/test/smalltable, numFiles=3, columns=[`*`],
files=[maprfs:/test/smalltable/2015/01/t1.csv,
maprfs:/test/smalltable/2015/02/t2.csv, maprfs:/test/smalltable/2016/t1.csv]]])
| {
head : {
version : 1,
generator : {
type : ExplainHandler,
info :
},
type : APACHE_DRILL_PHYSICAL,
options : [ ],
queue : 0,
resultMode : EXEC
},
graph : [ {
pop : fs-scan,
@id : 6,
files : [ maprfs:/test/smalltable/2015/01/t1.csv,
maprfs:/test/smalltable/2015/02/t2.csv, maprfs:/test/smalltable/2016/t1.csv
],
storage : {
type : file,
enabled : true,
connection : maprfs:///,
workspaces : {
root : {
location : /,
writable : false,
defaultInputFormat : null
},
...
...
{code}
Here is a similar example with parquet file where predicate a1=11 evaluates
to false.
{code}
0: jdbc:drill:schema=dfs explain plan for select * from bigtable where
dir0=maxdir('dfs.test','bigtable') and a1 = 11;
+++
|text|json|
+++
| 00-00Screen
00-01 Project(*=[$0])
00-02Project(T25¦¦*=[$0])
00-03 SelectionVectorRemover
00-04Filter(condition=[AND(=($1, '2016'), =($2, 11))])
00-05 Project(T25¦¦*=[$0], dir0=[$1], a1=[$2])
00-06Scan(groupscan=[ParquetGroupScan
[entries=[ReadEntryWithPath [path=maprfs:/test/bigtable/2016/01/0_0_0.parquet],
ReadEntryWithPath [path=maprfs:/test/bigtable/2016/parquet.file]],
selectionRoot=/test/bigtable, numFiles=2, columns=[`*`]]])
| {
head : {
version : 1,
generator : {
type : ExplainHandler,
info :
},
type : APACHE_DRILL_PHYSICAL,
options : [ ],
queue : 0,
resultMode : EXEC
},
graph : [ {
pop : parquet-scan,
@id : 6,
entries : [ {
path : maprfs:/test/bigtable/2016/01/0_0_0.parquet
}, {
path : maprfs:/test/bigtable/2016/parquet.file
} ],
{code}
And finally, when we use the same
Re: Review Request 32987: DRILL-2675: Implement a subset of User Exceptions to improve how errors are reported to the user
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32987/#review80136 --- common/src/main/java/org/apache/drill/common/exceptions/ErrorHelper.java https://reviews.apache.org/r/32987/#comment129910 maybe I should rename this to: getExceptionContext(Throwable) any suggestations ? - abdelhakim deneche On April 14, 2015, 10:35 p.m., abdelhakim deneche wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32987/ --- (Updated April 14, 2015, 10:35 p.m.) Review request for drill, Jacques Nadeau, Jason Altekruse, and Parth Chandra. Bugs: DRILL-2675 https://issues.apache.org/jira/browse/DRILL-2675 Repository: drill-git Description --- **INITIAL PATCH** with a working solution. This patch cleans the path for errors, especially user errors with meaningful messages, to be propagated properly to the client. The patch includes changes to 2 existing use cases where the error message was successfully improved. The general idea is: if a code wants to throw an exception that contains a meaningful error message, it throws a DrillUserException. The propagation code will make sure this exception is propagated to the client. The user exception object doesn't contain the final error message, but enough information about the error, the client will use this information to display a better error message. Any exception that is not a DrillUserException (or one of it's subclasses) will be considered as a system exception. For those exceptions the client will only display the error id and drillbit identity in case the user wants to check the logs for more informations about the error. Error objects sent to the client will still contain a stack trace that can be used to display more information if the client has enabled the error verbose mode. Diffs - common/src/main/java/org/apache/drill/common/exceptions/ErrorHelper.java PRE-CREATION common/src/main/java/org/apache/drill/common/exceptions/UserException.java PRE-CREATION common/src/main/java/org/apache/drill/common/exceptions/UserExceptionContext.java PRE-CREATION common/src/main/java/org/apache/drill/common/exceptions/UserRemoteException.java PRE-CREATION common/src/test/java/org/apache/drill/common/exceptions/TestUserException.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java 579cf7d exec/java-exec/src/main/java/org/apache/drill/exec/client/PrintingResultsListener.java 98948af exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java da2229c exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DrillSqlWorker.java b98778d exec/java-exec/src/main/java/org/apache/drill/exec/rpc/CoordinationQueue.java 0016d6a exec/java-exec/src/main/java/org/apache/drill/exec/rpc/RemoteRpcException.java 14ea873 exec/java-exec/src/main/java/org/apache/drill/exec/rpc/RpcBus.java b974963 exec/java-exec/src/main/java/org/apache/drill/exec/rpc/user/QueryResultHandler.java a1be83b exec/java-exec/src/main/java/org/apache/drill/exec/rpc/user/UserResultsListener.java 934a094 exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/QueryWrapper.java fbbf0b8 exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/json/JSONRecordReader.java cc7cb83 exec/java-exec/src/main/java/org/apache/drill/exec/work/ErrorHelper.java 0773d6c exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java 23ef0d3 exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/QueryManager.java 8626d5b exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/SqlUnsupportedException.java 2299afa exec/java-exec/src/main/java/org/apache/drill/exec/work/fragment/AbstractStatusReporter.java 1b0885d exec/java-exec/src/main/java/org/apache/drill/exec/work/fragment/FragmentExecutor.java a7e6c46 exec/java-exec/src/main/java/org/apache/drill/exec/work/fragment/StatusReporter.java 26b5d68 exec/java-exec/src/test/java/org/apache/drill/BaseTestQuery.java 0c2f0e5 exec/java-exec/src/test/java/org/apache/drill/SingleRowListener.java 5703bf9 exec/java-exec/src/test/java/org/apache/drill/TestBugFixes.java 875fb25 exec/java-exec/src/test/java/org/apache/drill/TestDisabledFunctionality.java f62f060 exec/java-exec/src/test/java/org/apache/drill/TestStarQueries.java effef9b exec/java-exec/src/test/java/org/apache/drill/TestUnionAll.java 11d83f9
[jira] [Created] (DRILL-2783) Abbreviated Flatten syntax for multi-level nested elements
Neeraja created DRILL-2783:
--
Summary: Abbreviated Flatten syntax for multi-level nested elements
Key: DRILL-2783
URL: https://issues.apache.org/jira/browse/DRILL-2783
Project: Apache Drill
Issue Type: Improvement
Components: Storage - JSON
Affects Versions: 0.7.0
Reporter: Neeraja
Assignee: Steven Phillips
Fix For: Future
The flatten function in Drill works great for any level of nesting, however the
applying nested flatten can complicate query syntax as the # of nested levels
increase due to the need to use subqueries, especially when the structure
contains maps of repeated lists.
Consider JSON below
{
Id:0,
level1:[
{
Id1:1,
level2:[
{
Id2:2,
level3:[
{
Id3:3,
Num:1
}
]
}
]
}
]
}
Below is a query we need to write to flatten the 3 levels.
select count(t2.td.id3) from
(select flatten(t1.tp.level3) as td from
(select flatten(t.ts.level2) as tp from
(select flatten(level1) as ts from
dfs.`default`.`/Users/nrentachintala/Downloads/flattenex.json`) as t) as t1) as
t2;
An abbreviated/intuitive syntax to apply flatten on maps with arrays would make
working with this kind of data much easier.
For ex:
The above query could potentially be rewritten as below.
select flatten(flatten(flatten(level1).level2).level3) as ts from
dfs.`default`.`/Users/nrentachintala/Downloads/flattenex.json`
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
