[jira] [Updated] (DRILL-7174) Expose complex to Json control in the Drill C++ Client
[ https://issues.apache.org/jira/browse/DRILL-7174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Volodymyr Vysotskyi updated DRILL-7174: --- Labels: ready-to-commit (was: ) > Expose complex to Json control in the Drill C++ Client > -- > > Key: DRILL-7174 > URL: https://issues.apache.org/jira/browse/DRILL-7174 > Project: Apache Drill > Issue Type: Task >Reporter: Rob Wu >Priority: Minor > Labels: ready-to-commit > Fix For: 1.17.0 > > > Arjun Gupta will be supplying a patch for this > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (DRILL-7388) Apache Drill fails to return results for partitions containing single offset record
[ https://issues.apache.org/jira/browse/DRILL-7388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937735#comment-16937735 ] Volodymyr Vysotskyi commented on DRILL-7388: [~atlassdan], could you please add more details to this Jira? For example, please share steps to reproduce this issue, data on which it is observed and expected behavior. > Apache Drill fails to return results for partitions containing single offset > record > --- > > Key: DRILL-7388 > URL: https://issues.apache.org/jira/browse/DRILL-7388 > Project: Apache Drill > Issue Type: Bug >Reporter: daniel kelly >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (DRILL-7376) Drill ignores Hive schema for MaprDB tables when group scan has star column
Volodymyr Vysotskyi created DRILL-7376: -- Summary: Drill ignores Hive schema for MaprDB tables when group scan has star column Key: DRILL-7376 URL: https://issues.apache.org/jira/browse/DRILL-7376 Project: Apache Drill Issue Type: Bug Affects Versions: 1.17.0 Reporter: Volodymyr Vysotskyi Assignee: Volodymyr Vysotskyi Fix For: 1.17.0 For the case when group scan has star column, fix for DRILL-7369 doesn't work and hive schema is ignored. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (DRILL-7372) MethodAnalyzer consumes too many memory
Volodymyr Vysotskyi created DRILL-7372:
--
Summary: MethodAnalyzer consumes too many memory
Key: DRILL-7372
URL: https://issues.apache.org/jira/browse/DRILL-7372
Project: Apache Drill
Issue Type: Bug
Affects Versions: 1.16.0
Reporter: Volodymyr Vysotskyi
Assignee: Volodymyr Vysotskyi
Fix For: 1.17.0
In the scope of DRILL-6524 was added logic for determining whether a variable
is assigned in conditional block to prevent incorrect scalar replacement for
such cases.
But for some queries, this logic consumes too many memory, for example, for the
following query:
{code:sql}
SELECT *
FROM cp.`employee.json`
WHERE employee_id+0 < employee_id
OR employee_id+1 < employee_id
AND employee_id+2 < employee_id
OR employee_id+3 < employee_id
AND employee_id+4 < employee_id
OR employee_id+5 < employee_id
AND employee_id+6 < employee_id
OR employee_id+7 < employee_id
AND employee_id+8 < employee_id
OR employee_id+9 < employee_id
AND employee_id+10 < employee_id
OR employee_id+11 < employee_id
AND employee_id+12 < employee_id
OR employee_id+13 < employee_id
AND employee_id+14 < employee_id
OR employee_id+15 < employee_id
AND employee_id+16 < employee_id
OR employee_id+17 < employee_id
AND employee_id+18 < employee_id
OR employee_id+19 < employee_id
AND employee_id+20 < employee_id
OR employee_id+21 < employee_id
AND employee_id+22 < employee_id
OR employee_id+23 < employee_id
AND employee_id+24 < employee_id
OR employee_id+25 < employee_id
AND employee_id+26 < employee_id
OR employee_id+27 < employee_id
AND employee_id+28 < employee_id
OR employee_id+29 < employee_id
AND employee_id+30 < employee_id
OR employee_id+31 < employee_id
AND employee_id+32 < employee_id
OR employee_id+33 < employee_id
AND employee_id+34 < employee_id
OR employee_id+35 < employee_id
AND employee_id+36 < employee_id
OR employee_id+37 < employee_id
AND employee_id+38 < employee_id
OR employee_id+39 < employee_id
AND employee_id+40 < employee_id
OR employee_id+41 < employee_id
AND employee_id+42 < employee_id
OR employee_id+43 < employee_id
AND employee_id+44 < employee_id
OR employee_id+45 < employee_id
AND employee_id+46 < employee_id
OR employee_id+47 < employee_id
AND employee_id+48 < employee_id
OR employee_id+49 < employee_id
AND TRUE;
{code}
Drill consumes more than 6 GB memory.
One of the issues to fix is to replace {{Deque>
localVariablesSet;}} with {{Deque}}, it will reduce memory usage
significantly.
Additionally should be investigated why these objects cannot be collected by GC.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (DRILL-7371) DST/UTC cast/to_timestamp problem
[
https://issues.apache.org/jira/browse/DRILL-7371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927480#comment-16927480
]
Volodymyr Vysotskyi commented on DRILL-7371:
[~benj641], I tried to reproduce this issue, but Drill returns correct results
for queries you have specified.
The locale was set as you have recommended:
{noformat}
localectl
System Locale: LC_TIME=fr_FR.UTF-8
{noformat}
And Drill was configured in UTC.
Could you please provide more info? Is it possible that this issue is
reproduced for specific date ranges? In this case, please provide a date which
should be set on the system.
> DST/UTC cast/to_timestamp problem
> -
>
> Key: DRILL-7371
> URL: https://issues.apache.org/jira/browse/DRILL-7371
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.16.0
>Reporter: benj
>Priority: Major
>
> With LC_TIME=fr_FR.UTF-8 and +drillbits configured in UTC+ (like specified in
> [http://www.openkb.info/2015/05/understanding-drills-timestamp-and.html#.VUzhotpVhHw]
> find from [https://drill.apache.org/docs/data-type-conversion/#to_timestamp])
> {code:sql}
> SELECT TIMEOFDAY();
> +-+
> | EXPR$0|
> +-+
> | 2019-09-11 08:20:12.247 UTC |
> +-+
> {code}
> Problems appears when _cast/to_timestamp_ date (date related to the DST
> (Daylight Save Time) of some countries).
> To illustrate, all the next requests give the same +wrong+ results:
> {code:sql}
> SELECT to_timestamp('2018-03-25 02:22:40 UTC','-MM-dd HH:mm:ss z');
> SELECT to_timestamp('2018-03-25 02:22:40','-MM-dd HH:mm:ss');
> SELECT cast('2018-03-25 02:22:40' as timestamp);
> SELECT cast('2018-03-25 02:22:40 +' as timestamp);
> +---+
> |EXPR$0 |
> +---+
> | 2018-03-25 03:22:40.0 |
> +---+
> {code}
> while the result should be "2018-03-25 +02+:22:40.0"
> An UTC date and time in string shouldn't change when casting to UTC timestamp.
> To illustrate, the next requests produce +good+ results:
> {code:java}
> SELECT to_timestamp('2018-03-26 02:22:40 UTC','-MM-dd HH:mm:ss z');
> +---+
> |EXPR$0 |
> +---+
> | 2018-03-26 02:22:40.0 |
> +---+
> SELECT CAST('2018-03-24 02:22:40' AS timestamp);
> +---+
> |EXPR$0 |
> +---+
> | 2018-03-24 02:22:40.0 |
> +---+
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (DRILL-7368) Query from Iceberg Metastore fails if filter column contains null
[ https://issues.apache.org/jira/browse/DRILL-7368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Volodymyr Vysotskyi updated DRILL-7368: --- Labels: ready-to-commit (was: ) > Query from Iceberg Metastore fails if filter column contains null > - > > Key: DRILL-7368 > URL: https://issues.apache.org/jira/browse/DRILL-7368 > Project: Apache Drill > Issue Type: Bug >Reporter: Arina Ielchiieva >Assignee: Arina Ielchiieva >Priority: Major > Labels: ready-to-commit > Fix For: 1.17.0 > > > When querying data from Drill Iceberg Metastore query fails if filter column > contains null. > Problem is in Iceberg implementation - > https://github.com/apache/incubator-iceberg/pull/443 > Fix steps: > upgrade to latest Iceberg commit which includes appropriate fix. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (DRILL-7168) Implement ALTER SCHEMA ADD / REMOVE COLUMN / PROPERTY commands
[
https://issues.apache.org/jira/browse/DRILL-7168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi updated DRILL-7168:
---
Labels: doc-impacting ready-to-commit (was: doc-impacting)
> Implement ALTER SCHEMA ADD / REMOVE COLUMN / PROPERTY commands
> --
>
> Key: DRILL-7168
> URL: https://issues.apache.org/jira/browse/DRILL-7168
> Project: Apache Drill
> Issue Type: Sub-task
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>Priority: Major
> Labels: doc-impacting, ready-to-commit
> Fix For: 1.17.0
>
>
> By [~Paul.Rogers]:
> {quote}
> Sooner or later users are going to ask for a command to update just the
> properties, or just add or remove a column, without having to spell out the
> entire new schema. ALTER TABLE SCHEMA ADD/REMOVE COLUMN/PROPERTY ...
> {quote}
> Syntax:
> {code:sql}
> alter schema
> (for table dfs.tmp.nation | path '/tmp/schema.json')
> add [or replace]
> [columns (col1 int, col2 varchar)]
> [properties ('prop1'='val1', 'prop2'='val2')]
> {code}
> Add command will fail if column or property with the same name exists, unless
> OR REPLACE keywords are indicated.
> Add command will fail, if schema file does not exit.
> {code:sql}
> alter schema
> (for table dfs.tmp.nation | path '/tmp/schema.json')
> remove
> [columns (`col1`, `col2`)]
> [properties ('prop1', 'prop2')]
> {code}
> Remove command won't fail if column or property does not exist but will fail
> if schema file is absent.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (DRILL-7369) Schema for MaprDB tables is not used for the case when several fields are queried
[ https://issues.apache.org/jira/browse/DRILL-7369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Volodymyr Vysotskyi updated DRILL-7369: --- Reviewer: Arina Ielchiieva > Schema for MaprDB tables is not used for the case when several fields are > queried > - > > Key: DRILL-7369 > URL: https://issues.apache.org/jira/browse/DRILL-7369 > Project: Apache Drill > Issue Type: Bug > Components: Storage - MapRDB >Affects Versions: 1.17.0 >Reporter: Volodymyr Vysotskyi >Assignee: Volodymyr Vysotskyi >Priority: Major > Fix For: 1.17.0 > > > In DRILL-7313 was allowed using Hive schema for MaprDB native reader when > field was empty. But current code creates additional fields from schema only > for the case when the incoming map is empty. > The aim of this Jira is to allow using provided schema for the case when some > fields already present. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (DRILL-7369) Schema for MaprDB tables is not used for the case when several fields are queried
[ https://issues.apache.org/jira/browse/DRILL-7369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Volodymyr Vysotskyi updated DRILL-7369: --- Summary: Schema for MaprDB tables is not used for the case when several fields are queried (was: Schema for MaprDB tables is not used for the case when several field are queried) > Schema for MaprDB tables is not used for the case when several fields are > queried > - > > Key: DRILL-7369 > URL: https://issues.apache.org/jira/browse/DRILL-7369 > Project: Apache Drill > Issue Type: Bug > Components: Storage - MapRDB >Affects Versions: 1.17.0 >Reporter: Volodymyr Vysotskyi >Assignee: Volodymyr Vysotskyi >Priority: Major > Fix For: 1.17.0 > > > In DRILL-7313 was allowed using Hive schema for MaprDB native reader when > field was empty. But current code creates additional fields from schema only > for the case when the incoming map is empty. > The aim of this Jira is to allow using provided schema for the case when some > fields already present. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (DRILL-7369) Schema for MaprDB tables is not used for the case when several field are queried
[ https://issues.apache.org/jira/browse/DRILL-7369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Volodymyr Vysotskyi updated DRILL-7369: --- Summary: Schema for MaprDB tables is not used for the case when several field are queried (was: Schema for MaprDB tables is not used for the case when some field are queried) > Schema for MaprDB tables is not used for the case when several field are > queried > > > Key: DRILL-7369 > URL: https://issues.apache.org/jira/browse/DRILL-7369 > Project: Apache Drill > Issue Type: Bug > Components: Storage - MapRDB >Affects Versions: 1.17.0 >Reporter: Volodymyr Vysotskyi >Assignee: Volodymyr Vysotskyi >Priority: Major > Fix For: 1.17.0 > > > In DRILL-7313 was allowed using Hive schema for MaprDB native reader when > field was empty. But current code creates additional fields from schema only > for the case when the incoming map is empty. > The aim of this Jira is to allow using provided schema for the case when some > fields already present. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (DRILL-7369) Schema for MaprDB tables is not used for the case when some field are queried
Volodymyr Vysotskyi created DRILL-7369: -- Summary: Schema for MaprDB tables is not used for the case when some field are queried Key: DRILL-7369 URL: https://issues.apache.org/jira/browse/DRILL-7369 Project: Apache Drill Issue Type: Bug Components: Storage - MapRDB Affects Versions: 1.17.0 Reporter: Volodymyr Vysotskyi Assignee: Volodymyr Vysotskyi Fix For: 1.17.0 In DRILL-7313 was allowed using Hive schema for MaprDB native reader when field was empty. But current code creates additional fields from schema only for the case when the incoming map is empty. The aim of this Jira is to allow using provided schema for the case when some fields already present. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (DRILL-7367) Remove Server details from response headers
[
https://issues.apache.org/jira/browse/DRILL-7367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi updated DRILL-7367:
---
Labels: ready-to-commit (was: )
> Remove Server details from response headers
> ---
>
> Key: DRILL-7367
> URL: https://issues.apache.org/jira/browse/DRILL-7367
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.16.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.17.0
>
>
> Drill response headers include Server information which is considered to be a
> vulnerability.
> {noformat}
> curl http://localhost:8047/cluster.json -v -k
> * Trying ::1...
> * TCP_NODELAY set
> * Connected to localhost (::1) port 8047 (#0)
> > GET /cluster.json HTTP/1.1
> > Host: localhost:8047
> > User-Agent: curl/7.54.0
> > Accept: */*
> >
> < HTTP/1.1 200 OK
> < Date: Thu, 05 Sep 2019 12:47:53 GMT
> < Content-Type: application/json
> < Content-Length: 436
> < Server: Jetty(9.3.25.v20180904)
> ...
> {noformat}
> https://pentest-tools.com/blog/essential-http-security-headers/
> After the fix headers should be without server information:
> {noformat}
> curl http://localhost:8047/cluster.json -v -k
> * Trying ::1...
> * TCP_NODELAY set
> * Connected to localhost (::1) port 8047 (#0)
> > GET /cluster.json HTTP/1.1
> > Host: localhost:8047
> > User-Agent: curl/7.54.0
> > Accept: */*
> >
> < HTTP/1.1 200 OK
> < Date: Thu, 05 Sep 2019 13:55:25 GMT
> < Content-Type: application/json
> < Content-Length: 436
> ...
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (DRILL-7362) COUNT(*) on JSON with outer list results in JsonParse error
[ https://issues.apache.org/jira/browse/DRILL-7362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Volodymyr Vysotskyi updated DRILL-7362: --- Labels: ready-to-commit (was: ) > COUNT(*) on JSON with outer list results in JsonParse error > --- > > Key: DRILL-7362 > URL: https://issues.apache.org/jira/browse/DRILL-7362 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.16.0 >Reporter: Oleg Zinoviev >Assignee: Oleg Zinoviev >Priority: Major > Labels: ready-to-commit > Fix For: 1.17.0 > > > Count from a JSON file with a outer array results in JsonParseException: > Cannot read from the middle of a record. Current token was START_ARRAY. > P.S. A simple select from such file works normally. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (DRILL-7096) Develop vector for canonical Map
[ https://issues.apache.org/jira/browse/DRILL-7096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Volodymyr Vysotskyi updated DRILL-7096: --- Labels: ready-to-commit (was: ) > Develop vector for canonical Map > - > > Key: DRILL-7096 > URL: https://issues.apache.org/jira/browse/DRILL-7096 > Project: Apache Drill > Issue Type: Sub-task >Reporter: Igor Guzenko >Assignee: Bohdan Kazydub >Priority: Major > Labels: ready-to-commit > Fix For: 1.17.0 > > > Canonical Map datatype can be represented using combination of three > value vectors: > keysVector - vector for storing keys of each map > valuesVector - vector for storing values of each map > offsetsVector - vector for storing of start indexes of next each map > So it's not very hard to create such Map vector, but there is a major issue > with such map representation. It's hard to search maps values by key in such > vector, need to investigate some advanced techniques to make such search > efficient. Or find other more suitable options to represent map datatype in > world of vectors. > After question about maps, Apache Arrow developers responded that for Java > they don't have real Map vector, for now they just have logical Map type > definition where they define Map like: List< Struct value:value_type> >. So implementation of value vector would be useful for > Arrow too. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (DRILL-7329) Implement metadata usage for Parquet format plugin
[ https://issues.apache.org/jira/browse/DRILL-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Volodymyr Vysotskyi updated DRILL-7329: --- Description: 1. Implement retrieval steps (include auto-refresh policy, retry attempts, fallback to file metadata). 2. Change the current group scan to leverage Schema from Metastore; 3. Verify that metadata is used correctly for metastore classes Add options: metastore.enabled metastore.retrival.retry_attempts metastore.metadata.store.depth_level metastore.metadata.use_schema metastore.metadata.use_statistics metastore.metadata.ctas.auto-collect metastore.metadata.fallback_to_file_metadata was: 1. Implement retrieval steps (include auto-refresh policy, retry attempts, fallback to file metadata). 2. Change the current group scan to leverage Schema from Metastore; 3. Verify that metadata is used correctly for metastore classes Add options: planner.metadata.use_schema planner.metadata.use_statistics metadata.ctas.auto_collect metadata.fallback_to_file_metadata metastore.auto-refresh > Implement metadata usage for Parquet format plugin > -- > > Key: DRILL-7329 > URL: https://issues.apache.org/jira/browse/DRILL-7329 > Project: Apache Drill > Issue Type: Sub-task >Reporter: Arina Ielchiieva >Assignee: Volodymyr Vysotskyi >Priority: Major > Fix For: 1.17.0 > > > 1. Implement retrieval steps (include auto-refresh policy, retry attempts, > fallback to file metadata). > 2. Change the current group scan to leverage Schema from Metastore; > 3. Verify that metadata is used correctly for metastore classes > Add options: > metastore.enabled > metastore.retrival.retry_attempts > metastore.metadata.store.depth_level > metastore.metadata.use_schema > metastore.metadata.use_statistics > metastore.metadata.ctas.auto-collect > metastore.metadata.fallback_to_file_metadata -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (DRILL-7360) Refactor WatchService code in Drillbit class
[ https://issues.apache.org/jira/browse/DRILL-7360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Volodymyr Vysotskyi updated DRILL-7360: --- Labels: ready-to-commit (was: ) > Refactor WatchService code in Drillbit class > > > Key: DRILL-7360 > URL: https://issues.apache.org/jira/browse/DRILL-7360 > Project: Apache Drill > Issue Type: Task >Affects Versions: 1.16.0 >Reporter: Arina Ielchiieva >Assignee: Arina Ielchiieva >Priority: Major > Labels: ready-to-commit > Fix For: 1.17.0 > > > Refactor WatchService to user proper code (see > https://docs.oracle.com/javase/tutorial/essential/io/notification.html for > details) in Drillbit class and fix concurrency issues connected with > variables assigning from different thread. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (DRILL-7273) Create operator for handling metadata
[ https://issues.apache.org/jira/browse/DRILL-7273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Volodymyr Vysotskyi updated DRILL-7273: --- Description: As described in the design document for [File metastore table metadata|https://docs.google.com/document/d/14pSIzKqDltjLEEpEebwmKnsDPxyS_6jGrPOjXu6M_NM/edit#heading=h.j079wdhtehuk] it is required to create an operator for handling metadata to be able to do incremental analyze. Add options: metastore.enabled metastore.metadata.store.depth_level Statement: ANALYZE TABLE REFRESH METADATA ... was: As described in the design document for [File metastore table metadata|https://docs.google.com/document/d/14pSIzKqDltjLEEpEebwmKnsDPxyS_6jGrPOjXu6M_NM/edit#heading=h.j079wdhtehuk] it is required to create an operator for handling metadata to be able to do incremental analyze. Add options: metastore.enabled metadata.store.depth_level Statement: ANALYZE TABLE REFRESH METADATA ... > Create operator for handling metadata > - > > Key: DRILL-7273 > URL: https://issues.apache.org/jira/browse/DRILL-7273 > Project: Apache Drill > Issue Type: Sub-task >Reporter: Arina Ielchiieva >Assignee: Volodymyr Vysotskyi >Priority: Major > Labels: doc-impacting > Fix For: 1.17.0 > > > As described in the design document for [File metastore table > metadata|https://docs.google.com/document/d/14pSIzKqDltjLEEpEebwmKnsDPxyS_6jGrPOjXu6M_NM/edit#heading=h.j079wdhtehuk] > it is required to create an operator for handling metadata to be able to do > incremental analyze. > Add options: > metastore.enabled > metastore.metadata.store.depth_level > Statement: > ANALYZE TABLE REFRESH METADATA ... -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (DRILL-7356) Introduce session options for the Drill Metastore
[
https://issues.apache.org/jira/browse/DRILL-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi updated DRILL-7356:
---
Parent: DRILL-6552
Issue Type: Sub-task (was: Task)
> Introduce session options for the Drill Metastore
> -
>
> Key: DRILL-7356
> URL: https://issues.apache.org/jira/browse/DRILL-7356
> Project: Apache Drill
> Issue Type: Sub-task
>Affects Versions: 1.17.0
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.17.0
>
>
> Introduce a new session option for enabling metastore: {{metastore.enabled}}.
> It will be used during the next steps of development.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (DRILL-7356) Introduce new session option for enabling metastore
[
https://issues.apache.org/jira/browse/DRILL-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi updated DRILL-7356:
---
Reviewer: Arina Ielchiieva
> Introduce new session option for enabling metastore
> ---
>
> Key: DRILL-7356
> URL: https://issues.apache.org/jira/browse/DRILL-7356
> Project: Apache Drill
> Issue Type: Task
>Affects Versions: 1.17.0
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.17.0
>
>
> Introduce a new session option for enabling metastore: {{metastore.enabled}}.
> It will be used during the next steps of development.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Created] (DRILL-7356) Introduce new session option for enabling metastore
Volodymyr Vysotskyi created DRILL-7356:
--
Summary: Introduce new session option for enabling metastore
Key: DRILL-7356
URL: https://issues.apache.org/jira/browse/DRILL-7356
Project: Apache Drill
Issue Type: Task
Affects Versions: 1.17.0
Reporter: Volodymyr Vysotskyi
Assignee: Volodymyr Vysotskyi
Fix For: 1.17.0
Introduce a new session option for enabling metastore: {{metastore.enabled}}.
It will be used during the next steps of development.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (DRILL-7350) Move RowSet related classes from test folder
[ https://issues.apache.org/jira/browse/DRILL-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Volodymyr Vysotskyi updated DRILL-7350: --- Labels: ready-to-commit (was: ) > Move RowSet related classes from test folder > > > Key: DRILL-7350 > URL: https://issues.apache.org/jira/browse/DRILL-7350 > Project: Apache Drill > Issue Type: Task >Reporter: Volodymyr Vysotskyi >Assignee: Volodymyr Vysotskyi >Priority: Major > Labels: ready-to-commit > Fix For: 1.17.0 > > > Move RowSet related classes from test folder to main to be able to use them > for Metastore. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (DRILL-7352) Introduce new checkstyle rules to make code style more consistent
Volodymyr Vysotskyi created DRILL-7352: -- Summary: Introduce new checkstyle rules to make code style more consistent Key: DRILL-7352 URL: https://issues.apache.org/jira/browse/DRILL-7352 Project: Apache Drill Issue Type: Task Reporter: Volodymyr Vysotskyi List of rules to be enabled: * [LeftCurly|https://checkstyle.sourceforge.io/config_blocks.html#LeftCurly] - force placement of a left curly brace at the end of the line. * [RightCurly|https://checkstyle.sourceforge.io/config_blocks.html#RightCurly] - force placement of a right curly brace * [NewlineAtEndOfFile|https://checkstyle.sourceforge.io/config_misc.html#NewlineAtEndOfFile] * [UnnecessaryParentheses|https://checkstyle.sourceforge.io/config_coding.html#UnnecessaryParentheses] * [MethodParamPad|https://checkstyle.sourceforge.io/config_whitespace.html#MethodParamPad] and other -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (DRILL-7350) Move RowSet related classes from test folder
Volodymyr Vysotskyi created DRILL-7350: -- Summary: Move RowSet related classes from test folder Key: DRILL-7350 URL: https://issues.apache.org/jira/browse/DRILL-7350 Project: Apache Drill Issue Type: Task Reporter: Volodymyr Vysotskyi Assignee: Volodymyr Vysotskyi Fix For: 1.17.0 Move RowSet related classes from test folder to main to be able to use them for Metastore. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (DRILL-7339) Upgrade to Iceberg latest commits to fix issue with orphan files after delete in transaction
[ https://issues.apache.org/jira/browse/DRILL-7339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Volodymyr Vysotskyi updated DRILL-7339: --- Labels: ready-to-commit (was: ) > Upgrade to Iceberg latest commits to fix issue with orphan files after delete > in transaction > > > Key: DRILL-7339 > URL: https://issues.apache.org/jira/browse/DRILL-7339 > Project: Apache Drill > Issue Type: Task >Reporter: Arina Ielchiieva >Assignee: Arina Ielchiieva >Priority: Major > Labels: ready-to-commit > Fix For: 1.17.0 > > > Drill Metastore executes many operations in transaction including delete. > Currently Iceberg creates orphan files when executing delete in transaction > and these files cannot be expired and keep pilling up. Iceberg issue - > https://github.com/apache/incubator-iceberg/issues/330. When #330 is fixed, > we need to update Iceberg commit to ensure these issue is resolved in Drill > as well. > PR with the fix - https://github.com/apache/incubator-iceberg/pull/352 -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Resolved] (DRILL-7345) Strange Behavior for UDFs with ComplexWriter Output
[ https://issues.apache.org/jira/browse/DRILL-7345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Volodymyr Vysotskyi resolved DRILL-7345. Resolution: Not A Problem Assignee: Volodymyr Vysotskyi Resolving this Jira since the described behavior is working as intended. > Strange Behavior for UDFs with ComplexWriter Output > --- > > Key: DRILL-7345 > URL: https://issues.apache.org/jira/browse/DRILL-7345 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.17.0 >Reporter: Charles Givre >Assignee: Volodymyr Vysotskyi >Priority: Minor > > I wrote some UDFs recently and noticed some strange behavior when debugging > them. > This behavior only occurs when there is ComplexWriter as output. > Basically, if the input to the UDF is nullable, Drill doesn't recognize the > UDF at all. I've found that the only way to get Drill to recognize UDFs that > have ComplexWriters as output is: > * Use a non-nullable holder as input > * Remove the null setting completely from the function parameters. > This approach has a drawback in that if the function receives a null value, > it will throw an error and halt execution. My preference would be to allow > null handling, but I've not figured out how to make that happen. > Note: This behavior ONLY occurs when using a ComplexWriter as output. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (DRILL-7345) Strange Behavior for UDFs with ComplexWriter Output
[ https://issues.apache.org/jira/browse/DRILL-7345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Volodymyr Vysotskyi updated DRILL-7345: --- Fix Version/s: 1.17.0 > Strange Behavior for UDFs with ComplexWriter Output > --- > > Key: DRILL-7345 > URL: https://issues.apache.org/jira/browse/DRILL-7345 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.17.0 >Reporter: Charles Givre >Assignee: Volodymyr Vysotskyi >Priority: Minor > Fix For: 1.17.0 > > > I wrote some UDFs recently and noticed some strange behavior when debugging > them. > This behavior only occurs when there is ComplexWriter as output. > Basically, if the input to the UDF is nullable, Drill doesn't recognize the > UDF at all. I've found that the only way to get Drill to recognize UDFs that > have ComplexWriters as output is: > * Use a non-nullable holder as input > * Remove the null setting completely from the function parameters. > This approach has a drawback in that if the function receives a null value, > it will throw an error and halt execution. My preference would be to allow > null handling, but I've not figured out how to make that happen. > Note: This behavior ONLY occurs when using a ComplexWriter as output. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (DRILL-6810) Disable NULL_IF_NULL NullHandling for functions with ComplexWriter
[ https://issues.apache.org/jira/browse/DRILL-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Volodymyr Vysotskyi updated DRILL-6810: --- Labels: doc-impacting ready-to-commit (was: ready-to-commit) > Disable NULL_IF_NULL NullHandling for functions with ComplexWriter > -- > > Key: DRILL-6810 > URL: https://issues.apache.org/jira/browse/DRILL-6810 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.14.0 >Reporter: Bohdan Kazydub >Assignee: Bohdan Kazydub >Priority: Major > Labels: doc-impacting, ready-to-commit > Fix For: 1.15.0 > > > Currently NullHandling.NULL_IF_NULL is allowed for UDFs with @Output of type > org.apache.drill.exec.vector.complex.writer.BaseWriter.ComplexWriter but no > null handling is performed for the kind of functions which leads to > confusion. The problem is ComplexWriter holds list/map values and Drill does > not yet support NULL values for the types (there is an issue to allow null > maps/lists in [DRILL-4824|https://issues.apache.org/jira/browse/DRILL-4824]). > For such functions support for NULL_IF_NULL will be disabled, as it is done > for aggregate functions, and NullHandling.INTERNAL should be used instead. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (DRILL-7050) RexNode convert exception in subquery
[
https://issues.apache.org/jira/browse/DRILL-7050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16905970#comment-16905970
]
Volodymyr Vysotskyi commented on DRILL-7050:
Hi, [~benj641], I'll try to reproduce this issue in Calcite.
> RexNode convert exception in subquery
> -
>
> Key: DRILL-7050
> URL: https://issues.apache.org/jira/browse/DRILL-7050
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.14.0, 1.15.0
>Reporter: Oleg Zinoviev
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.17.0
>
>
> If the query contains a subquery whose filters are associated with the main
> query, an error occurs: *PLAN ERROR: Cannot convert RexNode to equivalent
> Drill expression. RexNode Class: org.apache.calcite.rex.RexCorrelVariable*
> Steps to reproduce:
> 1) Create source table (or view, doesn't matter)
> {code:sql}
> create table dfs.root.source as (
> select 1 as id union all select 2 as id
> )
> {code}
> 2) Execute query
> {code:sql}
> select t1.id,
> (select count(t2.id)
> from dfs.root.source t2 where t2.id = t1.id)
> from dfs.root.source t1
> {code}
> Reason:
> Method
> {code:java}org.apache.calcite.sql2rel.SqlToRelConverter.Blackboard.lookupExp{code}
> call {code:java}RexBuilder.makeCorrel{code} in some cases
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (DRILL-7345) Strange Behavior for UDFs with ComplexWriter Output
[ https://issues.apache.org/jira/browse/DRILL-7345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16905520#comment-16905520 ] Volodymyr Vysotskyi commented on DRILL-7345: Yes, it should work, the result will be in the same column. > Strange Behavior for UDFs with ComplexWriter Output > --- > > Key: DRILL-7345 > URL: https://issues.apache.org/jira/browse/DRILL-7345 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.17.0 >Reporter: Charles Givre >Priority: Minor > > I wrote some UDFs recently and noticed some strange behavior when debugging > them. > This behavior only occurs when there is ComplexWriter as output. > Basically, if the input to the UDF is nullable, Drill doesn't recognize the > UDF at all. I've found that the only way to get Drill to recognize UDFs that > have ComplexWriters as output is: > * Use a non-nullable holder as input > * Remove the null setting completely from the function parameters. > This approach has a drawback in that if the function receives a null value, > it will throw an error and halt execution. My preference would be to allow > null handling, but I've not figured out how to make that happen. > Note: This behavior ONLY occurs when using a ComplexWriter as output. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (DRILL-7345) Strange Behavior for UDFs with ComplexWriter Output
[
https://issues.apache.org/jira/browse/DRILL-7345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16905511#comment-16905511
]
Volodymyr Vysotskyi commented on DRILL-7345:
{quote}
If I were to create an additional UDF which perhaps accepts a NullableVarChar
as an input parameter, and returns null, wouldn't that cause Drill to either
add extra columns or otherwise cause problems?
{quote}
You can create additional UDF which accepts NullableVarChar as an input
parameter and Drill will choose between them both, which one should be used. A
lot of inbuilt UDFs which uses internal nulls handling use this approach.
> Strange Behavior for UDFs with ComplexWriter Output
> ---
>
> Key: DRILL-7345
> URL: https://issues.apache.org/jira/browse/DRILL-7345
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.17.0
>Reporter: Charles Givre
>Priority: Minor
>
> I wrote some UDFs recently and noticed some strange behavior when debugging
> them.
> This behavior only occurs when there is ComplexWriter as output.
> Basically, if the input to the UDF is nullable, Drill doesn't recognize the
> UDF at all. I've found that the only way to get Drill to recognize UDFs that
> have ComplexWriters as output is:
> * Use a non-nullable holder as input
> * Remove the null setting completely from the function parameters.
> This approach has a drawback in that if the function receives a null value,
> it will throw an error and halt execution. My preference would be to allow
> null handling, but I've not figured out how to make that happen.
> Note: This behavior ONLY occurs when using a ComplexWriter as output.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (DRILL-7345) Strange Behavior for UDFs with ComplexWriter Output
[
https://issues.apache.org/jira/browse/DRILL-7345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16905443#comment-16905443
]
Volodymyr Vysotskyi commented on DRILL-7345:
Thanks, [~IhorHuzenko] for pointing to the Javadocs and specifying Jira ticket
where this Limitation was added.
[~cgivre], as pointed in DRILL-6810, currently Drill does not support NULL
values for list/map so it is incorrect to allow usage of NULL_IF_NULL
NullHandling for functions with ComplexWriter.
But you can create two UDF implementations: one of them accepts nullable values
and another - required and then handle nulls inside UDF in the way you choose -
set default values, return empty lists, etc.
In Jira description, you have written:
{quote}
if the input to the UDF is nullable, Drill doesn't recognize the UDF at all
{quote}
Is it meant that this UDF wasn't used for values with required data mode? In
the opposite case, it may be another issue, but we need to see UDF
implementation to find a root cause.
> Strange Behavior for UDFs with ComplexWriter Output
> ---
>
> Key: DRILL-7345
> URL: https://issues.apache.org/jira/browse/DRILL-7345
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.17.0
>Reporter: Charles Givre
>Priority: Minor
>
> I wrote some UDFs recently and noticed some strange behavior when debugging
> them.
> This behavior only occurs when there is ComplexWriter as output.
> Basically, if the input to the UDF is nullable, Drill doesn't recognize the
> UDF at all. I've found that the only way to get Drill to recognize UDFs that
> have ComplexWriters as output is:
> * Use a non-nullable holder as input
> * Remove the null setting completely from the function parameters.
> This approach has a drawback in that if the function receives a null value,
> it will throw an error and halt execution. My preference would be to allow
> null handling, but I've not figured out how to make that happen.
> Note: This behavior ONLY occurs when using a ComplexWriter as output.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (DRILL-4517) Reading emtpy Parquet file failes with java.lang.IllegalArgumentException
[
https://issues.apache.org/jira/browse/DRILL-4517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi updated DRILL-4517:
---
Labels: doc-impacting ready-to-commit (was: doc-impacting)
> Reading emtpy Parquet file failes with java.lang.IllegalArgumentException
> -
>
> Key: DRILL-4517
> URL: https://issues.apache.org/jira/browse/DRILL-4517
> Project: Apache Drill
> Issue Type: Improvement
> Components: Server
>Reporter: Tobias
>Assignee: Arina Ielchiieva
>Priority: Major
> Labels: doc-impacting, ready-to-commit
> Fix For: 1.17.0
>
> Attachments: empty.parquet, no_rows.parquet
>
>
> When querying a Parquet file that has a schema but no rows the Drill Server
> will fail with the below
> This looks similar to DRILL-3557
> {noformat}
> {{ParquetMetaData{FileMetaData{schema: message TRANSACTION_REPORT {
> required int64 MEMBER_ACCOUNT_ID;
> required int64 TIMESTAMP_IN_HOUR;
> optional int64 APPLICATION_ID;
> }
> , metadata: {}}}, blocks: []}
> {noformat}
> {noformat}
> Caused by: java.lang.IllegalArgumentException: MinorFragmentId 0 has no read
> entries assigned
> at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:92)
> ~[guava-14.0.1.jar:na]
> at
> org.apache.drill.exec.store.parquet.ParquetGroupScan.getSpecificScan(ParquetGroupScan.java:707)
> ~[drill-java-exec-1.5.0.jar:1.5.0]
> at
> org.apache.drill.exec.store.parquet.ParquetGroupScan.getSpecificScan(ParquetGroupScan.java:105)
> ~[drill-java-exec-1.5.0.jar:1.5.0]
> at
> org.apache.drill.exec.planner.fragment.Materializer.visitGroupScan(Materializer.java:68)
> ~[drill-java-exec-1.5.0.jar:1.5.0]
> at
> org.apache.drill.exec.planner.fragment.Materializer.visitGroupScan(Materializer.java:35)
> ~[drill-java-exec-1.5.0.jar:1.5.0]
> at
> org.apache.drill.exec.physical.base.AbstractGroupScan.accept(AbstractGroupScan.java:60)
> ~[drill-java-exec-1.5.0.jar:1.5.0]
> at
> org.apache.drill.exec.planner.fragment.Materializer.visitOp(Materializer.java:102)
> ~[drill-java-exec-1.5.0.jar:1.5.0]
> at
> org.apache.drill.exec.planner.fragment.Materializer.visitOp(Materializer.java:35)
> ~[drill-java-exec-1.5.0.jar:1.5.0]
> at
> org.apache.drill.exec.physical.base.AbstractPhysicalVisitor.visitProject(AbstractPhysicalVisitor.java:77)
> ~[drill-java-exec-1.5.0.jar:1.5.0]
> at
> org.apache.drill.exec.physical.config.Project.accept(Project.java:51)
> ~[drill-java-exec-1.5.0.jar:1.5.0]
> at
> org.apache.drill.exec.planner.fragment.Materializer.visitStore(Materializer.java:82)
> ~[drill-java-exec-1.5.0.jar:1.5.0]
> at
> org.apache.drill.exec.planner.fragment.Materializer.visitStore(Materializer.java:35)
> ~[drill-java-exec-1.5.0.jar:1.5.0]
> at
> org.apache.drill.exec.physical.base.AbstractPhysicalVisitor.visitScreen(AbstractPhysicalVisitor.java:195)
> ~[drill-java-exec-1.5.0.jar:1.5.0]
> at
> org.apache.drill.exec.physical.config.Screen.accept(Screen.java:97)
> ~[drill-java-exec-1.5.0.jar:1.5.0]
> at
> org.apache.drill.exec.planner.fragment.SimpleParallelizer.generateWorkUnit(SimpleParallelizer.java:355)
> ~[drill-java-exec-1.5.0.jar:1.5.0]
> at
> org.apache.drill.exec.planner.fragment.SimpleParallelizer.getFragments(SimpleParallelizer.java:134)
> ~[drill-java-exec-1.5.0.jar:1.5.0]
> at
> org.apache.drill.exec.work.foreman.Foreman.getQueryWorkUnit(Foreman.java:518)
> [drill-java-exec-1.5.0.jar:1.5.0]
> at
> org.apache.drill.exec.work.foreman.Foreman.runPhysicalPlan(Foreman.java:405)
> [drill-java-exec-1.5.0.jar:1.5.0]
> at
> org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:926)
> [drill-java-exec-1.5.0.jar:1.5.0]
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Resolved] (DRILL-7321) split function doesn't work without from
[
https://issues.apache.org/jira/browse/DRILL-7321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi resolved DRILL-7321.
Resolution: Fixed
Fix Version/s: 1.17.0
Fixed in the sope of DRILL-7337.
> split function doesn't work without from
>
>
> Key: DRILL-7321
> URL: https://issues.apache.org/jira/browse/DRILL-7321
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.16.0
>Reporter: benj
>Assignee: Volodymyr Vysotskyi
>Priority: Minor
> Fix For: 1.17.0
>
>
> {code:java}
> SELECT upper('foo') AS a /* OK */;
> +-+
> | a |
> +-+
> | foo |
> +-+
> {code}
> but
> {code:java}
> SELECT split('foo,bar,buz',',') AS a /* NOK */;
> Error: PLAN ERROR: Failure while materializing expression in constant
> expression evaluator [SPLIT('foo,bar,buz', ',')]. Errors:
> Error in expression at index -1. Error: Only ProjectRecordBatch could have
> complex writer function. You are using complex writer function split in a
> non-project operation!. Full expression: --UNKNOWN EXPRESSION--.{code}
> Note that
> {code:java}
> SELECT split(a,',') AS a FROM (SELECT 'foo,bar,buz' AS a) /* OK */;
> +-+
> | a |
> +-+
> | ["foo","bar","buz"] |
> +-+
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Assigned] (DRILL-7321) split function doesn't work without from
[
https://issues.apache.org/jira/browse/DRILL-7321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi reassigned DRILL-7321:
--
Assignee: Volodymyr Vysotskyi
> split function doesn't work without from
>
>
> Key: DRILL-7321
> URL: https://issues.apache.org/jira/browse/DRILL-7321
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.16.0
>Reporter: benj
>Assignee: Volodymyr Vysotskyi
>Priority: Minor
>
> {code:java}
> SELECT upper('foo') AS a /* OK */;
> +-+
> | a |
> +-+
> | foo |
> +-+
> {code}
> but
> {code:java}
> SELECT split('foo,bar,buz',',') AS a /* NOK */;
> Error: PLAN ERROR: Failure while materializing expression in constant
> expression evaluator [SPLIT('foo,bar,buz', ',')]. Errors:
> Error in expression at index -1. Error: Only ProjectRecordBatch could have
> complex writer function. You are using complex writer function split in a
> non-project operation!. Full expression: --UNKNOWN EXPRESSION--.{code}
> Note that
> {code:java}
> SELECT split(a,',') AS a FROM (SELECT 'foo,bar,buz' AS a) /* OK */;
> +-+
> | a |
> +-+
> | ["foo","bar","buz"] |
> +-+
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Created] (DRILL-7340) Filter is not pushed to JDBC database when several databases are used in the query
Volodymyr Vysotskyi created DRILL-7340:
--
Summary: Filter is not pushed to JDBC database when several
databases are used in the query
Key: DRILL-7340
URL: https://issues.apache.org/jira/browse/DRILL-7340
Project: Apache Drill
Issue Type: Bug
Components: Storage - JDBC
Affects Versions: 1.16.0
Reporter: Volodymyr Vysotskyi
Fix For: Future
For the case when several databases are used in the query, some rules weren't
added to the rule set for one of the conventions. It is observed in queries
similar to the next query:
{code:sql}
select * from mysql.`drill_mysql_test`.person t1
INNER JOIN h2.drill_h2_test.person t2 on t1.person_id = t2.person_id where
t1.first_name = 'first_name_1' and t2.last_name = 'last_name_1
{code}
Plan for this query is the following:
{noformat}
00-00Screen
00-01 Project(person_id=[$0], first_name=[$1], last_name=[$2],
address=[$3], city=[$4], state=[$5], zip=[$6], json=[$7], bigint_field=[$8],
smallint_field=[$9], numeric_field=[$10], boolean_field=[$11],
double_field=[$12], float_field=[$13], real_field=[$14], time_field=[$15],
timestamp_field=[$16], date_field=[$17], datetime_field=[$18],
year_field=[$19], text_field=[$20], tiny_text_field=[$21],
medium_text_field=[$22], long_text_field=[$23], blob_field=[$24],
bit_field=[$25], enum_field=[$26], PERSON_ID0=[$27], FIRST_NAME0=[$28],
LAST_NAME0=[$29], ADDRESS0=[$30], CITY0=[$31], STATE0=[$32], ZIP0=[$33],
JSON0=[$34], BIGINT_FIELD0=[$35], SMALLINT_FIELD0=[$36], NUMERIC_FIELD0=[$37],
BOOLEAN_FIELD0=[$38], DOUBLE_FIELD0=[$39], FLOAT_FIELD0=[$40],
REAL_FIELD0=[$41], TIME_FIELD0=[$42], TIMESTAMP_FIELD0=[$43],
DATE_FIELD0=[$44], CLOB_FIELD=[$45])
00-02HashJoin(condition=[=($0, $27)], joinType=[inner], semi-join:
=[false])
00-03 Project(PERSON_ID0=[$0], FIRST_NAME0=[$1], LAST_NAME0=[$2],
ADDRESS0=[$3], CITY0=[$4], STATE0=[$5], ZIP0=[$6], JSON0=[$7],
BIGINT_FIELD0=[$8], SMALLINT_FIELD0=[$9], NUMERIC_FIELD0=[$10],
BOOLEAN_FIELD0=[$11], DOUBLE_FIELD0=[$12], FLOAT_FIELD0=[$13],
REAL_FIELD0=[$14], TIME_FIELD0=[$15], TIMESTAMP_FIELD0=[$16],
DATE_FIELD0=[$17], CLOB_FIELD=[$18])
00-05SelectionVectorRemover
00-06 Filter(condition=[=($2, 'last_name_1')])
00-07Jdbc(sql=[SELECT * FROM "TMP"."DRILL_H2_TEST"."PERSON" ])
00-04 Jdbc(sql=[SELECT * FROM `drill_mysql_test`.`person` WHERE
`first_name` = 'first_name_1' ])
{noformat}
{{DrillJdbcFilterRule}} wasn't applied for H2 convention and Filter wasn't
pushed to H2 database.
This issue may be fixed by specifying {{JdbcConvention}} in rules descriptions
in Drill {{DrillJdbcFilterRule}} and {{DrillJdbcProjectRule}} rules and other
rules should be fixed in Calcite in the scope of CALCITE-3115.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (DRILL-6961) Error Occurred: Cannot connect to the db. query INFORMATION_SCHEMA.VIEWS : Maybe you have incorrect connection params or db unavailable now (timeout)
[
https://issues.apache.org/jira/browse/DRILL-6961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi updated DRILL-6961:
---
Labels: ready-to-commit (was: )
> Error Occurred: Cannot connect to the db. query INFORMATION_SCHEMA.VIEWS :
> Maybe you have incorrect connection params or db unavailable now (timeout)
> -
>
> Key: DRILL-6961
> URL: https://issues.apache.org/jira/browse/DRILL-6961
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - Information Schema
>Affects Versions: 1.13.0
>Reporter: Khurram Faraaz
>Assignee: Anton Gozhiy
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.17.0
>
>
> Trying to query drill information_schema.views table returns error. Disabling
> openTSDB plugin resolves the problem.
> Drill 1.13.0
> Failing query :
> {noformat}
> SELECT TABLE_CATALOG, TABLE_SCHEMA, TABLE_NAME, VIEW_DEFINITION FROM
> INFORMATION_SCHEMA.`VIEWS` where VIEW_DEFINITION not like 'kraken';
> {noformat}
> Stack Trace from drillbit.log
> {noformat}
> 2019-01-07 15:36:21,975 [23cc39aa-2618-e9f0-e77e-4fafa6edc314:foreman] INFO
> o.a.drill.exec.work.foreman.Foreman - Query text for query id
> 23cc39aa-2618-e9f0-e77e-4fafa6edc314: SELECT TABLE_CATALOG, TABLE_SCHEMA,
> TABLE_NAME, VIEW_DEFINITION FROM INFORMATION_SCHEMA.`VIEWS` where
> VIEW_DEFINITION not like 'kraken'
> 2019-01-07 15:36:35,221 [23cc39aa-2618-e9f0-e77e-4fafa6edc314:frag:0:0] INFO
> o.a.d.e.s.o.c.services.ServiceImpl - User Error Occurred: Cannot connect to
> the db. Maybe you have incorrect connection params or db unavailable now
> (timeout)
> org.apache.drill.common.exceptions.UserException: CONNECTION ERROR: Cannot
> connect to the db. Maybe you have incorrect connection params or db
> unavailable now
> [Error Id: f8b4c074-ba62-4691-b142-a8ea6e4f6b2a ]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
> ~[drill-common-1.13.0-mapr.jar:1.13.0-mapr]
> at
> org.apache.drill.exec.store.openTSDB.client.services.ServiceImpl.getTableNames(ServiceImpl.java:107)
> [drill-opentsdb-storage-1.13.0-mapr.jar:1.13.0-mapr]
> at
> org.apache.drill.exec.store.openTSDB.client.services.ServiceImpl.getAllMetricNames(ServiceImpl.java:70)
> [drill-opentsdb-storage-1.13.0-mapr.jar:1.13.0-mapr]
> at
> org.apache.drill.exec.store.openTSDB.schema.OpenTSDBSchemaFactory$OpenTSDBSchema.getTableNames(OpenTSDBSchemaFactory.java:78)
> [drill-opentsdb-storage-1.13.0-mapr.jar:1.13.0-mapr]
> at
> org.apache.calcite.jdbc.SimpleCalciteSchema.addImplicitTableToBuilder(SimpleCalciteSchema.java:106)
> [calcite-core-1.15.0-drill-r0.jar:1.15.0-drill-r0]
> at
> org.apache.calcite.jdbc.CalciteSchema.getTableNames(CalciteSchema.java:318)
> [calcite-core-1.15.0-drill-r0.jar:1.15.0-drill-r0]
> at
> org.apache.calcite.jdbc.CalciteSchema$SchemaPlusImpl.getTableNames(CalciteSchema.java:587)
> [calcite-core-1.15.0-drill-r0.jar:1.15.0-drill-r0]
> at
> org.apache.calcite.jdbc.CalciteSchema$SchemaPlusImpl.getTableNames(CalciteSchema.java:548)
> [calcite-core-1.15.0-drill-r0.jar:1.15.0-drill-r0]
> at
> org.apache.drill.exec.store.ischema.InfoSchemaRecordGenerator.visitTables(InfoSchemaRecordGenerator.java:227)
> [drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr]
> at
> org.apache.drill.exec.store.ischema.InfoSchemaRecordGenerator.scanSchema(InfoSchemaRecordGenerator.java:216)
> [drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr]
> at
> org.apache.drill.exec.store.ischema.InfoSchemaRecordGenerator.scanSchema(InfoSchemaRecordGenerator.java:209)
> [drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr]
> at
> org.apache.drill.exec.store.ischema.InfoSchemaRecordGenerator.scanSchema(InfoSchemaRecordGenerator.java:196)
> [drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr]
> at
> org.apache.drill.exec.store.ischema.InfoSchemaTableType.getRecordReader(InfoSchemaTableType.java:58)
> [drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr]
> at
> org.apache.drill.exec.store.ischema.InfoSchemaBatchCreator.getBatch(InfoSchemaBatchCreator.java:34)
> [drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr]
> at
> org.apache.drill.exec.store.ischema.InfoSchemaBatchCreator.getBatch(InfoSchemaBatchCreator.java:30)
> [drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr]
> at
> org.apache.drill.exec.physical.impl.ImplCreator$2.run(ImplCreator.java:146)
> [drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr]
> at
> org.apache.drill.exec.physical.impl.ImplCreator$2.run(ImplCreator.java:142)
> [drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr]
> at java.security.AccessController.doPrivileged(Native Method) [na:1.8.0_144]
> at javax.security.auth.Subject.doAs(Subject.java:422) [na:1.8.0_144]
> at
>
[jira] [Updated] (DRILL-7337) Add vararg UDFs support
[
https://issues.apache.org/jira/browse/DRILL-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi updated DRILL-7337:
---
Labels: doc-impacting ready-to-commit (was: doc-impacting)
> Add vararg UDFs support
> ---
>
> Key: DRILL-7337
> URL: https://issues.apache.org/jira/browse/DRILL-7337
> Project: Apache Drill
> Issue Type: Sub-task
>Affects Versions: 1.16.0
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Labels: doc-impacting, ready-to-commit
> Fix For: 1.17.0
>
>
> The aim of this Jira is to add support for vararg UDFs to simplify UDFs
> creation for the case when it is required to accept different numbers of
> arguments.
> h2. Requirements for vararg UDFs:
> * It should be possible to register vararg UDFs with the same name, but with
> different argument types;
> * Only vararg UDFs with a single variable-length argument placed after all
> other arguments should be allowed;
> * Vararg UDF should have less priority than the regular one for the case
> when they both are suitable;
> * Besides simple functions, vararg support should be added to the aggregate
> functions.
> h2. Implementation details
> The lifecycle of UDF is the following:
> * UDF is validated in {{FunctionConverter}} class and for the case when
> there is no problem (UDF has required fields with required types, required
> annotations, etc.), it is converted to the {{DrillFuncHolder}} to be
> registered in the function registry. Also, corresponding {{SqlFunction}}
> instances are created based on {{DrillFuncHolder}} to be used in Calcite;
> * When a query uses this UDF, Calcite validate that UDF with required name,
> arguments number and arguments types (for Drill arguments types are not
> checked at this stage) exists;
> * After Calcite was able to find the required {{SqlFunction instance}}, it
> uses Drill to find required {{DrillFuncHolder}}. All the work for determining
> the most suitable function is done in {{FunctionResolver}} and in
> {{TypeCastRules.getCost()}};
> * At the execution stage, {{DrillFuncHolder}} found again using
> {{FunctionCall}} instance;
> * {{DrillFuncHolder}} is used for code generation.
> Considering these steps, the first thing to be done for adding support for
> vararg UDFs is updating logic in {{FunctionConverter}} to allow registering
> vararg UDFs taking into account requirements declared above.
> Calcite uses {{SqlOperandTypeChecker}} to verify arguments number, so Drill
> should provide its own for vararg UDFs to be able to use them. To determine
> whether UDF is vararg, new {{isVarArg}} property will be added to the
> {{FunctionTemplate}}.
> {{TypeCastRules.getCost()}} method should be updated to be able to find
> vararg UDFs and prioritize regular UDFs.
> Code generation logic should be updated to handle vararg UDFs. Generated code
> for varag argument will look in the following way:
> {code:java}
> NullableVarCharHolder[] inputs = new
> NullableVarCharHolder[3];
> inputs[0] = out14;
> inputs[1] = out19;
> inputs[2] = out24;
> {code}
> To create own varagr UDF, new {{isVarArg}} property should be set to {{true}}
> in {{FunctionTemplate}}.
> After that, required vararg input should be declared as an array.
> Here is an example if vararg UDF:
> {code:java}
> @FunctionTemplate(name = "concat_varchar",
> isVarArg = true,
> scope = FunctionTemplate.FunctionScope.SIMPLE)
> public class VarCharConcatFunction implements DrillSimpleFunc {
> @Param *VarCharHolder[] inputs*;
> @Output VarCharHolder out;
> @Inject DrillBuf buffer;
>
> @Override
> public void setup() {
> }
> @Override
> public void eval() {
> int length = 0;
> for (VarCharHolder input : inputs) {
> length += input.end - input.start;
> }
> out.buffer = buffer = buffer.reallocIfNeeded(length);
> out.start = out.end = 0;
> for (VarCharHolder input : inputs) {
> for (int id = input.start; id < input.end; id++) {
> out.buffer.setByte(out.end++, input.buffer.getByte(id));
> }
> }
> }
> }
> {code}
> h2. Limitations connected with VarArg UDFs:
> * Specified nulls handling in FunctionTemplate does not affect vararg
> parameters, i.e. the user should add UDFs with non-nullable and nullable
> value holder vararg fields;
> * VarArg UDFs supports only values of the same type including nullability
> for vararg arguments for value holder vararg fields. If vararg field is
> FieldReader, all the responsibility for handling types and nullability of
> input vararg fields is placed on the UDF implementation;
> * The scalar replacement
[jira] [Created] (DRILL-7337) Add vararg UDFs support
Volodymyr Vysotskyi created DRILL-7337:
--
Summary: Add vararg UDFs support
Key: DRILL-7337
URL: https://issues.apache.org/jira/browse/DRILL-7337
Project: Apache Drill
Issue Type: Sub-task
Affects Versions: 1.16.0
Reporter: Volodymyr Vysotskyi
Assignee: Volodymyr Vysotskyi
Fix For: 1.17.0
The aim of this Jira is to add support for vararg UDFs to simplify UDFs
creation for the case when it is required to accept different numbers of
arguments.
h2. Requirements for vararg UDFs:
* It should be possible to register vararg UDFs with the same name, but with
different argument types;
* Only vararg UDFs with a single variable-length argument placed after all
other arguments should be allowed;
* Vararg UDF should have less priority than the regular one for the case when
they both are suitable;
* Besides simple functions, vararg support should be added to the aggregate
functions.
h2. Implementation details
The lifecycle of UDF is the following:
* UDF is validated in {{FunctionConverter}} class and for the case when there
is no problem (UDF has required fields with required types, required
annotations, etc.), it is converted to the {{DrillFuncHolder}} to be registered
in the function registry. Also, corresponding {{SqlFunction}} instances are
created based on {{DrillFuncHolder}} to be used in Calcite;
* When a query uses this UDF, Calcite validate that UDF with required name,
arguments number and arguments types (for Drill arguments types are not checked
at this stage) exists;
* After Calcite was able to find the required {{SqlFunction instance}}, it
uses Drill to find required {{DrillFuncHolder}}. All the work for determining
the most suitable function is done in {{FunctionResolver}} and in
{{TypeCastRules.getCost()}};
* At the execution stage, {{DrillFuncHolder}} found again using
{{FunctionCall}} instance;
* {{DrillFuncHolder}} is used for code generation.
Considering these steps, the first thing to be done for adding support for
vararg UDFs is updating logic in {{FunctionConverter}} to allow registering
vararg UDFs taking into account requirements declared above.
Calcite uses {{SqlOperandTypeChecker}} to verify arguments number, so Drill
should provide its own for vararg UDFs to be able to use them. To determine
whether UDF is vararg, new {{isVarArg}} property will be added to the
{{FunctionTemplate}}.
{{TypeCastRules.getCost()}} method should be updated to be able to find vararg
UDFs and prioritize regular UDFs.
Code generation logic should be updated to handle vararg UDFs. Generated code
for varag argument will look in the following way:
{code:java}
NullableVarCharHolder[] inputs = new NullableVarCharHolder[3];
inputs[0] = out14;
inputs[1] = out19;
inputs[2] = out24;
{code}
To create own varagr UDF, new {{isVarArg}} property should be set to {{true}}
in {{FunctionTemplate}}.
After that, required vararg input should be declared as an array.
Here is an example if vararg UDF:
{code:java}
@FunctionTemplate(name = "concat_varchar",
isVarArg = true,
scope = FunctionTemplate.FunctionScope.SIMPLE)
public class VarCharConcatFunction implements DrillSimpleFunc {
@Param *VarCharHolder[] inputs*;
@Output VarCharHolder out;
@Inject DrillBuf buffer;
@Override
public void setup() {
}
@Override
public void eval() {
int length = 0;
for (VarCharHolder input : inputs) {
length += input.end - input.start;
}
out.buffer = buffer = buffer.reallocIfNeeded(length);
out.start = out.end = 0;
for (VarCharHolder input : inputs) {
for (int id = input.start; id < input.end; id++) {
out.buffer.setByte(out.end++, input.buffer.getByte(id));
}
}
}
}
{code}
h2. Limitations connected with VarArg UDFs:
* Specified nulls handling in FunctionTemplate does not affect vararg
parameters, i.e. the user should add UDFs with non-nullable and nullable value
holder vararg fields;
* VarArg UDFs supports only values of the same type including nullability for
vararg arguments for value holder vararg fields. If vararg field is
FieldReader, all the responsibility for handling types and nullability of input
vararg fields is placed on the UDF implementation;
* The scalar replacement does not happen for vararg arguments;
* UDF implementation should consider the case when vararg field is empty.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (DRILL-7334) Update Iceberg Metastore Parquet write mode
[ https://issues.apache.org/jira/browse/DRILL-7334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Volodymyr Vysotskyi updated DRILL-7334: --- Labels: ready-to-commit (was: ) > Update Iceberg Metastore Parquet write mode > --- > > Key: DRILL-7334 > URL: https://issues.apache.org/jira/browse/DRILL-7334 > Project: Apache Drill > Issue Type: Improvement >Reporter: Arina Ielchiieva >Assignee: Arina Ielchiieva >Priority: Major > Labels: ready-to-commit > Fix For: 1.17.0 > > > Initially Iceberg Parquet writer was using OVERWRITE mode by default. After > f4fc8ff, default mode is CREATE. Need to update Iceberg Metastore code to > comply with the latest changes. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (DRILL-7327) Log Regex Plugin Won't Recognize Schema
[
https://issues.apache.org/jira/browse/DRILL-7327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi updated DRILL-7327:
---
Fix Version/s: 1.17.0
> Log Regex Plugin Won't Recognize Schema
> ---
>
> Key: DRILL-7327
> URL: https://issues.apache.org/jira/browse/DRILL-7327
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.17.0
>Reporter: Charles Givre
>Assignee: Paul Rogers
>Priority: Major
> Fix For: 1.17.0
>
> Attachments: firewall.ssdlog
>
>
> When I attempt to create a define a schema for the new `logRegex` plugin,
> Drill does not recognize the plugin if the configuration includes a schema.
> {code:json}
> {,
> "ssdlog": {
> "type": "logRegex",
> "regex":
> "(\\w{3}\\s\\d{1,2}\\s\\d{4}\\s\\d{2}:\\d{2}:\\d{2})\\s+(\\w+)\\[(\\d+)\\]:\\s(.*?(\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}).*?)",
> "extension": "ssdlog",
> "maxErrors": 10,
> "schema": []
> }
> {code}
> This configuration works, however, this does not:
> {code:json}
> {,
> "ssdlog": {
> "type": "logRegex",
> "regex":
> "(\\w{3}\\s\\d{1,2}\\s\\d{4}\\s\\d{2}:\\d{2}:\\d{2})\\s+(\\w+)\\[(\\d+)\\]:\\s(.*?(\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}).*?)",
> "extension": "ssdlog",
> "maxErrors": 10,
> "schema": [
> {"fieldName":"eventDate"}
> ]
> }
> {code}
> [~paul-rogers]
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Comment Edited] (DRILL-7205) Drill fails to start when authentication is disabled
[
https://issues.apache.org/jira/browse/DRILL-7205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16895097#comment-16895097
]
Volodymyr Vysotskyi edited comment on DRILL-7205 at 7/29/19 9:24 AM:
-
Merged into Apache master with commit id
[275fb27|https://github.com/apache/drill/commit/275fb27126b35fe3229f6ef6dd5296f4138670fc].
was (Author: vvysotskyi):
Merged into Apache master with commit id
[275fb27https://github.com/apache/drill/commit/275fb27126b35fe3229f6ef6dd5296f4138670fc]
> Drill fails to start when authentication is disabled
>
>
> Key: DRILL-7205
> URL: https://issues.apache.org/jira/browse/DRILL-7205
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.15.0, 1.16.0
>Reporter: Arina Ielchiieva
>Assignee: Anton Gozhiy
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.17.0
>
>
> drill-override.conf content:
> {noformat}
> drill.exec: {
> cluster-id: "drillbits1",
> zk.connect: "localhost:2181",
>impersonation: {
> enabled: true,
> max_chained_user_hops: 3
> },
> security.user.auth: {
> enabled: false,
> packages += "org.apache.drill.exec.rpc.user.security",
> impl: "pam",
> pam_profiles: [ "sudo", "login" ]
> }
> }
> {noformat}
> Note that authentication is disabled {{enabled: false}}.
> Drill fails during start up:
> {noformat}
> DRILL_ARGS - " -u jdbc:drill:zk=local"
> Calculating HADOOP_CLASSPATH ...
> HBASE_HOME not detected...
> Calculating Drill classpath...
> Error: Failure in starting embedded Drillbit:
> org.apache.drill.exec.exception.DrillbitStartupException: Problem in finding
> the native library of JPAM (Pluggable Authenticator Module API). Make sure to
> set Drillbit JVM option 'java.library.path' to point to the directory where
> the native JPAM exists.:no jpam in java.library.path (state=,code=0)
> java.sql.SQLException: Failure in starting embedded Drillbit:
> org.apache.drill.exec.exception.DrillbitStartupException: Problem in finding
> the native library of JPAM (Pluggable Authenticator Module API). Make sure to
> set Drillbit JVM option 'java.library.path' to point to the directory where
> the native JPAM exists.:no jpam in java.library.path
> at
> org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:143)
> at
> org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:67)
> at
> org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:67)
> at
> org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)
> at org.apache.drill.jdbc.Driver.connect(Driver.java:72)
> at sqlline.DatabaseConnection.connect(DatabaseConnection.java:130)
> at
> sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:179)
> at sqlline.Commands.connect(Commands.java:1278)
> at sqlline.Commands.connect(Commands.java:1172)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:38)
> at sqlline.SqlLine.dispatch(SqlLine.java:736)
> at sqlline.SqlLine.initArgs(SqlLine.java:428)
> at sqlline.SqlLine.begin(SqlLine.java:531)
> at sqlline.SqlLine.start(SqlLine.java:270)
> at sqlline.SqlLine.main(SqlLine.java:201)
> Caused by: org.apache.drill.exec.exception.DrillbitStartupException: Problem
> in finding the native library of JPAM (Pluggable Authenticator Module API).
> Make sure to set Drillbit JVM option 'java.library.path' to point to the
> directory where the native JPAM exists.:no jpam in java.library.path
> at
> org.apache.drill.exec.rpc.user.security.PamUserAuthenticator.setup(PamUserAuthenticator.java:52)
> at
> org.apache.drill.exec.rpc.user.security.UserAuthenticatorFactory.createAuthenticator(UserAuthenticatorFactory.java:98)
> at
> org.apache.drill.exec.rpc.security.AuthenticatorProviderImpl.(AuthenticatorProviderImpl.java:66)
> at
> org.apache.drill.exec.server.BootStrapContext.(BootStrapContext.java:83)
> at org.apache.drill.exec.server.Drillbit.(Drillbit.java:161)
> at org.apache.drill.exec.server.Drillbit.(Drillbit.java:125)
> at
> org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:133)
> ... 18 more
> Caused by: java.lang.UnsatisfiedLinkError: no jpam in java.library.path
> at
[jira] [Updated] (DRILL-7332) Drill requires parentheses in the empty file for 'LOAD' argument in the 'CREATE SCHEMA' command
[
https://issues.apache.org/jira/browse/DRILL-7332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi updated DRILL-7332:
---
Labels: ready-to-commit (was: )
> Drill requires parentheses in the empty file for 'LOAD' argument in the
> 'CREATE SCHEMA' command
> ---
>
> Key: DRILL-7332
> URL: https://issues.apache.org/jira/browse/DRILL-7332
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.16.0
>Reporter: Denys Ordynskiy
>Assignee: Arina Ielchiieva
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.17.0
>
>
> Description:
> - created csvh table: {color:#205081}create table dfs.tmp.`test_table`
> (col1) as select full_name from cp.`employee.json` limit 3;{color}
> - created text file on the local file system `for_load` with text without
> parentheses: {color:#14892c}col1 varchar not null{color}
> - created some schema file: {color:#205081}create or replace schema *LOAD*
> 'file:///home/user/for_load' for table dfs.tmp.`test_table` properties
> ('drill.strict' = 'false');{color} - schema was created successfully;
> - remove all text from the `for_load` file and try to create the same schema:
> {color:#205081}create or replace schema *LOAD* 'file:///home/user/for_load'
> for table dfs.tmp.`test_table` properties ('drill.strict' = 'false');{color}
> *Actual result:*
> Drill throws an error:
> {color:#d04437}Error: RESOURCE ERROR: Unable to parse schema []: Line [1],
> position [0], offending symbol [@0,0:-1='',<-1>,1:0]: mismatched input
> '' expecting {'(', ID, QUOTED_ID}
> Error while preparing / creating schema for [%s] dfs.tmp.test_table
> [Error Id: faad9c09-2e3e-428c-bd7a-6da8832a943b ] (state=,code=0){color}
> *Expected result:*
> Since Drill doesn't require parentheses for non empty `for_load` file (with
> some columns),
> It couldn't require parentheses for the empty file, used in the `LOAD`
> argument.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (DRILL-7272) Implement Drill Iceberg Metastore plugin
[ https://issues.apache.org/jira/browse/DRILL-7272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Volodymyr Vysotskyi updated DRILL-7272: --- Labels: doc-impacting ready-to-commit (was: doc-impacting) > Implement Drill Iceberg Metastore plugin > > > Key: DRILL-7272 > URL: https://issues.apache.org/jira/browse/DRILL-7272 > Project: Apache Drill > Issue Type: Sub-task >Reporter: Arina Ielchiieva >Assignee: Arina Ielchiieva >Priority: Major > Labels: doc-impacting, ready-to-commit > Fix For: 1.17.0 > > > PR contains two README.md files which describe the changes. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Resolved] (DRILL-4123) DirectoryExplorers should refer to fully qualified variable names
[
https://issues.apache.org/jira/browse/DRILL-4123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi resolved DRILL-4123.
Resolution: Fixed
Fixed in DRILL-3944.
> DirectoryExplorers should refer to fully qualified variable names
> -
>
> Key: DRILL-4123
> URL: https://issues.apache.org/jira/browse/DRILL-4123
> Project: Apache Drill
> Issue Type: Bug
>Reporter: Hanifi Gunes
>Assignee: Hanifi Gunes
>Priority: Major
>
> Execution fails with {code}CompileException: Line 75, Column 70: Unknown
> variable or type "FILE_SEPARATOR"{code} in case a directory explorer is used
> in a projection. Also FILE_SEPARATOR should not be platform dependent.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Assigned] (DRILL-4576) Add StoragePlugin API to register materialization into planner
[
https://issues.apache.org/jira/browse/DRILL-4576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi reassigned DRILL-4576:
--
Assignee: Laurent Goujon (was: Volodymyr Vysotskyi)
> Add StoragePlugin API to register materialization into planner
> --
>
> Key: DRILL-4576
> URL: https://issues.apache.org/jira/browse/DRILL-4576
> Project: Apache Drill
> Issue Type: Bug
>Reporter: Laurent Goujon
>Assignee: Laurent Goujon
>Priority: Major
>
> There's no currently a good way to register materializations into Drill
> planner. Calcite's MaterializationService.instance() would be the way to go,
> but the registration happens in
> {{org.apache.calcite.prepare.Prepare.PreparedResult#prepareSql()}}, which is
> not called by Drill.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Assigned] (DRILL-4576) Add StoragePlugin API to register materialization into planner
[
https://issues.apache.org/jira/browse/DRILL-4576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi reassigned DRILL-4576:
--
Assignee: Volodymyr Vysotskyi
> Add StoragePlugin API to register materialization into planner
> --
>
> Key: DRILL-4576
> URL: https://issues.apache.org/jira/browse/DRILL-4576
> Project: Apache Drill
> Issue Type: Bug
>Reporter: Laurent Goujon
>Assignee: Volodymyr Vysotskyi
>Priority: Major
>
> There's no currently a good way to register materializations into Drill
> planner. Calcite's MaterializationService.instance() would be the way to go,
> but the registration happens in
> {{org.apache.calcite.prepare.Prepare.PreparedResult#prepareSql()}}, which is
> not called by Drill.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (DRILL-7205) Drill fails to start when authentication is disabled
[
https://issues.apache.org/jira/browse/DRILL-7205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi updated DRILL-7205:
---
Labels: ready-to-commit (was: )
> Drill fails to start when authentication is disabled
>
>
> Key: DRILL-7205
> URL: https://issues.apache.org/jira/browse/DRILL-7205
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.15.0, 1.16.0
>Reporter: Arina Ielchiieva
>Assignee: Anton Gozhiy
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.17.0
>
>
> drill-override.conf content:
> {noformat}
> drill.exec: {
> cluster-id: "drillbits1",
> zk.connect: "localhost:2181",
>impersonation: {
> enabled: true,
> max_chained_user_hops: 3
> },
> security.user.auth: {
> enabled: false,
> packages += "org.apache.drill.exec.rpc.user.security",
> impl: "pam",
> pam_profiles: [ "sudo", "login" ]
> }
> }
> {noformat}
> Note that authentication is disabled {{enabled: false}}.
> Drill fails during start up:
> {noformat}
> DRILL_ARGS - " -u jdbc:drill:zk=local"
> Calculating HADOOP_CLASSPATH ...
> HBASE_HOME not detected...
> Calculating Drill classpath...
> Error: Failure in starting embedded Drillbit:
> org.apache.drill.exec.exception.DrillbitStartupException: Problem in finding
> the native library of JPAM (Pluggable Authenticator Module API). Make sure to
> set Drillbit JVM option 'java.library.path' to point to the directory where
> the native JPAM exists.:no jpam in java.library.path (state=,code=0)
> java.sql.SQLException: Failure in starting embedded Drillbit:
> org.apache.drill.exec.exception.DrillbitStartupException: Problem in finding
> the native library of JPAM (Pluggable Authenticator Module API). Make sure to
> set Drillbit JVM option 'java.library.path' to point to the directory where
> the native JPAM exists.:no jpam in java.library.path
> at
> org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:143)
> at
> org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:67)
> at
> org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:67)
> at
> org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)
> at org.apache.drill.jdbc.Driver.connect(Driver.java:72)
> at sqlline.DatabaseConnection.connect(DatabaseConnection.java:130)
> at
> sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:179)
> at sqlline.Commands.connect(Commands.java:1278)
> at sqlline.Commands.connect(Commands.java:1172)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:38)
> at sqlline.SqlLine.dispatch(SqlLine.java:736)
> at sqlline.SqlLine.initArgs(SqlLine.java:428)
> at sqlline.SqlLine.begin(SqlLine.java:531)
> at sqlline.SqlLine.start(SqlLine.java:270)
> at sqlline.SqlLine.main(SqlLine.java:201)
> Caused by: org.apache.drill.exec.exception.DrillbitStartupException: Problem
> in finding the native library of JPAM (Pluggable Authenticator Module API).
> Make sure to set Drillbit JVM option 'java.library.path' to point to the
> directory where the native JPAM exists.:no jpam in java.library.path
> at
> org.apache.drill.exec.rpc.user.security.PamUserAuthenticator.setup(PamUserAuthenticator.java:52)
> at
> org.apache.drill.exec.rpc.user.security.UserAuthenticatorFactory.createAuthenticator(UserAuthenticatorFactory.java:98)
> at
> org.apache.drill.exec.rpc.security.AuthenticatorProviderImpl.(AuthenticatorProviderImpl.java:66)
> at
> org.apache.drill.exec.server.BootStrapContext.(BootStrapContext.java:83)
> at org.apache.drill.exec.server.Drillbit.(Drillbit.java:161)
> at org.apache.drill.exec.server.Drillbit.(Drillbit.java:125)
> at
> org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:133)
> ... 18 more
> Caused by: java.lang.UnsatisfiedLinkError: no jpam in java.library.path
> at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
> at java.lang.Runtime.loadLibrary0(Runtime.java:870)
> at java.lang.System.loadLibrary(System.java:1122)
> at net.sf.jpam.Pam.(Pam.java:51)
> at
> org.apache.drill.exec.rpc.user.security.PamUserAuthenticator.setup(PamUserAuthenticator.java:46)
> ... 24
[jira] [Resolved] (DRILL-3676) Group by ordinal number of an output column results in parse error
[
https://issues.apache.org/jira/browse/DRILL-3676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi resolved DRILL-3676.
Resolution: Fixed
Fix Version/s: (was: Future)
1.17.0
Fixed in DRILL-1248.
> Group by ordinal number of an output column results in parse error
> --
>
> Key: DRILL-3676
> URL: https://issues.apache.org/jira/browse/DRILL-3676
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning Optimization
>Affects Versions: 1.2.0
>Reporter: Khurram Faraaz
>Priority: Major
> Fix For: 1.17.0
>
>
> Group by number results in parse error.
> {code}
> 0: jdbc:drill:schema=dfs.tmp> select sub_q.col1 from (select col1 from
> FEWRWSPQQ_101) sub_q group by 1;
> Error: PARSE ERROR: At line 1, column 8: Expression 'q.col1' is not being
> grouped
> [Error Id: 0eedafd9-372e-4610-b7a8-d97e26458d58 on centos-02.qa.lab:31010]
> (state=,code=0)
> {code}
> When we use the column name instead of the number, the query compiles and
> returns results.
> {code}
> 0: jdbc:drill:schema=dfs.tmp> select col1 from (select col1 from
> FEWRWSPQQ_101) group by col1;
> +--+
> | col1 |
> +--+
> | 65534|
> | 1000 |
> | -1 |
> | 0|
> | 1|
> | 13 |
> | 17 |
> | 23 |
> | 1000 |
> | 999 |
> | 30 |
> | 25 |
> | 1001 |
> | -65535 |
> | 5000 |
> | 3000 |
> | 200 |
> | 197 |
> | 4611686018427387903 |
> | 9223372036854775806 |
> | 9223372036854775807 |
> | 92233720385475807|
> +--+
> 22 rows selected (0.218 seconds)
> {code}
> {code}
> 0: jdbc:drill:schema=dfs.tmp> select sub_query.col1 from (select col1 from
> FEWRWSPQQ_101) sub_query group by sub_query.col1;
> +--+
> | col1 |
> +--+
> | 65534|
> | 1000 |
> | -1 |
> | 0|
> | 1|
> | 13 |
> | 17 |
> | 23 |
> | 1000 |
> | 999 |
> | 30 |
> | 25 |
> | 1001 |
> | -65535 |
> | 5000 |
> | 3000 |
> | 200 |
> | 197 |
> | 4611686018427387903 |
> | 9223372036854775806 |
> | 9223372036854775807 |
> | 92233720385475807|
> +--+
> 22 rows selected (0.177 seconds)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-3664) CAST integer zero , one to boolean false , true
[
https://issues.apache.org/jira/browse/DRILL-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi updated DRILL-3664:
---
Fix Version/s: (was: Future)
1.17.0
> CAST integer zero , one to boolean false , true
> ---
>
> Key: DRILL-3664
> URL: https://issues.apache.org/jira/browse/DRILL-3664
> Project: Apache Drill
> Issue Type: Bug
> Components: SQL Parser
>Affects Versions: 1.2.0
>Reporter: Khurram Faraaz
>Priority: Major
> Fix For: 1.17.0
>
>
> We should be able to cast (zero) 0 to false and (one) 1 to true, currently we
> report a parse error when an explicit cast is used in query.
> col7 is of type Boolean in the below input parquet file.
> {code}
> 0: jdbc:drill:schema=dfs.tmp> select col7 from FEWRWSPQQ_101 where col7 IN
> (cast(0 as boolean),cast(1 as boolean));
> Error: PARSE ERROR: From line 1, column 47 to line 1, column 64: Cast
> function cannot convert value of type INTEGER to type BOOLEAN
> [Error Id: d751945f-8a0f-4369-ae9e-c42504f6d978 on centos-04.qa.lab:31010]
> (state=,code=0)
> {code}
> Without explicit cast we see SchemaChangeException.
> {code}
> 0: jdbc:drill:schema=dfs.tmp> select col7 from FEWRWSPQQ_101 where col7 IN
> (0,1);
> Error: SYSTEM ERROR: SchemaChangeException: Failure while trying to
> materialize incoming schema. Errors:
>
> Error in expression at index -1. Error: Missing function implementation:
> [castINT(BIT-OPTIONAL)]. Full expression: --UNKNOWN EXPRESSION--.
> Error in expression at index -1. Error: Missing function implementation:
> [castINT(BIT-OPTIONAL)]. Full expression: --UNKNOWN EXPRESSION--..
> Fragment 0:0
> [Error Id: ecf51dae-62c5-40d7-b0f5-3b9bf9fd3377 on centos-04.qa.lab:31010]
> (state=,code=0)
> {code}
> Postgres results for the same query.
> {code}
> postgres=# select col7 from FEWRWSPQQ_101 where col7 IN (cast(0 as
> boolean),cast(1 as boolean));
> col7
> --
> f
> t
> f
> t
> f
> t
> f
> t
> f
> t
> f
> t
> f
> t
> f
> t
> f
> t
> f
> t
> f
> t
> (22 rows)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (DRILL-3664) CAST integer zero , one to boolean false , true
[
https://issues.apache.org/jira/browse/DRILL-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi resolved DRILL-3664.
Resolution: Fixed
Fixed in DRILL-4674
> CAST integer zero , one to boolean false , true
> ---
>
> Key: DRILL-3664
> URL: https://issues.apache.org/jira/browse/DRILL-3664
> Project: Apache Drill
> Issue Type: Bug
> Components: SQL Parser
>Affects Versions: 1.2.0
>Reporter: Khurram Faraaz
>Priority: Major
> Fix For: Future
>
>
> We should be able to cast (zero) 0 to false and (one) 1 to true, currently we
> report a parse error when an explicit cast is used in query.
> col7 is of type Boolean in the below input parquet file.
> {code}
> 0: jdbc:drill:schema=dfs.tmp> select col7 from FEWRWSPQQ_101 where col7 IN
> (cast(0 as boolean),cast(1 as boolean));
> Error: PARSE ERROR: From line 1, column 47 to line 1, column 64: Cast
> function cannot convert value of type INTEGER to type BOOLEAN
> [Error Id: d751945f-8a0f-4369-ae9e-c42504f6d978 on centos-04.qa.lab:31010]
> (state=,code=0)
> {code}
> Without explicit cast we see SchemaChangeException.
> {code}
> 0: jdbc:drill:schema=dfs.tmp> select col7 from FEWRWSPQQ_101 where col7 IN
> (0,1);
> Error: SYSTEM ERROR: SchemaChangeException: Failure while trying to
> materialize incoming schema. Errors:
>
> Error in expression at index -1. Error: Missing function implementation:
> [castINT(BIT-OPTIONAL)]. Full expression: --UNKNOWN EXPRESSION--.
> Error in expression at index -1. Error: Missing function implementation:
> [castINT(BIT-OPTIONAL)]. Full expression: --UNKNOWN EXPRESSION--..
> Fragment 0:0
> [Error Id: ecf51dae-62c5-40d7-b0f5-3b9bf9fd3377 on centos-04.qa.lab:31010]
> (state=,code=0)
> {code}
> Postgres results for the same query.
> {code}
> postgres=# select col7 from FEWRWSPQQ_101 where col7 IN (cast(0 as
> boolean),cast(1 as boolean));
> col7
> --
> f
> t
> f
> t
> f
> t
> f
> t
> f
> t
> f
> t
> f
> t
> f
> t
> f
> t
> f
> t
> f
> t
> (22 rows)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-7307) casthigh for decimal type can lead to the issues with VarDecimalHolder
[
https://issues.apache.org/jira/browse/DRILL-7307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi updated DRILL-7307:
---
Labels: ready-to-commit (was: )
> casthigh for decimal type can lead to the issues with VarDecimalHolder
> --
>
> Key: DRILL-7307
> URL: https://issues.apache.org/jira/browse/DRILL-7307
> Project: Apache Drill
> Issue Type: Bug
>Reporter: Dmytriy Grinchenko
>Assignee: Dmytriy Grinchenko
>Priority: Critical
> Labels: ready-to-commit
> Fix For: 1.17.0
>
>
> The decimal cast may lead to issues with VarDercimal transformation and
> issues at uml functions which using casthigh under the hood
> Example:
> {code}
> apache drill> select casthigh(cast(1025.0 as decimal(28,8)));
> Error: SYSTEM ERROR: CompileException: Line 25, Column 60: "isSet" is neither
> a method, a field, nor a member class of
> "org.apache.drill.exec.expr.holders.VarDecimalHolder"
> Fragment 0:0
> Please, refer to logs for more information.
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-7317) Close ClassLoaders used for udf jars uploading when closing FunctionImplementationRegistry
[
https://issues.apache.org/jira/browse/DRILL-7317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi updated DRILL-7317:
---
Labels: ready-to-commit (was: )
> Close ClassLoaders used for udf jars uploading when closing
> FunctionImplementationRegistry
> --
>
> Key: DRILL-7317
> URL: https://issues.apache.org/jira/browse/DRILL-7317
> Project: Apache Drill
> Issue Type: Task
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.17.0
>
>
> For the case when {{FunctionImplementationRegistry}} is closed, class loaders
> which were used for uploading jars are still open and don't close file
> descriptors for these jars.
> The proposal is to unregister jars to close its {{ClassLoader}} when
> {{FunctionImplementationRegistry}} is closed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-7314) Use TupleMetadata instead of concrete implementation
[ https://issues.apache.org/jira/browse/DRILL-7314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Volodymyr Vysotskyi updated DRILL-7314: --- Labels: ready-to-commit (was: ) > Use TupleMetadata instead of concrete implementation > > > Key: DRILL-7314 > URL: https://issues.apache.org/jira/browse/DRILL-7314 > Project: Apache Drill > Issue Type: Task >Reporter: Arina Ielchiieva >Assignee: Arina Ielchiieva >Priority: Major > Labels: ready-to-commit > Fix For: 1.17.0 > > > Use TupleMetadata instead of concrete implementation. > Currently TupleSchema is used during ser / de since TupleMetadata is > interface. This Jira aims to support TupleMetadata ser / de instead of > concrete implementation. > This would be achieved using json sub types. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-7316) Move classes from org.apache.drill.metastore into org.apache.drill.exec.metastore package in java-exec module
[
https://issues.apache.org/jira/browse/DRILL-7316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi updated DRILL-7316:
---
Reviewer: Igor Guzenko (was: Arina Ielchiieva)
> Move classes from org.apache.drill.metastore into
> org.apache.drill.exec.metastore package in java-exec module
> -
>
> Key: DRILL-7316
> URL: https://issues.apache.org/jira/browse/DRILL-7316
> Project: Apache Drill
> Issue Type: Task
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.17.0
>
>
> Move classes from {{org.apache.drill.metastore}} into
> {{org.apache.drill.exec.metastore}} package in {{java-exec}} module
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-7315) Revise precision and scale order in the method arguments
[ https://issues.apache.org/jira/browse/DRILL-7315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Volodymyr Vysotskyi updated DRILL-7315: --- Reviewer: Igor Guzenko (was: Arina Ielchiieva) > Revise precision and scale order in the method arguments > > > Key: DRILL-7315 > URL: https://issues.apache.org/jira/browse/DRILL-7315 > Project: Apache Drill > Issue Type: Task >Affects Versions: 1.16.0 >Reporter: Volodymyr Vysotskyi >Assignee: Volodymyr Vysotskyi >Priority: Major > Fix For: 1.17.0 > > > The current code has different variations of scale and precision orderings in > the method arguments. The goal for this Jira is to make it more consistent. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-7317) Close ClassLoaders used for udf jars uploading when closing FunctionImplementationRegistry
[
https://issues.apache.org/jira/browse/DRILL-7317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi updated DRILL-7317:
---
Reviewer: Arina Ielchiieva
> Close ClassLoaders used for udf jars uploading when closing
> FunctionImplementationRegistry
> --
>
> Key: DRILL-7317
> URL: https://issues.apache.org/jira/browse/DRILL-7317
> Project: Apache Drill
> Issue Type: Task
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.17.0
>
>
> For the case when {{FunctionImplementationRegistry}} is closed, class loaders
> which were used for uploading jars are still open and don't close file
> descriptors for these jars.
> The proposal is to unregister jars to close its {{ClassLoader}} when
> {{FunctionImplementationRegistry}} is closed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-7317) Close ClassLoaders used for udf jars uploading when closing FunctionImplementationRegistry
Volodymyr Vysotskyi created DRILL-7317:
--
Summary: Close ClassLoaders used for udf jars uploading when
closing FunctionImplementationRegistry
Key: DRILL-7317
URL: https://issues.apache.org/jira/browse/DRILL-7317
Project: Apache Drill
Issue Type: Task
Reporter: Volodymyr Vysotskyi
Assignee: Volodymyr Vysotskyi
Fix For: 1.17.0
For the case when {{FunctionImplementationRegistry}} is closed, class loaders
which were used for uploading jars are still open and don't close file
descriptors for these jars.
The proposal is to unregister jars to close its {{ClassLoader}} when
{{FunctionImplementationRegistry}} is closed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-7316) Move classes from org.apache.drill.metastore into org.apache.drill.exec.metastore package in java-exec module
Volodymyr Vysotskyi created DRILL-7316:
--
Summary: Move classes from org.apache.drill.metastore into
org.apache.drill.exec.metastore package in java-exec module
Key: DRILL-7316
URL: https://issues.apache.org/jira/browse/DRILL-7316
Project: Apache Drill
Issue Type: Task
Reporter: Volodymyr Vysotskyi
Assignee: Volodymyr Vysotskyi
Fix For: 1.17.0
Move classes from {{org.apache.drill.metastore}} into
{{org.apache.drill.exec.metastore}} package in {{java-exec}} module
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-7313) Use Hive schema for MaprDB native reader when field was empty and support all text mode
[
https://issues.apache.org/jira/browse/DRILL-7313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi updated DRILL-7313:
---
Description:
Currently, when an external Hive MaprDB table is queried using hive plugin with
enabled {{store.hive.maprdb_json.optimize_scan_with_native_reader}}, some
queries may fail due to soft schema change, though Hive knows actual data types.
For example, when we have a table with several fields, and one of them has only
several non-null values, queries with grouping by such field will fail due to
schema change.
The goal of this Jira is to allow using types from Hive when a non-existing
field is created, so it will allow avoiding such issues.
In the scope of this Jira is also added new session option
{{store.hive.maprdb_json.all_text_mode}} to read all data from the maprDB JSON
tables as VARCHAR when hive plugin is used and Drill native MaprDB JSON reader
usage is enabled.
was:
Currently, when an external Hive MaprDB table is queried using hive plugin with
enabled {{store.hive.maprdb_json.optimize_scan_with_native_reader}}, some
queries may fail due to soft schema change, though Hive knows actual data types.
For example, when we have a table with several fields, and one of them has only
several non-null values, queries with grouping by such field will fail due to
schema change.
The goal of this Jira is to allow using types from Hive when a non-existing
field is created, so it will allow avoiding such issues.
> Use Hive schema for MaprDB native reader when field was empty and support all
> text mode
> ---
>
> Key: DRILL-7313
> URL: https://issues.apache.org/jira/browse/DRILL-7313
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.16.0
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Labels: doc-impacting, ready-to-commit
> Fix For: 1.17.0
>
>
> Currently, when an external Hive MaprDB table is queried using hive plugin
> with enabled {{store.hive.maprdb_json.optimize_scan_with_native_reader}},
> some queries may fail due to soft schema change, though Hive knows actual
> data types.
> For example, when we have a table with several fields, and one of them has
> only several non-null values, queries with grouping by such field will fail
> due to schema change.
> The goal of this Jira is to allow using types from Hive when a non-existing
> field is created, so it will allow avoiding such issues.
> In the scope of this Jira is also added new session option
> {{store.hive.maprdb_json.all_text_mode}} to read all data from the maprDB
> JSON tables as VARCHAR when hive plugin is used and Drill native MaprDB JSON
> reader usage is enabled.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-7200) Update Calcite to 1.19.0 / 1.20.0
[ https://issues.apache.org/jira/browse/DRILL-7200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Volodymyr Vysotskyi updated DRILL-7200: --- Labels: ready-to-commit (was: ) > Update Calcite to 1.19.0 / 1.20.0 > - > > Key: DRILL-7200 > URL: https://issues.apache.org/jira/browse/DRILL-7200 > Project: Apache Drill > Issue Type: Task >Affects Versions: 1.16.0 >Reporter: Bohdan Kazydub >Assignee: Bohdan Kazydub >Priority: Major > Labels: ready-to-commit > Fix For: 1.17.0 > > > Calcite has released the 1.19.0 version. Upgrade Calcite dependency in Drill > to the newest version. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-7315) Revise precision and scale order in the method arguments
Volodymyr Vysotskyi created DRILL-7315: -- Summary: Revise precision and scale order in the method arguments Key: DRILL-7315 URL: https://issues.apache.org/jira/browse/DRILL-7315 Project: Apache Drill Issue Type: Task Affects Versions: 1.16.0 Reporter: Volodymyr Vysotskyi Assignee: Volodymyr Vysotskyi Fix For: 1.17.0 The current code has different variations of scale and precision orderings in the method arguments. The goal for this Jira is to make it more consistent. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-7313) Use Hive schema for MaprDB native reader when field was empty
Volodymyr Vysotskyi created DRILL-7313:
--
Summary: Use Hive schema for MaprDB native reader when field was
empty
Key: DRILL-7313
URL: https://issues.apache.org/jira/browse/DRILL-7313
Project: Apache Drill
Issue Type: Task
Affects Versions: 1.16.0
Reporter: Volodymyr Vysotskyi
Assignee: Volodymyr Vysotskyi
Fix For: 1.17.0
Currently, when an external Hive MaprDB table is queried using hive plugin with
enabled {{store.hive.maprdb_json.optimize_scan_with_native_reader}}, some
queries may fail due to soft schema change, though Hive knows actual data types.
For example, when we have a table with several fields, and one of them has only
several non-null values, queries with grouping by such field will fail due to
schema change.
The goal of this Jira is to allow using types from Hive when a non-existing
field is created, so it will allow avoiding such issues.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-7310) Move schema-related classes from exec module to be able to use them in metastore module
Volodymyr Vysotskyi created DRILL-7310:
--
Summary: Move schema-related classes from exec module to be able
to use them in metastore module
Key: DRILL-7310
URL: https://issues.apache.org/jira/browse/DRILL-7310
Project: Apache Drill
Issue Type: Sub-task
Reporter: Volodymyr Vysotskyi
Assignee: Volodymyr Vysotskyi
Fix For: 1.17.0
Currently, most of the schema related classes are placed in the {{exec}}
module, but some of them should be used in {{metastore}} module. {{metastore}}
module doesn't have a dependency onto exec one.
The solution is to move these classes from {{exec}} into another module which
is used by {{metastore}}, so they will be accessible for {{metastore}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6711) Use jitpack repository for Drill Calcite project artifacts instead of repository.mapr.com
[ https://issues.apache.org/jira/browse/DRILL-6711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Volodymyr Vysotskyi updated DRILL-6711: --- Reviewer: Arina Ielchiieva > Use jitpack repository for Drill Calcite project artifacts instead of > repository.mapr.com > - > > Key: DRILL-6711 > URL: https://issues.apache.org/jira/browse/DRILL-6711 > Project: Apache Drill > Issue Type: Task >Reporter: Arina Ielchiieva >Assignee: Volodymyr Vysotskyi >Priority: Major > Fix For: 1.17.0 > > > Simplify deployment of Drill Calcite project artifacts by using > [https://jitpack.io/]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6711) Use jitpack repository for Drill Calcite project artifacts instead of repository.mapr.com
[ https://issues.apache.org/jira/browse/DRILL-6711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Volodymyr Vysotskyi updated DRILL-6711: --- Description: Simplify deployment of Drill Calcite project artifacts by using [https://jitpack.io/]. (was: Publish Drill Calcite project artifacts to Apache maven repository instead of a proprietary one.) > Use jitpack repository for Drill Calcite project artifacts instead of > repository.mapr.com > - > > Key: DRILL-6711 > URL: https://issues.apache.org/jira/browse/DRILL-6711 > Project: Apache Drill > Issue Type: Task >Reporter: Arina Ielchiieva >Assignee: Volodymyr Vysotskyi >Priority: Major > Fix For: 1.17.0 > > > Simplify deployment of Drill Calcite project artifacts by using > [https://jitpack.io/]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6711) Use jitpack repository for Drill Calcite project artifacts instead of repository.mapr.com
[ https://issues.apache.org/jira/browse/DRILL-6711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Volodymyr Vysotskyi updated DRILL-6711: --- Summary: Use jitpack repository for Drill Calcite project artifacts instead of repository.mapr.com (was: Publish Drill Calcite project artifacts to Apache maven repository ) > Use jitpack repository for Drill Calcite project artifacts instead of > repository.mapr.com > - > > Key: DRILL-6711 > URL: https://issues.apache.org/jira/browse/DRILL-6711 > Project: Apache Drill > Issue Type: Task >Reporter: Arina Ielchiieva >Assignee: Volodymyr Vysotskyi >Priority: Major > Fix For: 1.17.0 > > > Publish Drill Calcite project artifacts to Apache maven repository instead of > a proprietary one. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (DRILL-6711) Publish Drill Calcite project artifacts to Apache maven repository
[ https://issues.apache.org/jira/browse/DRILL-6711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Volodymyr Vysotskyi reopened DRILL-6711: Reopened due to discussion in [https://lists.apache.org/thread.html/7a05ad9031f1f3155d0ff0701091efdb079ee328ef0299d80fd6f140@%3Cdev.drill.apache.org%3E] > Publish Drill Calcite project artifacts to Apache maven repository > --- > > Key: DRILL-6711 > URL: https://issues.apache.org/jira/browse/DRILL-6711 > Project: Apache Drill > Issue Type: Task >Reporter: Arina Ielchiieva >Assignee: Volodymyr Vysotskyi >Priority: Major > > Publish Drill Calcite project artifacts to Apache maven repository instead of > a proprietary one. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6711) Publish Drill Calcite project artifacts to Apache maven repository
[ https://issues.apache.org/jira/browse/DRILL-6711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Volodymyr Vysotskyi updated DRILL-6711: --- Fix Version/s: 1.17.0 > Publish Drill Calcite project artifacts to Apache maven repository > --- > > Key: DRILL-6711 > URL: https://issues.apache.org/jira/browse/DRILL-6711 > Project: Apache Drill > Issue Type: Task >Reporter: Arina Ielchiieva >Assignee: Volodymyr Vysotskyi >Priority: Major > Fix For: 1.17.0 > > > Publish Drill Calcite project artifacts to Apache maven repository instead of > a proprietary one. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-7271) Refactor Metadata interfaces and classes to contain all needed information for the File based Metastore
[
https://issues.apache.org/jira/browse/DRILL-7271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi updated DRILL-7271:
---
Description:
1. Merge info from metadataStatistics + statisticsKinds into one holder:
Map.
2. Rename hasStatistics to hasDescriptiveStatistics
3. Remove drill-file-metastore-plugin
4. Move
org.apache.drill.exec.physical.base.AbstractGroupScanWithMetadata.MetadataLevel
to metadata module, rename to MetadataType and add new value: SEGMENT.
5. Add JSON ser/de for ColumnStatistics, StatisticsHolder.
6. Add new info classes:
{noformat}
class TableInfo {
String storagePlugin;
String workspace;
String name;
String type;
String owner;
}
class MetadataInfo {
public static final String GENERAL_INFO_KEY = "GENERAL_INFO";
public static final String DEFAULT_SEGMENT_KEY = "DEFAULT_SEGMENT";
MetadataType type (enum);
String key;
String identifier;
}
{noformat}
7. Modify existing metadata classes:
org.apache.drill.metastore.FileTableMetadata
{noformat}
missing fields
--
storagePlugin, workspace, tableType -> will be covered by TableInfo class
metadataType, metadataKey -> will be covered by MetadataInfo class
interestingColumns
fields to modify
private final Map tableStatistics;
private final Map statisticsKinds;
private final Set partitionKeys; -> Map
{noformat}
org.apache.drill.metastore.PartitionMetadata
{noformat}
missing fields
--
storagePlugin, workspace -> will be covered by TableInfo class
metadataType, metadataKey, metadataIdentifier -> will be covered by
MetadataInfo class
partitionValues (List)
location (String) (for directory level metadata) - directory location
fields to modify
private final Map tableStatistics;
private final Map statisticsKinds;
private final Set location; -> locations
{noformat}
org.apache.drill.metastore.FileMetadata
{noformat}
missing fields
--
storagePlugin, workspace -> will be covered by TableInfo class
metadataType, metadataKey, metadataIdentifier -> will be covered by
MetadataInfo class
path - path to file
fields to modify
private final Map tableStatistics;
private final Map statisticsKinds;
private final Path location; - should contain directory to which file belongs
{noformat}
org.apache.drill.metastore.RowGroupMetadata
{noformat}
missing fields
--
storagePlugin, workspace -> will be covered by TableInfo class
metadataType, metadataKey, metadataIdentifier -> will be covered by
MetadataInfo class
path - path to file
fields to modify
private final Map tableStatistics;
private final Map statisticsKinds;
private final Path location; - should contain directory to which file belongs
{noformat}
8. Remove org.apache.drill.exec package from metastore module.
9. Rename ColumnStatisticsImpl class.
10. Separate existing classes in org.apache.drill.metastore package into
sub-packages.
11. Rename FileTableMetadata -> BaseTableMetadata
12. TableMetadataProvider.getNonInterestingColumnsMeta() ->
getNonInterestingColumnsMetadata
13. Introduce segment-level metadata class:
{noformat}
class SegmentMetadata {
TableInfo tableInfo;
MetadataInfo metadataInfo;
SchemaPath column;
TupleMetadata schema;
String location;
Map columnsStatistics;
Map statistics;
List partitionValues;
List locations;
long lastModifiedTime;
}
{noformat}
h1. Segment metadata
In the fix for this Jira, one of the changes is introducing segment level
metadata.
For now, metadata hierarchy is the following:
- Table
- Segment
- Partition
- File
- Row group
Segment represents some a part of the table united using some specific
qualities. For example for file system tables, segment may correspond to
directories with its data. For hive tables, segment corresponds to hive
partitions.
In opposite, partition metadata, will correspond to "drill partitions". It is
groups of data which have the same values for specific columns within a file or
row group.
So filtering will be produced for table level, then for segments, after that
for partitions, for files and then for row groups.
was:
1. Merge info from metadataStatistics + statisticsKinds into one holder:
Map.
2. Rename hasStatistics to hasDescriptiveStatistics
3. Remove drill-file-metastore-plugin
4. Move
org.apache.drill.exec.physical.base.AbstractGroupScanWithMetadata.MetadataLevel
to metadata module, rename to MetadataType and add new value: SEGMENT.
5. Add JSON ser/de for ColumnStatistics, StatisticsHolder.
6. Add new info classes:
{noformat}
class TableInfo {
String storagePlugin;
String workspace;
String name;
String type;
String owner;
}
class MetadataInfo {
public static final String GENERAL_INFO_KEY = "GENERAL_INFO";
public static final String DEFAULT_SEGMENT_KEY = "DEFAULT_SEGMENT";
MetadataType type (enum);
String key;
String identifier;
}
[jira] [Updated] (DRILL-7302) Bump Apache Avro from 1.8.2 to 1.9.0
[ https://issues.apache.org/jira/browse/DRILL-7302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Volodymyr Vysotskyi updated DRILL-7302: --- Fix Version/s: 1.17.0 > Bump Apache Avro from 1.8.2 to 1.9.0 > > > Key: DRILL-7302 > URL: https://issues.apache.org/jira/browse/DRILL-7302 > Project: Apache Drill > Issue Type: Improvement >Reporter: Fokko Driesprong >Priority: Major > Labels: ready-to-commit > Fix For: 1.17.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-7302) Bump Apache Avro from 1.8.2 to 1.9.0
[ https://issues.apache.org/jira/browse/DRILL-7302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Volodymyr Vysotskyi updated DRILL-7302: --- Labels: ready-to-commit (was: ) > Bump Apache Avro from 1.8.2 to 1.9.0 > > > Key: DRILL-7302 > URL: https://issues.apache.org/jira/browse/DRILL-7302 > Project: Apache Drill > Issue Type: Improvement >Reporter: Fokko Driesprong >Priority: Major > Labels: ready-to-commit > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (DRILL-7302) Bump Apache Avro from 1.8.2 to 1.9.0
[ https://issues.apache.org/jira/browse/DRILL-7302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Volodymyr Vysotskyi reassigned DRILL-7302: -- Assignee: (was: Volodymyr Vysotskyi) > Bump Apache Avro from 1.8.2 to 1.9.0 > > > Key: DRILL-7302 > URL: https://issues.apache.org/jira/browse/DRILL-7302 > Project: Apache Drill > Issue Type: Improvement >Reporter: Fokko Driesprong >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (DRILL-7302) Bump Apache Avro from 1.8.2 to 1.9.0
[ https://issues.apache.org/jira/browse/DRILL-7302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Volodymyr Vysotskyi reassigned DRILL-7302: -- Assignee: Volodymyr Vysotskyi > Bump Apache Avro from 1.8.2 to 1.9.0 > > > Key: DRILL-7302 > URL: https://issues.apache.org/jira/browse/DRILL-7302 > Project: Apache Drill > Issue Type: Improvement >Reporter: Fokko Driesprong >Assignee: Volodymyr Vysotskyi >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-7297) Query hangs in planning stage when Error is thrown
[
https://issues.apache.org/jira/browse/DRILL-7297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi updated DRILL-7297:
---
Fix Version/s: 1.17.0
> Query hangs in planning stage when Error is thrown
> --
>
> Key: DRILL-7297
> URL: https://issues.apache.org/jira/browse/DRILL-7297
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.16.0
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.17.0
>
>
> Query hangs in the planning stage when Error (not OOM or AssertionError) is
> thrown during query planning. After canceling the query it will stay in
> Cancellation Requested state.
> Such error may be thrown due to the mistake in the code, including UDF. Since
> the user may provide custom UDFs, Drill should be able to handle such cases
> also.
> Steps to reproduce this issue:
> 1. Create UDF which throws Error in either {{eval()}} or {{setup()}} method
> (instructions how to create custom UDF may be found
> [here|https://drill.apache.org/docs/tutorial-develop-a-simple-function/].
> 2. Register custom UDF which throws an error (instruction is
> [here|https://drill.apache.org/docs/adding-custom-functions-to-drill-introduction/]).
> 3. Run the query with this UDF.
> After submitting the query, the following stack trace is printed:
> {noformat}
> Exception in thread "drill-executor-1" java.lang.Error
> at
> org.apache.drill.contrib.function.FunctionExample.setup(FunctionExample.java:19)
> at
> org.apache.drill.exec.expr.fn.interpreter.InterpreterEvaluator.evaluateFunction(InterpreterEvaluator.java:139)
> at
> org.apache.drill.exec.expr.fn.interpreter.InterpreterEvaluator$EvalVisitor.visitFunctionHolderExpression(InterpreterEvaluator.java:355)
> at
> org.apache.drill.exec.expr.fn.interpreter.InterpreterEvaluator$EvalVisitor.visitFunctionHolderExpression(InterpreterEvaluator.java:204)
> at
> org.apache.drill.common.expression.FunctionHolderExpression.accept(FunctionHolderExpression.java:53)
> at
> org.apache.drill.exec.expr.fn.interpreter.InterpreterEvaluator.evaluateConstantExpr(InterpreterEvaluator.java:70)
> at
> org.apache.drill.exec.planner.logical.DrillConstExecutor.reduce(DrillConstExecutor.java:152)
> at
> org.apache.calcite.rel.rules.ReduceExpressionsRule.reduceExpressionsInternal(ReduceExpressionsRule.java:620)
> at
> org.apache.calcite.rel.rules.ReduceExpressionsRule.reduceExpressions(ReduceExpressionsRule.java:541)
> at
> org.apache.calcite.rel.rules.ReduceExpressionsRule$ProjectReduceExpressionsRule.onMatch(ReduceExpressionsRule.java:288)
> at
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:212)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:643)
> at
> org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:339)
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform(DefaultSqlHandler.java:430)
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform(DefaultSqlHandler.java:370)
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToRawDrel(DefaultSqlHandler.java:250)
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:319)
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:177)
> at
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan(DrillSqlWorker.java:226)
> at
> org.apache.drill.exec.planner.sql.DrillSqlWorker.convertPlan(DrillSqlWorker.java:124)
> at
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:90)
> at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:593)
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:276)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> 4. Check that query is still in progress state, cancel query.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-7268) Read Hive array with parquet native reader
[ https://issues.apache.org/jira/browse/DRILL-7268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Volodymyr Vysotskyi updated DRILL-7268: --- Labels: ready-to-commit (was: ) > Read Hive array with parquet native reader > -- > > Key: DRILL-7268 > URL: https://issues.apache.org/jira/browse/DRILL-7268 > Project: Apache Drill > Issue Type: Sub-task >Reporter: Igor Guzenko >Assignee: Igor Guzenko >Priority: Major > Labels: ready-to-commit > Fix For: 1.17.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (DRILL-7297) Query hangs in planning stage when Error is thrown
[
https://issues.apache.org/jira/browse/DRILL-7297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi reassigned DRILL-7297:
--
Assignee: Volodymyr Vysotskyi
> Query hangs in planning stage when Error is thrown
> --
>
> Key: DRILL-7297
> URL: https://issues.apache.org/jira/browse/DRILL-7297
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.16.0
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
>
> Query hangs in the planning stage when Error (not OOM or AssertionError) is
> thrown during query planning. After canceling the query it will stay in
> Cancellation Requested state.
> Such error may be thrown due to the mistake in the code, including UDF. Since
> the user may provide custom UDFs, Drill should be able to handle such cases
> also.
> Steps to reproduce this issue:
> 1. Create UDF which throws Error in either {{eval()}} or {{setup()}} method
> (instructions how to create custom UDF may be found
> [here|https://drill.apache.org/docs/tutorial-develop-a-simple-function/].
> 2. Register custom UDF which throws an error (instruction is
> [here|https://drill.apache.org/docs/adding-custom-functions-to-drill-introduction/]).
> 3. Run the query with this UDF.
> After submitting the query, the following stack trace is printed:
> {noformat}
> Exception in thread "drill-executor-1" java.lang.Error
> at
> org.apache.drill.contrib.function.FunctionExample.setup(FunctionExample.java:19)
> at
> org.apache.drill.exec.expr.fn.interpreter.InterpreterEvaluator.evaluateFunction(InterpreterEvaluator.java:139)
> at
> org.apache.drill.exec.expr.fn.interpreter.InterpreterEvaluator$EvalVisitor.visitFunctionHolderExpression(InterpreterEvaluator.java:355)
> at
> org.apache.drill.exec.expr.fn.interpreter.InterpreterEvaluator$EvalVisitor.visitFunctionHolderExpression(InterpreterEvaluator.java:204)
> at
> org.apache.drill.common.expression.FunctionHolderExpression.accept(FunctionHolderExpression.java:53)
> at
> org.apache.drill.exec.expr.fn.interpreter.InterpreterEvaluator.evaluateConstantExpr(InterpreterEvaluator.java:70)
> at
> org.apache.drill.exec.planner.logical.DrillConstExecutor.reduce(DrillConstExecutor.java:152)
> at
> org.apache.calcite.rel.rules.ReduceExpressionsRule.reduceExpressionsInternal(ReduceExpressionsRule.java:620)
> at
> org.apache.calcite.rel.rules.ReduceExpressionsRule.reduceExpressions(ReduceExpressionsRule.java:541)
> at
> org.apache.calcite.rel.rules.ReduceExpressionsRule$ProjectReduceExpressionsRule.onMatch(ReduceExpressionsRule.java:288)
> at
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:212)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:643)
> at
> org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:339)
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform(DefaultSqlHandler.java:430)
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform(DefaultSqlHandler.java:370)
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToRawDrel(DefaultSqlHandler.java:250)
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:319)
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:177)
> at
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan(DrillSqlWorker.java:226)
> at
> org.apache.drill.exec.planner.sql.DrillSqlWorker.convertPlan(DrillSqlWorker.java:124)
> at
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:90)
> at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:593)
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:276)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> 4. Check that query is still in progress state, cancel query.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-7297) Query hangs in planning stage when Error is thrown
Volodymyr Vysotskyi created DRILL-7297:
--
Summary: Query hangs in planning stage when Error is thrown
Key: DRILL-7297
URL: https://issues.apache.org/jira/browse/DRILL-7297
Project: Apache Drill
Issue Type: Bug
Affects Versions: 1.16.0
Reporter: Volodymyr Vysotskyi
Query hangs in the planning stage when Error (not OOM or AssertionError) is
thrown during query planning. After canceling the query it will stay in
Cancellation Requested state.
Such error may be thrown due to the mistake in the code, including UDF. Since
the user may provide custom UDFs, Drill should be able to handle such cases
also.
Steps to reproduce this issue:
1. Create UDF which throws Error in either {{eval()}} or {{setup()}} method
(instructions how to create custom UDF may be found
[here|https://drill.apache.org/docs/tutorial-develop-a-simple-function/].
2. Register custom UDF which throws an error (instruction is
[here|https://drill.apache.org/docs/adding-custom-functions-to-drill-introduction/]).
3. Run the query with this UDF.
After submitting the query, the following stack trace is printed:
{noformat}
Exception in thread "drill-executor-1" java.lang.Error
at
org.apache.drill.contrib.function.FunctionExample.setup(FunctionExample.java:19)
at
org.apache.drill.exec.expr.fn.interpreter.InterpreterEvaluator.evaluateFunction(InterpreterEvaluator.java:139)
at
org.apache.drill.exec.expr.fn.interpreter.InterpreterEvaluator$EvalVisitor.visitFunctionHolderExpression(InterpreterEvaluator.java:355)
at
org.apache.drill.exec.expr.fn.interpreter.InterpreterEvaluator$EvalVisitor.visitFunctionHolderExpression(InterpreterEvaluator.java:204)
at
org.apache.drill.common.expression.FunctionHolderExpression.accept(FunctionHolderExpression.java:53)
at
org.apache.drill.exec.expr.fn.interpreter.InterpreterEvaluator.evaluateConstantExpr(InterpreterEvaluator.java:70)
at
org.apache.drill.exec.planner.logical.DrillConstExecutor.reduce(DrillConstExecutor.java:152)
at
org.apache.calcite.rel.rules.ReduceExpressionsRule.reduceExpressionsInternal(ReduceExpressionsRule.java:620)
at
org.apache.calcite.rel.rules.ReduceExpressionsRule.reduceExpressions(ReduceExpressionsRule.java:541)
at
org.apache.calcite.rel.rules.ReduceExpressionsRule$ProjectReduceExpressionsRule.onMatch(ReduceExpressionsRule.java:288)
at
org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:212)
at
org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:643)
at
org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:339)
at
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform(DefaultSqlHandler.java:430)
at
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform(DefaultSqlHandler.java:370)
at
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToRawDrel(DefaultSqlHandler.java:250)
at
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:319)
at
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:177)
at
org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan(DrillSqlWorker.java:226)
at
org.apache.drill.exec.planner.sql.DrillSqlWorker.convertPlan(DrillSqlWorker.java:124)
at
org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:90)
at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:593)
at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:276)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{noformat}
4. Check that query is still in progress state, cancel query.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-7283) Unit Tests for UDFs in Contrib folder failing
[
https://issues.apache.org/jira/browse/DRILL-7283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16865868#comment-16865868
]
Volodymyr Vysotskyi commented on DRILL-7283:
[~cgivre], did you have a chance to check it or try running these tests after
the project is built?
> Unit Tests for UDFs in Contrib folder failing
> -
>
> Key: DRILL-7283
> URL: https://issues.apache.org/jira/browse/DRILL-7283
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.17.0
>Reporter: Charles Givre
>Priority: Critical
>
> The unit tests for UDFs in the contrib/ folder seem to be failing. It would
> seem that the ClusterFixture doesn't seem to be missing configuration
> variables. I've set a series of them and it keeps asking me for more.
>
> The errors are shown below:
> {{java.lang.IllegalStateException: Cluster fixture setup failed}}{{at
> org.apache.drill.test.ClusterFixture.(ClusterFixture.java:152)}}
> {{ at
> org.apache.drill.test.ClusterFixtureBuilder.build(ClusterFixtureBuilder.java:283)}}
> {{ at org.apache.drill.test.ClusterTest.startCluster(ClusterTest.java:83)}}
> {{ at
> org.apache.drill.exec.udfs.TestCryptoFunctions.setup(TestCryptoFunctions.java:40)}}
> {{Caused by: com.typesafe.config.ConfigException$Missing: No configuration
> setting found for key 'drill.exec.profiles'}}
> {{ at com.typesafe.config.impl.SimpleConfig.findKey(SimpleConfig.java:115)}}
> {{ at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:138)}}
> {{ at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:142)}}
> {{ at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:142)}}
> {{ at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:150)}}
> {{ at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:155)}}
> {{ at
> com.typesafe.config.impl.SimpleConfig.getBoolean(SimpleConfig.java:165)}}
> {{ at
> org.apache.drill.common.config.NestedConfig.getBoolean(NestedConfig.java:86)}}
> {{ at
> org.apache.drill.common.config.DrillConfig.getBoolean(DrillConfig.java:44)}}
> {{ at
> org.apache.drill.common.config.NestedConfig.getBoolean(NestedConfig.java:86)}}
> {{ at
> org.apache.drill.common.config.DrillConfig.getBoolean(DrillConfig.java:44)}}
> {{ at org.apache.drill.exec.server.Drillbit.(Drillbit.java:186)}}
> {{ at org.apache.drill.exec.server.Drillbit.(Drillbit.java:140)}}
> {{ at
> org.apache.drill.test.ClusterFixture.startDrillbits(ClusterFixture.java:228)}}
> {{ at org.apache.drill.test.ClusterFixture.(ClusterFixture.java:146)}}
> {{ ... 3 more}}
>
> The GIS unit tests fail with the error below:
> {{com.typesafe.config.ConfigException$Missing: No configuration setting found
> for key 'drill.exec.options'}}{{at
> com.typesafe.config.impl.SimpleConfig.findKey(SimpleConfig.java:115)}}
> {{ at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:138)}}
> {{ at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:142)}}
> {{ at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:142)}}
> {{ at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:150)}}
> {{ at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:155)}}
> {{ at
> com.typesafe.config.impl.SimpleConfig.getBoolean(SimpleConfig.java:165)}}
> {{ at
> org.apache.drill.common.config.NestedConfig.getBoolean(NestedConfig.java:86)}}
> {{ at
> org.apache.drill.common.config.DrillConfig.getBoolean(DrillConfig.java:44)}}
> {{ at
> org.apache.drill.exec.server.options.SystemOptionManager.populateDefaultValues(SystemOptionManager.java:475)}}
> {{ at
> org.apache.drill.exec.server.options.SystemOptionManager.(SystemOptionManager.java:331)}}
> {{ at
> org.apache.drill.exec.server.options.SystemOptionManager.(SystemOptionManager.java:321)}}
> {{ at org.apache.drill.exec.ExecTest.setupOptionManager(ExecTest.java:76)}}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-7294) Prevent generating java beans using protostuff to avoid overriding classes with the same simple name declared as nested in the proto files
Volodymyr Vysotskyi created DRILL-7294:
--
Summary: Prevent generating java beans using protostuff to avoid
overriding classes with the same simple name declared as nested in the proto
files
Key: DRILL-7294
URL: https://issues.apache.org/jira/browse/DRILL-7294
Project: Apache Drill
Issue Type: Bug
Affects Versions: 1.16.0
Reporter: Volodymyr Vysotskyi
Assignee: Volodymyr Vysotskyi
Fix For: 1.17.0
Currently, {{protostuff-maven-plugin}} generates java-bean classes from proto
files. But these classes already generated by protobuf, the single difference
is that they are placed in a different package, and preserved the nesting of
the classes as they are declared in the proto files.
protostuff creates new files for nested classes, and it causes problems for the
case when several nested classes have the same name - they override each other,
for example here is Travis failure caused by this problem:
https://travis-ci.org/apache/drill/jobs/545013395
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-7271) Refactor Metadata interfaces and classes to contain all needed information for the File based Metastore
[
https://issues.apache.org/jira/browse/DRILL-7271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi updated DRILL-7271:
---
Description:
1. Merge info from metadataStatistics + statisticsKinds into one holder:
Map.
2. Rename hasStatistics to hasDescriptiveStatistics
3. Remove drill-file-metastore-plugin
4. Move
org.apache.drill.exec.physical.base.AbstractGroupScanWithMetadata.MetadataLevel
to metadata module, rename to MetadataType and add new value: DIRECTORY.
5. Add JSON ser/de for ColumnStatistics, StatisticsHolder.
6. Add new info classes:
{noformat}
class TableInfo {
String storagePlugin;
String workspace;
String name;
String type;
String owner;
}
class MetadataInfo {
public static final String GENERAL_INFO_KEY = "GENERAL_INFO";
public static final String DEFAULT_PARTITION_KEY = "DEFAULT_PARTITION";
MetadataType type (enum);
String key;
String identifier;
}
{noformat}
7. Modify existing metadata classes:
org.apache.drill.metastore.FileTableMetadata
{noformat}
missing fields
--
storagePlugin, workspace, tableType -> will be covered by TableInfo class
metadataType, metadataKey -> will be covered by MetadataInfo class
interestingColumns
fields to modify
private final Map tableStatistics;
private final Map statisticsKinds;
private final Set partitionKeys; -> Map
{noformat}
org.apache.drill.metastore.PartitionMetadata
{noformat}
missing fields
--
storagePlugin, workspace -> will be covered by TableInfo class
metadataType, metadataKey, metadataIdentifier -> will be covered by
MetadataInfo class
partitionValues (List)
location (String) (for directory level metadata) - directory location
fields to modify
private final Map tableStatistics;
private final Map statisticsKinds;
private final Set location; -> locations
{noformat}
org.apache.drill.metastore.FileMetadata
{noformat}
missing fields
--
storagePlugin, workspace -> will be covered by TableInfo class
metadataType, metadataKey, metadataIdentifier -> will be covered by
MetadataInfo class
path - path to file
fields to modify
private final Map tableStatistics;
private final Map statisticsKinds;
private final Path location; - should contain directory to which file belongs
{noformat}
org.apache.drill.metastore.RowGroupMetadata
{noformat}
missing fields
--
storagePlugin, workspace -> will be covered by TableInfo class
metadataType, metadataKey, metadataIdentifier -> will be covered by
MetadataInfo class
path - path to file
fields to modify
private final Map tableStatistics;
private final Map statisticsKinds;
private final Path location; - should contain directory to which file belongs
{noformat}
8. Remove org.apache.drill.exec package from metastore module.
9. Rename ColumnStatisticsImpl class.
10. Separate existing classes in org.apache.drill.metastore package into
sub-packages.
11. Rename FileTableMetadata -> BaseTableMetadata
12. TableMetadataProvider.getNonInterestingColumnsMeta() ->
getNonInterestingColumnsMetadata
13. Introduce segment-level metadata class:
{noformat}
class SegmentMetadata {
TableInfo tableInfo;
MetadataInfo metadataInfo;
SchemaPath column;
TupleMetadata schema;
String location;
Map columnsStatistics;
Map statistics;
List partitionValues;
List locations;
long lastModifiedTime;
}
{noformat}
was:
1. Merge info from metadataStatistics + statisticsKinds into one holder:
Map.
2. Rename hasStatistics to hasDescriptiveStatistics
3. Remove drill-file-metastore-plugin
4. Move
org.apache.drill.exec.physical.base.AbstractGroupScanWithMetadata.MetadataLevel
to metadata module, rename to MetadataType and add new value: DIRECTORY.
5. Add JSON ser/de for ColumnStatistics, StatisticsHolder.
6. Add new info classes:
{noformat}
class TableInfo {
String storagePlugin;
String workspace;
String name;
String type;
String owner;
}
class MetadataInfo {
public static final String GENERAL_INFO_KEY = "GENERAL_INFO";
public static final String DEFAULT_PARTITION_KEY = "DEFAULT_PARTITION";
MetadataType type (enum);
String key;
String identifier;
}
{noformat}
7. Modify existing metadata classes:
org.apache.drill.metastore.FileTableMetadata
{noformat}
missing fields
--
storagePlugin, workspace, tableType -> will be covered by TableInfo class
metadataType, metadataKey -> will be covered by MetadataInfo class
interestingColumns
fields to modify
private final Map tableStatistics;
private final Map statisticsKinds;
private final Set partitionKeys; -> Map
{noformat}
org.apache.drill.metastore.PartitionMetadata
{noformat}
missing fields
--
storagePlugin, workspace -> will be covered by TableInfo class
metadataType, metadataKey, metadataIdentifier -> will be covered by
MetadataInfo class
partitionValues (List)
location
[jira] [Commented] (DRILL-7283) Unit Tests for UDFs in Contrib folder failing
[
https://issues.apache.org/jira/browse/DRILL-7283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16856375#comment-16856375
]
Volodymyr Vysotskyi commented on DRILL-7283:
[~cgivre], please check that java-exec module is built and its target folder
wasn't cleaned before running the tests.
> Unit Tests for UDFs in Contrib folder failing
> -
>
> Key: DRILL-7283
> URL: https://issues.apache.org/jira/browse/DRILL-7283
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.17.0
>Reporter: Charles Givre
>Priority: Critical
>
> The unit tests for UDFs in the contrib/ folder seem to be failing. It would
> seem that the ClusterFixture doesn't seem to be missing configuration
> variables. I've set a series of them and it keeps asking me for more.
>
> The errors are shown below:
> {{java.lang.IllegalStateException: Cluster fixture setup failed}}{{at
> org.apache.drill.test.ClusterFixture.(ClusterFixture.java:152)}}
> {{ at
> org.apache.drill.test.ClusterFixtureBuilder.build(ClusterFixtureBuilder.java:283)}}
> {{ at org.apache.drill.test.ClusterTest.startCluster(ClusterTest.java:83)}}
> {{ at
> org.apache.drill.exec.udfs.TestCryptoFunctions.setup(TestCryptoFunctions.java:40)}}
> {{Caused by: com.typesafe.config.ConfigException$Missing: No configuration
> setting found for key 'drill.exec.profiles'}}
> {{ at com.typesafe.config.impl.SimpleConfig.findKey(SimpleConfig.java:115)}}
> {{ at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:138)}}
> {{ at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:142)}}
> {{ at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:142)}}
> {{ at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:150)}}
> {{ at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:155)}}
> {{ at
> com.typesafe.config.impl.SimpleConfig.getBoolean(SimpleConfig.java:165)}}
> {{ at
> org.apache.drill.common.config.NestedConfig.getBoolean(NestedConfig.java:86)}}
> {{ at
> org.apache.drill.common.config.DrillConfig.getBoolean(DrillConfig.java:44)}}
> {{ at
> org.apache.drill.common.config.NestedConfig.getBoolean(NestedConfig.java:86)}}
> {{ at
> org.apache.drill.common.config.DrillConfig.getBoolean(DrillConfig.java:44)}}
> {{ at org.apache.drill.exec.server.Drillbit.(Drillbit.java:186)}}
> {{ at org.apache.drill.exec.server.Drillbit.(Drillbit.java:140)}}
> {{ at
> org.apache.drill.test.ClusterFixture.startDrillbits(ClusterFixture.java:228)}}
> {{ at org.apache.drill.test.ClusterFixture.(ClusterFixture.java:146)}}
> {{ ... 3 more}}
>
> The GIS unit tests fail with the error below:
> {{com.typesafe.config.ConfigException$Missing: No configuration setting found
> for key 'drill.exec.options'}}{{at
> com.typesafe.config.impl.SimpleConfig.findKey(SimpleConfig.java:115)}}
> {{ at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:138)}}
> {{ at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:142)}}
> {{ at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:142)}}
> {{ at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:150)}}
> {{ at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:155)}}
> {{ at
> com.typesafe.config.impl.SimpleConfig.getBoolean(SimpleConfig.java:165)}}
> {{ at
> org.apache.drill.common.config.NestedConfig.getBoolean(NestedConfig.java:86)}}
> {{ at
> org.apache.drill.common.config.DrillConfig.getBoolean(DrillConfig.java:44)}}
> {{ at
> org.apache.drill.exec.server.options.SystemOptionManager.populateDefaultValues(SystemOptionManager.java:475)}}
> {{ at
> org.apache.drill.exec.server.options.SystemOptionManager.(SystemOptionManager.java:331)}}
> {{ at
> org.apache.drill.exec.server.options.SystemOptionManager.(SystemOptionManager.java:321)}}
> {{ at org.apache.drill.exec.ExecTest.setupOptionManager(ExecTest.java:76)}}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-7236) SqlLine 1.8 upgrade
[ https://issues.apache.org/jira/browse/DRILL-7236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Volodymyr Vysotskyi updated DRILL-7236: --- Labels: ready-to-commit (was: ) > SqlLine 1.8 upgrade > --- > > Key: DRILL-7236 > URL: https://issues.apache.org/jira/browse/DRILL-7236 > Project: Apache Drill > Issue Type: Task >Affects Versions: 1.16.0 >Reporter: Arina Ielchiieva >Assignee: Arina Ielchiieva >Priority: Major > Labels: ready-to-commit > Fix For: 1.17.0 > > > SqlLine 1.8 upgrade -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-7251) Read Hive array w/o nulls
[ https://issues.apache.org/jira/browse/DRILL-7251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Volodymyr Vysotskyi updated DRILL-7251: --- Labels: ready-to-commit (was: ) > Read Hive array w/o nulls > - > > Key: DRILL-7251 > URL: https://issues.apache.org/jira/browse/DRILL-7251 > Project: Apache Drill > Issue Type: Sub-task > Components: Storage - Hive >Reporter: Igor Guzenko >Assignee: Igor Guzenko >Priority: Major > Labels: ready-to-commit > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-7281) Unable to submit physical plan with maprdb-json-scan operator
Volodymyr Vysotskyi created DRILL-7281:
--
Summary: Unable to submit physical plan with maprdb-json-scan
operator
Key: DRILL-7281
URL: https://issues.apache.org/jira/browse/DRILL-7281
Project: Apache Drill
Issue Type: Bug
Affects Versions: 1.16.0
Reporter: Volodymyr Vysotskyi
Assignee: Volodymyr Vysotskyi
When submitting the following plan which corresponds to a simple query on
MaprDB table:
{code:sql}
select * from dfs.`/tmp/nulls`
{code}
{noformat}
{
"head" : {
"version" : 1,
"generator" : {
"type" : "ExplainHandler",
"info" : ""
},
"type" : "APACHE_DRILL_PHYSICAL",
"options" : [ {
"kind" : "BOOLEAN",
"accessibleScopes" : "ALL",
"name" : "store.hive.maprdb_json.optimize_scan_with_native_reader",
"bool_val" : true,
"scope" : "SESSION"
}, {
"kind" : "BOOLEAN",
"accessibleScopes" : "ALL",
"name" : "planner.enable_hashagg",
"bool_val" : false,
"scope" : "SESSION"
}, {
"kind" : "LONG",
"accessibleScopes" : "ALL",
"name" : "planner.slice_target",
"num_val" : 1,
"scope" : "SESSION"
} ],
"queue" : 0,
"hasResourcePlan" : false,
"resultMode" : "EXEC"
},
"graph" : [ {
"pop" : "maprdb-json-scan",
"@id" : 2,
"userName" : "mapr",
"scanSpec" : {
"tableName" : "/tmp/nulls",
"indexDesc" : null,
"startRow" : "",
"stopRow" : "",
"serializedFilter" : null,
"secondaryIndex" : false
},
"storage" : {
"type" : "file",
"connection" : "maprfs:///",
"config" : null,
"workspaces" : {
"tmp" : {
"location" : "/tmp",
"writable" : true,
"defaultInputFormat" : null,
"allowAccessOutsideWorkspace" : false
},
"root" : {
"location" : "/",
"writable" : false,
"defaultInputFormat" : null,
"allowAccessOutsideWorkspace" : false
}
},
"formats" : {
"psv" : {
"type" : "text",
"extensions" : [ "tbl" ],
"delimiter" : "|"
},
"csv" : {
"type" : "text",
"extensions" : [ "csv" ],
"delimiter" : ","
},
"tsv" : {
"type" : "text",
"extensions" : [ "tsv" ],
"delimiter" : "\t"
},
"httpd" : {
"type" : "httpd",
"logFormat" : "%h %t \"%r\" %>s %b \"%{Referer}i\""
},
"parquet" : {
"type" : "parquet"
},
"json" : {
"type" : "json",
"extensions" : [ "json" ]
},
"pcap" : {
"type" : "pcap"
},
"pcapng" : {
"type" : "pcapng",
"extensions" : [ "pcapng" ]
},
"avro" : {
"type" : "avro"
},
"sequencefile" : {
"type" : "sequencefile",
"extensions" : [ "seq" ]
},
"csvh" : {
"type" : "text",
"extensions" : [ "csvh" ],
"extractHeader" : true,
"delimiter" : ","
},
"image" : {
"type" : "image",
"extensions" : [ "jpg", "jpeg", "jpe", "tif", "tiff", "dng", "psd",
"png", "bmp", "gif", "ico", "pcx", "wav", "wave", "avi", "webp", "mov", "mp4",
"m4a", "m4p", "m4b", "m4r", "m4v", "3gp", "3g2", "eps", "epsf", "epsi", "ai",
"arw", "crw", "cr2", "nef", "orf", "raf", "rw2", "rwl", "srw", "x3f" ]
},
"maprdb" : {
"type" : "maprdb",
"readTimestampWithZoneOffset" : true
}
},
"enabled" : true
},
"disablePushdown" : false,
"cost" : {
"memoryCost" : 1.6777216E7,
"outputRowCount" : 1.0
}
}, {
"pop" : "project",
"@id" : 1,
"exprs" : [ {
"ref" : "`**`",
"expr" : "`**`"
} ],
"child" : 2,
"outputProj" : true,
"initialAllocation" : 100,
"maxAllocation" : 100,
"cost" : {
"memoryCost" : 1.6777216E7,
"outputRowCount" : 1.0
}
}, {
"pop" : "screen",
"@id" : 0,
"child" : 1,
"initialAllocation" : 100,
"maxAllocation" : 100,
"cost" : {
"memoryCost" : 1.6777216E7,
"outputRowCount" : 1.0
}
} ]
}
{noformat}
Drill fails with error:
{noformat}
2019-05-30 08:27:43,062 [23100990-f25e-23a5-34a0-b6ac3bec0841:foreman] ERROR
o.a.drill.exec.work.foreman.Foreman - SYSTEM ERROR: InvalidDefinitionException:
Cannot find a (Map) Key deserializer for type [simple type, class
org.apache.drill.exec.store.mapr.db.TabletFragmentInfo]
at [Source: (String)"{
"head" : {
"version" : 1,
"generator" : {
"type" : "ExplainHandler",
"info" : ""
},
"type" : "APACHE_DRILL_PHYSICAL",
"options" : [ {
"kind" : "BOOLEAN",
"accessibleScopes" : "ALL",
[jira] [Updated] (DRILL-7276) xss(bug) in apache drill Web UI latest verion 1.16.0 when authenticated
[ https://issues.apache.org/jira/browse/DRILL-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Volodymyr Vysotskyi updated DRILL-7276: --- Labels: ready-to-commit (was: ) > xss(bug) in apache drill Web UI latest verion 1.16.0 when authenticated > > > Key: DRILL-7276 > URL: https://issues.apache.org/jira/browse/DRILL-7276 > Project: Apache Drill > Issue Type: Bug > Components: Web Server >Affects Versions: 1.16.0 >Reporter: shuiboye >Assignee: Anton Gozhiy >Priority: Major > Labels: ready-to-commit > Fix For: 1.17.0 > > Attachments: 1.png, 2.png, 4.png > > > In the query page,I select the "SQL" of the "Query Type" and in the "Query" > field I input "*select '' FROM cp.`employee.json`*". > !1.png! > After submitting,I get the Query Profile whose url is > "*[http://127.0.0.1:8047/profiles/231beb11-4b43-0762-8b90-76a9af2edd24]*;. > !2.png! > Any user who visits the profile page and clicks "JSON profile" at the bottom > to see the FULL JSON Profile will see two alert boxes as shown below. > !4.png! > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-5983) Unsupported nullable converted type INT_8 for primitive type INT32 error
[ https://issues.apache.org/jira/browse/DRILL-5983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16844669#comment-16844669 ] Volodymyr Vysotskyi commented on DRILL-5983: [~vezir], [~davlee1...@yahoo.com], could you please share parquet files for which Drill fails, since I tried files from the DRILL-4764, and on Drill 1.16.0 everything works fine. > Unsupported nullable converted type INT_8 for primitive type INT32 error > > > Key: DRILL-5983 > URL: https://issues.apache.org/jira/browse/DRILL-5983 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Data Types >Affects Versions: 1.10.0, 1.11.0 > Environment: NAME="Ubuntu" > VERSION="16.04.2 LTS (Xenial Xerus)" >Reporter: Hakan Sarıbıyık >Priority: Major > Labels: parquet, read, types > > When I query a table with byte in it, then it gives an error; > _Query Failed: An Error Occurred > org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: > ExecutionSetupException: Unsupported nullable converted type INT_8 for > primitive type INT32 Fragment 1:6 [Error Id: > 46636b05-cff5-455b-ba25-527217346b3e on bigdata7:31010]_ > Actualy, it has been solved with > [DRILL-4764] - Parquet file with INT_16, etc. logical types not supported by > simple SELECT > according to https://drill.apache.org/docs/apache-drill-1-10-0-release-notes/ > But i tried it with even 1-11-0 it didnt worked. > I am querying parquet formatted file with pySpark > tablo1 > sourceid: byte (nullable = true) > select sourceid from tablo1 > works as expected with pySpark. But not with Drill v1.11.0 > Thanx. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-7192) Drill limits rows when autoLimit is disabled
[
https://issues.apache.org/jira/browse/DRILL-7192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16843804#comment-16843804
]
Volodymyr Vysotskyi commented on DRILL-7192:
[~kkhatua], thanks for looking into this issue, the last your proposal makes
sense for me.
> Drill limits rows when autoLimit is disabled
>
>
> Key: DRILL-7192
> URL: https://issues.apache.org/jira/browse/DRILL-7192
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.16.0
>Reporter: Volodymyr Vysotskyi
>Assignee: Kunal Khatua
>Priority: Major
> Fix For: 1.17.0
>
>
> In DRILL-7048 was implemented autoLimit for JDBC and rest clients.
> *Steps to reproduce the issue:*
> 1. Check that autoLimit was disabled, if not, disable it and restart Drill.
> 2. Submit any query, and verify that rows count is correct, for example,
> {code:sql}
> SELECT * FROM cp.`employee.json`;
> {code}
> returns 1,155 rows
> 3. Enable autoLimit for sqlLine sqlLine client:
> {code:sql}
> !set rowLimit 10
> {code}
> 4. Submit the same query and verify that the result has 10 rows.
> 5. Disable autoLimit:
> {code:sql}
> !set rowLimit 0
> {code}
> 6. Submit the same query, but for this time, *it returns 10 rows instead of
> 1,155*.
> Correct rows count is returned only after creating a new connection.
> The same issue is also observed for SQuirreL SQL client, but for example, for
> Postgres, it works correctly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-7050) RexNode convert exception in subquery
[
https://issues.apache.org/jira/browse/DRILL-7050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840597#comment-16840597
]
Volodymyr Vysotskyi commented on DRILL-7050:
[~le.louch], thanks for pointing to this case, but looks like it is another
problem connected with decorrelating sub-queries (RelDecorrelator) and should
be fixed in Calcite.
> RexNode convert exception in subquery
> -
>
> Key: DRILL-7050
> URL: https://issues.apache.org/jira/browse/DRILL-7050
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.14.0, 1.15.0
>Reporter: Oleg Zinoviev
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.17.0
>
>
> If the query contains a subquery whose filters are associated with the main
> query, an error occurs: *PLAN ERROR: Cannot convert RexNode to equivalent
> Drill expression. RexNode Class: org.apache.calcite.rex.RexCorrelVariable*
> Steps to reproduce:
> 1) Create source table (or view, doesn't matter)
> {code:sql}
> create table dfs.root.source as (
> select 1 as id union all select 2 as id
> )
> {code}
> 2) Execute query
> {code:sql}
> select t1.id,
> (select count(t2.id)
> from dfs.root.source t2 where t2.id = t1.id)
> from dfs.root.source t1
> {code}
> Reason:
> Method
> {code:java}org.apache.calcite.sql2rel.SqlToRelConverter.Blackboard.lookupExp{code}
> call {code:java}RexBuilder.makeCorrel{code} in some cases
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-7250) Query with CTE fails when its name matches to the table name without access
Volodymyr Vysotskyi created DRILL-7250:
--
Summary: Query with CTE fails when its name matches to the table
name without access
Key: DRILL-7250
URL: https://issues.apache.org/jira/browse/DRILL-7250
Project: Apache Drill
Issue Type: Bug
Affects Versions: 1.16.0
Reporter: Volodymyr Vysotskyi
Assignee: Volodymyr Vysotskyi
Fix For: 1.17.0
When impersonation is enabled, and for example, we have {{lineitem}} table with
permissions {{750}} which is owned by {{user0_1:group0_1}} and {{user2_1}}
don't have access to it.
The following query:
{code:sql}
use mini_dfs_plugin.user0_1;
with lineitem as (SELECT 1 as a) select * from lineitem
{code}
submitted from {{user2_1}} fails with the following error:
{noformat}
java.lang.Exception: org.apache.hadoop.security.AccessControlException:
Permission denied: user=user2_1, access=READ_EXECUTE,
inode="/user/user0_1/lineitem":user0_1:group0_1:drwxr-x---
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:317)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:229)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:199)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1752)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1736)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPathAccess(FSDirectory.java:1710)
at
org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getListingInt(FSDirStatAndListingOp.java:70)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getListing(FSNamesystem.java:4432)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getListing(NameNodeRpcServer.java:999)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getListing(ClientNamenodeProtocolServerSideTranslatorPB.java:646)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2217)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2213)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2213)
at ...(:0) ~[na:na]
at
org.apache.drill.exec.util.FileSystemUtil.listRecursive(FileSystemUtil.java:253)
~[classes/:na]
at
org.apache.drill.exec.util.FileSystemUtil.list(FileSystemUtil.java:208)
~[classes/:na]
at
org.apache.drill.exec.util.FileSystemUtil.listFiles(FileSystemUtil.java:104)
~[classes/:na]
at
org.apache.drill.exec.util.DrillFileSystemUtil.listFiles(DrillFileSystemUtil.java:86)
~[classes/:na]
at
org.apache.drill.exec.store.dfs.FileSelection.minusDirectories(FileSelection.java:178)
~[classes/:na]
at
org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory$WorkspaceSchema.detectEmptySelection(WorkspaceSchemaFactory.java:669)
~[classes/:na]
at
org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory$WorkspaceSchema.create(WorkspaceSchemaFactory.java:633)
~[classes/:na]
at
org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory$WorkspaceSchema.create(WorkspaceSchemaFactory.java:283)
~[classes/:na]
at
org.apache.drill.exec.planner.sql.ExpandingConcurrentMap.getNewEntry(ExpandingConcurrentMap.java:96)
~[classes/:na]
at
org.apache.drill.exec.planner.sql.ExpandingConcurrentMap.get(ExpandingConcurrentMap.java:90)
~[classes/:na]
at
org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory$WorkspaceSchema.getTable(WorkspaceSchemaFactory.java:439)
~[classes/:na]
at
org.apache.calcite.jdbc.SimpleCalciteSchema.getImplicitTable(SimpleCalciteSchema.java:83)
~[calcite-core-1.18.0-drill-r1.jar:1.18.0-drill-r1]
at
org.apache.calcite.jdbc.CalciteSchema.getTable(CalciteSchema.java:286)
~[calcite-core-1.18.0-drill-r1.jar:1.18.0-drill-r1]
at
org.apache.calcite.sql.validate.SqlValidatorUtil.getTableEntryFrom(SqlValidatorUtil.java:1046)
~[calcite-core-1.18.0-drill-r1.jar:1.18.0-drill-r1]
at
org.apache.calcite.sql.validate.SqlValidatorUtil.getTableEntry(SqlValidatorUtil.java:1003)
~[calcite-core-1.18.0-drill-r1.jar:1.18.0-drill-r1]
at
[jira] [Resolved] (DRILL-3995) Scalar replacement bug with Common Subexpression Elimination
[
https://issues.apache.org/jira/browse/DRILL-3995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi resolved DRILL-3995.
Resolution: Cannot Reproduce
Fix Version/s: 1.17.0
> Scalar replacement bug with Common Subexpression Elimination
>
>
> Key: DRILL-3995
> URL: https://issues.apache.org/jira/browse/DRILL-3995
> Project: Apache Drill
> Issue Type: Bug
>Reporter: Steven Phillips
>Priority: Major
> Fix For: 1.17.0
>
>
> The following query:
> {code}
> select t1.full_name from cp.`employee.json` t1, cp.`department.json` t2 where
> t1.department_id = t2.department_id and t1.position_id = t2.department_id
> {code}
> fails with the following:
> org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
> RuntimeException: Error at instruction 43: Expected an object reference, but
> found . setValue(II)V
> 0 R I I . . . . : :L0
> 1 R I I . . . . : : LINENUMBER 249 L0
> 2 R I I . . . . : : ICONST_0
> 3 R I I . . . . : I : ISTORE 3
> 4 R I I I . . . : : LCONST_0
> 5 R I I I . . . : J : LSTORE 4
> 6 R I I I J . . : :L1
> 7 R I I I J . . : : LINENUMBER 251 L1
> 8 R I I I J . . : : ALOAD 0
> 9 R I I I J . . : R : GETFIELD
> org/apache/drill/exec/test/generated/HashTableGen2$BatchHolder.vv20 :
> Lorg/apache/drill/exec/vector/NullableBigIntVector;
> 00010 R I I I J . . : R : INVOKEVIRTUAL
> org/apache/drill/exec/vector/NullableBigIntVector.getAccessor
> ()Lorg/apache/drill/exec/vector/NullableBigIntVector$Accessor;
> 00011 R I I I J . . : R : ILOAD 1
> 00012 R I I I J . . : R I : INVOKEVIRTUAL
> org/apache/drill/exec/vector/NullableBigIntVector$Accessor.isSet (I)I
> 00013 R I I I J . . : I : ISTORE 3
> 00014 R I I I J . . : :L2
> 00015 R I I I J . . : : LINENUMBER 252 L2
> 00016 R I I I J . . : : ILOAD 3
> 00017 R I I I J . . : I : ICONST_1
> 00018 R I I I J . . : I I : IF_ICMPNE L3
> 00019 R I I I J . . : :L4
> 00020 ? : LINENUMBER 253 L4
> 00021 ? : ALOAD 0
> 00022 ? : GETFIELD
> org/apache/drill/exec/test/generated/HashTableGen2$BatchHolder.vv20 :
> Lorg/apache/drill/exec/vector/NullableBigIntVector;
> 00023 ? : INVOKEVIRTUAL
> org/apache/drill/exec/vector/NullableBigIntVector.getAccessor
> ()Lorg/apache/drill/exec/vector/NullableBigIntVector$Accessor;
> 00024 ? : ILOAD 1
> 00025 ? : INVOKEVIRTUAL
> org/apache/drill/exec/vector/NullableBigIntVector$Accessor.get (I)J
> 00026 ? : LSTORE 4
> 00027 R I I I J . . : :L3
> 00028 R I I I J . . : : LINENUMBER 256 L3
> 00029 R I I I J . . : : ILOAD 3
> 00030 R I I I J . . : I : ICONST_0
> 00031 R I I I J . . : I I : IF_ICMPEQ L5
> 00032 R I I I J . . : :L6
> 00033 ? : LINENUMBER 257 L6
> 00034 ? : ALOAD 0
> 00035 ? : GETFIELD
> org/apache/drill/exec/test/generated/HashTableGen2$BatchHolder.vv24 :
> Lorg/apache/drill/exec/vector/NullableBigIntVector;
> 00036 ? : INVOKEVIRTUAL
> org/apache/drill/exec/vector/NullableBigIntVector.getMutator
> ()Lorg/apache/drill/exec/vector/NullableBigIntVector$Mutator;
> 00037 ? : ILOAD 2
> 00038 ? : ILOAD 3
> 00039 ? : LLOAD 4
> 00040 ? : INVOKEVIRTUAL
> org/apache/drill/exec/vector/NullableBigIntVector$Mutator.set (IIJ)V
> 00041 R I I I J . . : :L5
> 00042 R I I I J . . : : LINENUMBER 259 L5
> 00043 R I I I J . . : : ALOAD 6
> 00044 ? : GETFIELD
> org/apache/drill/exec/expr/holders/NullableBigIntHolder.isSet : I
> 00045 ? : ICONST_0
> 00046 ? : IF_ICMPEQ L7
> 00047 ? :L8
> 00048 ? : LINENUMBER 260 L8
> 00049 ? : ALOAD 0
> 00050 ? : GETFIELD
> org/apache/drill/exec/test/generated/HashTableGen2$BatchHolder.vv27 :
> Lorg/apache/drill/exec/vector/NullableBigIntVector;
> 00051 ? : INVOKEVIRTUAL
> org/apache/drill/exec/vector/NullableBigIntVector.getMutator
> ()Lorg/apache/drill/exec/vector/NullableBigIntVector$Mutator;
> 00052 ? : ILOAD 2
> 00053 ? : ALOAD 6
> 00054 ? : GETFIELD
> org/apache/drill/exec/expr/holders/NullableBigIntHolder.isSet : I
> 00055 ? : ALOAD 6
> 00056 ? : GETFIELD
> org/apache/drill/exec/expr/holders/NullableBigIntHolder.value : J
> 00057 ? : INVOKEVIRTUAL
> org/apache/drill/exec/vector/NullableBigIntVector$Mutator.set (IIJ)V
> 00058 ? :L7
> 00059 ?
[jira] [Resolved] (DRILL-4098) Assembly code in drillbit.log
[
https://issues.apache.org/jira/browse/DRILL-4098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi resolved DRILL-4098.
Resolution: Fixed
Fix Version/s: 1.16.0
Fixed in DRILL-2326
> Assembly code in drillbit.log
> -
>
> Key: DRILL-4098
> URL: https://issues.apache.org/jira/browse/DRILL-4098
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.3.0
>Reporter: Khurram Faraaz
>Priority: Major
> Fix For: 1.16.0
>
>
> We are seeing the below assembly code and the stack trace in drillbit.log
> after a Functional test run on a 4 node cluster on CentOS using MapR-Drill
> 1.3 RPM (latest 11/11).
> From drillbit.log
> {code}
> 2015-11-12 06:36:53,553 [29bbcc7a-36dd-dc7a-d77a-388b228896a4:frag:0:0] INFO
> o.a.d.e.w.f.FragmentStatusReporter -
> 29bbcc7a-36dd-dc7a-d77a-388b228896a4:0:0: State to report: RUNNING
> 2015-11-12 06:36:53,588 [29bbcc7a-36dd-dc7a-d77a-388b228896a4:frag:0:0] ERROR
> o.a.drill.exec.compile.MergeAdapter - Failure while merging classes.
> java.lang.RuntimeException: Error at instruction 26: Expected an object
> reference, but found . doEval(II)V
> 0 R I I . . . : :L0
> 1 R I I . . . : : LINENUMBER 104 L0
> 2 R I I . . . : : LCONST_0
> 3 R I I . . . : J : LSTORE 3
> 4 R I I J . . : :L1
> 5 R I I J . . : : LINENUMBER 106 L1
> 6 R I I J . . : : ALOAD 0
> 7 R I I J . . : R : GETFIELD
> org/apache/drill/exec/test/generated/ProjectorGen4245.vv0 :
> Lorg/apache/drill/exec/vector/BigIntVector;
> 8 R I I J . . : R : INVOKEVIRTUAL
> org/apache/drill/exec/vector/BigIntVector.getAccessor
> ()Lorg/apache/drill/exec/vector/BigIntVector$Accessor;
> 9 R I I J . . : R : ILOAD 1
> 00010 R I I J . . : R I : INVOKEVIRTUAL
> org/apache/drill/exec/vector/BigIntVector$Accessor.get (I)J
> 00011 R I I J . . : J : LSTORE 3
> 00012 R I I J . . : :L2
> 00013 R I I J . . : : LINENUMBER 108 L2
> 00014 R I I J . . : : ALOAD 0
> 00015 R I I J . . : R : GETFIELD
> org/apache/drill/exec/test/generated/ProjectorGen4245.vv4 :
> Lorg/apache/drill/exec/vector/BigIntVector;
> 00016 R I I J . . : R : INVOKEVIRTUAL
> org/apache/drill/exec/vector/BigIntVector.getMutator
> ()Lorg/apache/drill/exec/vector/BigIntVector$Mutator;
> 00017 R I I J . . : R : ILOAD 2
> 00018 R I I J . . : R I : LLOAD 3
> 00019 R I I J . . : R I J : INVOKEVIRTUAL
> org/apache/drill/exec/vector/BigIntVector$Mutator.set (IJ)V
> 00020 R I I J . . : :L3
> 00021 R I I J . . : : LINENUMBER 109 L3
> 00022 R I I J . . : : ALOAD 0
> 00023 R I I J . . : R : GETFIELD
> org/apache/drill/exec/test/generated/ProjectorGen4245.vv7 :
> Lorg/apache/drill/exec/vector/BigIntVector;
> 00024 R I I J . . : R : INVOKEVIRTUAL
> org/apache/drill/exec/vector/BigIntVector.getMutator
> ()Lorg/apache/drill/exec/vector/BigIntVector$Mutator;
> 00025 R I I J . . : R : ILOAD 2
> 00026 R I I J . . : R I : ALOAD 5
> 00027 ? : GETFIELD
> org/apache/drill/exec/expr/holders/BigIntHolder.value : J
> 00028 ? : INVOKEVIRTUAL
> org/apache/drill/exec/vector/BigIntVector$Mutator.set (IJ)V
> 00029 ? :L4
> 00030 ? : LINENUMBER 110 L4
> 00031 ? : ALOAD 0
> 00032 ? : GETFIELD
> org/apache/drill/exec/test/generated/ProjectorGen4245.vv10 :
> Lorg/apache/drill/exec/vector/BigIntVector;
> 00033 ? : INVOKEVIRTUAL
> org/apache/drill/exec/vector/BigIntVector.getMutator
> ()Lorg/apache/drill/exec/vector/BigIntVector$Mutator;
> 00034 ? : ILOAD 2
> 00035 ? : ALOAD 5
> 00036 ? : GETFIELD
> org/apache/drill/exec/expr/holders/BigIntHolder.value : J
> 00037 ? : INVOKEVIRTUAL
> org/apache/drill/exec/vector/BigIntVector$Mutator.set (IJ)V
> 00038 ? :L5
> 00039 ? : LINENUMBER 111 L5
> 00040 ? : ALOAD 0
> 00041 ? : GETFIELD
> org/apache/drill/exec/test/generated/ProjectorGen4245.vv13 :
> Lorg/apache/drill/exec/vector/BigIntVector;
> 00042 ? : INVOKEVIRTUAL
> org/apache/drill/exec/vector/BigIntVector.getMutator
> ()Lorg/apache/drill/exec/vector/BigIntVector$Mutator;
> 00043 ? : ILOAD 2
> 00044 ? : ALOAD 5
> 00045 ? : GETFIELD
> org/apache/drill/exec/expr/holders/BigIntHolder.value : J
> 00046 ? : INVOKEVIRTUAL
> org/apache/drill/exec/vector/BigIntVector$Mutator.set (IJ)V
> 00047 ? :L6
> 00048 ? : LINENUMBER 112 L6
> 00049 ? : ALOAD 0
> 00050 ? : GETFIELD
>
[jira] [Resolved] (DRILL-4299) Query that involves convert_from succeeds, exception is found in drillbit.log
[
https://issues.apache.org/jira/browse/DRILL-4299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi resolved DRILL-4299.
Resolution: Fixed
Fix Version/s: 1.16.0
Fixed in DRILL-2326
> Query that involves convert_from succeeds, exception is found in drillbit.log
> -
>
> Key: DRILL-4299
> URL: https://issues.apache.org/jira/browse/DRILL-4299
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Codegen
>Reporter: Victoria Markman
>Priority: Major
> Fix For: 1.16.0
>
> Attachments: data, drillbit.log
>
>
> Here is example of the exception, please refer to drillbit.log that is
> attached.
> {code}
> 2016-01-22 00:16:12,065 [295e8b32-e2e1-6966-2b13-389cd724ac73:foreman] INFO
> o.a.drill.exec.work.foreman.Foreman - Query text for query id
> 295e8b32-e2e1-6966-2b13-389cd724ac73: select c_row, c_float4,
> convert_from(convert_to(c_float4, 'FLOAT'), 'FLOAT') from data
> 2016-01-22 00:16:12,166 [295e8b32-e2e1-6966-2b13-389cd724ac73:foreman] INFO
> o.a.d.exec.store.parquet.Metadata - Took 0 ms to get file statuses
> 2016-01-22 00:16:12,173 [295e8b32-e2e1-6966-2b13-389cd724ac73:foreman] INFO
> o.a.d.exec.store.parquet.Metadata - Fetch parquet metadata: Executed 1 out of
> 1 using 1 threads. Time: 6ms total, 6.100645ms avg, 6ms max.
> 2016-01-22 00:16:12,173 [295e8b32-e2e1-6966-2b13-389cd724ac73:foreman] INFO
> o.a.d.exec.store.parquet.Metadata - Fetch parquet metadata: Executed 1 out of
> 1 using 1 threads. Earliest start: 1.338000 μs, Latest start: 1.338000 μs,
> Average start: 1.338000 μs .
> 2016-01-22 00:16:12,173 [295e8b32-e2e1-6966-2b13-389cd724ac73:foreman] INFO
> o.a.d.exec.store.parquet.Metadata - Took 6 ms to read file metadata
> 2016-01-22 00:16:12,229 [295e8b32-e2e1-6966-2b13-389cd724ac73:frag:0:0] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 295e8b32-e2e1-6966-2b13-389cd724ac73:0:0: State change requested
> AWAITING_ALLOCATION --> RUNNING
> 2016-01-22 00:16:12,229 [295e8b32-e2e1-6966-2b13-389cd724ac73:frag:0:0] INFO
> o.a.d.e.w.f.FragmentStatusReporter -
> 295e8b32-e2e1-6966-2b13-389cd724ac73:0:0: State to report: RUNNING
> 2016-01-22 00:16:12,300 [295e8b32-e2e1-6966-2b13-389cd724ac73:frag:0:0] ERROR
> o.a.drill.exec.compile.MergeAdapter - Failure while merging classes.
> java.lang.RuntimeException: Error at instruction 147: Expected an object
> reference, but found . doEval(II)V
> 0 R I I . . . . . . . . . . . . . . . . . . . . . . : :L0
> 1 R I I . . . . . . . . . . . . . . . . . . . . . . : : LINENUMBER
> 60 L0
> 2 R I I . . . . . . . . . . . . . . . . . . . . . . : : ICONST_0
> 3 R I I . . . . . . . . . . . . . . . . . . . . . . : I : ISTORE 3
> 4 R I I I . . . . . . . . . . . . . . . . . . . . . : : FCONST_0
> 5 R I I I . . . . . . . . . . . . . . . . . . . . . : F : FSTORE 4
> 6 R I I I F . . . . . . . . . . . . . . . . . . . . : :L1
> 7 R I I I F . . . . . . . . . . . . . . . . . . . . : : LINENUMBER
> 62 L1
> 8 R I I I F . . . . . . . . . . . . . . . . . . . . : : ALOAD 0
> 9 R I I I F . . . . . . . . . . . . . . . . . . . . : R : GETFIELD
> org/apache/drill/exec/test/generated/ProjectorGen11.vv1 :
> Lorg/apache/drill/exec/vector/NullableFloat4Vector;
> 00010 R I I I F . . . . . . . . . . . . . . . . . . . . : R :
> INVOKEVIRTUAL org/apache/drill/exec/vector/NullableFloat4Vector.getAccessor
> ()Lorg/apache/drill/exec/vector/NullableFloat4Vector$Accessor;
> 00011 R I I I F . . . . . . . . . . . . . . . . . . . . : R : ILOAD 1
> 00012 R I I I F . . . . . . . . . . . . . . . . . . . . : R I :
> INVOKEVIRTUAL
> org/apache/drill/exec/vector/NullableFloat4Vector$Accessor.isSet (I)I
> 00013 R I I I F . . . . . . . . . . . . . . . . . . . . : I : ISTORE 3
> 00014 R I I I F . . . . . . . . . . . . . . . . . . . . : :L2
> 00015 R I I I F . . . . . . . . . . . . . . . . . . . . : : LINENUMBER
> 63 L2
> 00016 R I I I F . . . . . . . . . . . . . . . . . . . . : : ILOAD 3
> 00017 R I I I F . . . . . . . . . . . . . . . . . . . . : I : ICONST_1
> 00018 R I I I F . . . . . . . . . . . . . . . . . . . . : I I :
> IF_ICMPNE L3
> 00019 R I I I F . . . . . . . . . . . . . . . . . . . . : :L4
> 00020 ? : LINENUMBER 64 L4
> 00021 ? : ALOAD 0
> 00022 ? : GETFIELD
> org/apache/drill/exec/test/generated/ProjectorGen11.vv1 :
> Lorg/apache/drill/exec/vector/NullableFloat4Vector;
> 00023 ? : INVOKEVIRTUAL
> org/apache/drill/exec/vector/NullableFloat4Vector.getAccessor
>
[jira] [Commented] (DRILL-7091) Query with EXISTS and correlated subquery fails with NPE in HashJoinMemoryCalculatorImpl$BuildSidePartitioningImpl
[
https://issues.apache.org/jira/browse/DRILL-7091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16837466#comment-16837466
]
Volodymyr Vysotskyi commented on DRILL-7091:
{{HashJoinMemoryCalculatorImpl.BuildSidePartitioningImpl}} should handle such
cases with star columns in Project, since data sources which do not support
pushing the project into scan may be queried, and we will receive the same NPE.
> Query with EXISTS and correlated subquery fails with NPE in
> HashJoinMemoryCalculatorImpl$BuildSidePartitioningImpl
> --
>
> Key: DRILL-7091
> URL: https://issues.apache.org/jira/browse/DRILL-7091
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.15.0
>Reporter: Volodymyr Vysotskyi
>Assignee: Boaz Ben-Zvi
>Priority: Major
>
> Steps to reproduce:
> 1. Create view:
> {code:sql}
> create view dfs.tmp.nation_view as select * from cp.`tpch/nation.parquet`;
> {code}
> Run the following query:
> {code:sql}
> SELECT n_nationkey, n_name
> FROM dfs.tmp.nation_view a
> WHERE EXISTS (SELECT 1
> FROM cp.`tpch/region.parquet` b
> WHERE b.r_regionkey = a.n_regionkey)
> {code}
> This query fails with NPE:
> {noformat}
> [Error Id: 9a592635-f792-4403-965c-bd2eece7e8fc on cv1:31010]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
> ~[drill-common-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:364)
> [drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:219)
> [drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:330)
> [drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
> [drill-common-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [na:1.8.0_161]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [na:1.8.0_161]
> at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]
> Caused by: java.lang.NullPointerException: null
> at
> org.apache.drill.exec.physical.impl.join.HashJoinMemoryCalculatorImpl$BuildSidePartitioningImpl.initialize(HashJoinMemoryCalculatorImpl.java:267)
> ~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.join.HashJoinBatch.executeBuildPhase(HashJoinBatch.java:959)
> ~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.join.HashJoinBatch.innerNext(HashJoinBatch.java:525)
> ~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:186)
> ~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:126)
> ~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:116)
> ~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:63)
> ~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:141)
> ~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:186)
> ~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:126)
> ~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at
> org.apache.drill.exec.test.generated.HashAggregatorGen2.doWork(HashAggTemplate.java:642)
> ~[na:na]
> at
> org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.innerNext(HashAggBatch.java:295)
> ~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:186)
> ~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:126)
> ~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:116)
>
[jira] [Updated] (DRILL-7237) IllegalStateException in aggregation function 'single_value' when there is a varchar datatype in the subquery results
[
https://issues.apache.org/jira/browse/DRILL-7237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi updated DRILL-7237:
---
Description:
*Description:*
The following issue can be reproduced on the fix for the
[DRILL-7050|https://issues.apache.org/jira/browse/DRILL-7050].
_For the query with > 1 row in subquery results where the data type of these
results *is not varchar*:_
{noformat}
SELECT
e.full_name,
(
SELECT
ine.employee_id
FROM
cp.`employee.json` ine
WHERE
ine.position_id = e.position_id
) as empl_id
FROM
cp.`employee.json` e
LIMIT 20
{noformat}
_We have the following correct and informative error:_
{noformat}
Query Failed: An Error Occurred
org.apache.drill.common.exceptions.UserRemoteException: FUNCTION ERROR: Input
for single_value function has more than one row Fragment 0:0 [Error Id:
b770098f-b1c7-4647-9f41-9e986a0e47b7 on maprhost:31010]
{noformat}
_But when in the result set of the subquery we have *a varchar data type*:_
{noformat}
SELECT
e.full_name,
(
SELECT
ine.first_name
FROM
cp.`employee.json` ine
WHERE
ine.position_id = e.position_id
) as empl_id
FROM
cp.`employee.json` e
LIMIT 20
{noformat}
*Actual result:*
_Drill throws the following error:_
{noformat}
org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
IllegalStateException: Workspace variable 'value' in aggregation function
'single_value' is not allowed to have variable length type. Fragment 0:0
Please, refer to logs for more information. [Error Id:
32325ba9-d2b3-4216-acf6-8e80dfe4a56a on maprhost:31010]
{noformat}
Log file is in the attachment "drillbit.log"
*Expected result:*
Drill should return the same informative error to any of a data types in the
subquery result set.
was:
*Description:*
The following issue can be reproduced on the EBF for the
[DRILL-7050|https://issues.apache.org/jira/browse/DRILL-7050].
_For the query with > 1 row in subquery results where the data type of these
results *is not varchar*:_
{noformat}
SELECT
e.full_name,
(
SELECT
ine.employee_id
FROM
cp.`employee.json` ine
WHERE
ine.position_id = e.position_id
) as empl_id
FROM
cp.`employee.json` e
LIMIT 20
{noformat}
_We have the following correct and informative error:_
{noformat}
Query Failed: An Error Occurred
org.apache.drill.common.exceptions.UserRemoteException: FUNCTION ERROR: Input
for single_value function has more than one row Fragment 0:0 [Error Id:
b770098f-b1c7-4647-9f41-9e986a0e47b7 on maprhost:31010]
{noformat}
_But when in the result set of the subquery we have *a varchar data type*:_
{noformat}
SELECT
e.full_name,
(
SELECT
ine.first_name
FROM
cp.`employee.json` ine
WHERE
ine.position_id = e.position_id
) as empl_id
FROM
cp.`employee.json` e
LIMIT 20
{noformat}
*Actual result:*
_Drill throws the following error:_
{noformat}
org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
IllegalStateException: Workspace variable 'value' in aggregation function
'single_value' is not allowed to have variable length type. Fragment 0:0
Please, refer to logs for more information. [Error Id:
32325ba9-d2b3-4216-acf6-8e80dfe4a56a on maprhost:31010]
{noformat}
Log file is in the attachment "drillbit.log"
*Expected result:*
Drill should return the same informative error to any of a data types in the
subquery result set.
> IllegalStateException in aggregation function 'single_value' when there is a
> varchar datatype in the subquery results
> -
>
> Key: DRILL-7237
> URL: https://issues.apache.org/jira/browse/DRILL-7237
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.14.0, 1.15.0
>Reporter: Denys Ordynskiy
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.17.0
>
> Attachments: drillbit.log
>
>
> *Description:*
> The following issue can be reproduced on the fix for the
> [DRILL-7050|https://issues.apache.org/jira/browse/DRILL-7050].
> _For the query with > 1 row in subquery results where the data type of these
> results *is not varchar*:_
> {noformat}
> SELECT
> e.full_name,
> (
> SELECT
> ine.employee_id
> FROM
> cp.`employee.json` ine
> WHERE
> ine.position_id = e.position_id
> ) as empl_id
> FROM
> cp.`employee.json` e
> LIMIT 20
> {noformat}
> _We have the following correct and informative error:_
> {noformat}
> Query Failed: An Error Occurred
> org.apache.drill.common.exceptions.UserRemoteException: FUNCTION ERROR: Input
> for single_value function has more than one row Fragment 0:0 [Error Id:
> b770098f-b1c7-4647-9f41-9e986a0e47b7 on maprhost:31010]
> {noformat}
>
[jira] [Created] (DRILL-7241) Hash aggregate does not work with interval types
Volodymyr Vysotskyi created DRILL-7241:
--
Summary: Hash aggregate does not work with interval types
Key: DRILL-7241
URL: https://issues.apache.org/jira/browse/DRILL-7241
Project: Apache Drill
Issue Type: Bug
Affects Versions: 1.16.0
Reporter: Volodymyr Vysotskyi
Queries with hash aggregation for interval data types fail with Code generation
error.
*Steps to reproduce:*
disable stream aggregate to force hash aggregate:
{code:sql}
set `planner.enable_streamagg`=false;
{code}
Submit query with aggregation for interval type:
{code:sql}
select max(age) as max_age from (select AGE('1957-06-13') as age) group by age;
{code}
It fails with the error:
{noformat}
Error: UNSUPPORTED_OPERATION ERROR: Code generation error - likely an error in
the code.
Fragment 0:0
[Error Id: fdd9ee7d-9991-42e2-ad2b-eb36503051f8 on user515050-pc:31019]
(state=,code=0)
{noformat}
Stack trace from logs:
{noformat}
2019-05-06 16:42:06,643 [Client-1] INFO o.a.d.j.i.DrillCursor$ResultsListener
- [#10] Query failed:
org.apache.drill.common.exceptions.UserRemoteException: UNSUPPORTED_OPERATION
ERROR: Code generation error - likely an error in the code.
Fragment 0:0
[Error Id: 65d849df-408c-4851-88be-b13d99c7b6e6 on user515050-pc:31019]
at
org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:123)
[drill-java-exec-1.17.0-SNAPSHOT.jar:1.17.0-SNAPSHOT]
at
org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:422)
[drill-java-exec-1.17.0-SNAPSHOT.jar:1.17.0-SNAPSHOT]
at org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:96)
[drill-java-exec-1.17.0-SNAPSHOT.jar:1.17.0-SNAPSHOT]
at
org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:273)
[drill-rpc-1.17.0-SNAPSHOT.jar:1.17.0-SNAPSHOT]
at
org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:243)
[drill-rpc-1.17.0-SNAPSHOT.jar:1.17.0-SNAPSHOT]
at
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
[netty-codec-4.0.48.Final.jar:4.0.48.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
[netty-transport-4.0.48.Final.jar:4.0.48.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
[netty-transport-4.0.48.Final.jar:4.0.48.Final]
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
[netty-transport-4.0.48.Final.jar:4.0.48.Final]
at
io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
[netty-handler-4.0.48.Final.jar:4.0.48.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
[netty-transport-4.0.48.Final.jar:4.0.48.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
[netty-transport-4.0.48.Final.jar:4.0.48.Final]
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
[netty-transport-4.0.48.Final.jar:4.0.48.Final]
at
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
[netty-codec-4.0.48.Final.jar:4.0.48.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
[netty-transport-4.0.48.Final.jar:4.0.48.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
[netty-transport-4.0.48.Final.jar:4.0.48.Final]
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
[netty-transport-4.0.48.Final.jar:4.0.48.Final]
at
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312)
[netty-codec-4.0.48.Final.jar:4.0.48.Final]
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286)
[netty-codec-4.0.48.Final.jar:4.0.48.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
[netty-transport-4.0.48.Final.jar:4.0.48.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
[netty-transport-4.0.48.Final.jar:4.0.48.Final]
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
[netty-transport-4.0.48.Final.jar:4.0.48.Final]
at
io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
[netty-transport-4.0.48.Final.jar:4.0.48.Final]
at
[jira] [Updated] (DRILL-6965) Adjust table function usage for all storage plugins and implement schema parameter
[
https://issues.apache.org/jira/browse/DRILL-6965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi updated DRILL-6965:
---
Labels: doc-impacting ready-to-commit (was: doc-impacting)
> Adjust table function usage for all storage plugins and implement schema
> parameter
> --
>
> Key: DRILL-6965
> URL: https://issues.apache.org/jira/browse/DRILL-6965
> Project: Apache Drill
> Issue Type: Sub-task
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>Priority: Major
> Labels: doc-impacting, ready-to-commit
> Fix For: 1.17.0
>
>
> Schema can be used while reading the table into two ways:
> a. schema is created in the table root folder using CREATE SCHEMA command
> and schema usage command is enabled;
> b. schema indicated in table function.
> This Jira implements point b.
> Schema indication using table function is useful when user does not want to
> persist schema in table root location or when reading from file, not folder.
> Schema parameter can be used as individual unit or in together with for
> format plugin table properties.
> Usage examples:
> Pre-requisites:
> V3 reader must be enabled: {{set `exec.storage.enable_v3_text_reader` =
> true;}}
> Query examples:
> 1. There is folder with files or just one file (ex: dfs.tmp.text_table) and
> user wants to apply schema to them:
> a. indicate schema inline:
> {noformat}
> select * from table(dfs.tmp.`text_table`(
> schema => 'inline=(col1 date properties {`drill.format` = `-MM-dd`})
> properties {`drill.strict` = `false`}'))
> {noformat}
> To indicate only table properties use the following syntax:
> {noformat}
> select * from table(dfs.tmp.`text_table`(
> schema => 'inline=()
> properties {`drill.strict` = `false`}'))
> {noformat}
> b. indicate schema using path:
> First schema was created in some location using CREATE SCHEMA command. For
> example:
> {noformat}
> create schema
> (col int)
> path '/tmp/my_schema'
> {noformat}
> Now user wants to apply this schema in table function:
> {noformat}
> select * from table(dfs.tmp.`text_table`(schema => 'path=`/tmp/my_schema`'))
> {noformat}
> 2. User wants to apply schema along side with format plugin table function
> parameters.
> Assuming that user has CSV file with headers with extension that does not
> comply to default text file with headers extension (ex: cars.csvh-test):
> {noformat}
> select * from table(dfs.tmp.`cars.csvh-test`(type => 'text',
> fieldDelimiter => ',', extractHeader => true,
> schema => 'inline=(col1 date)'))
> {noformat}
> More details about syntax can be found in design document:
>
> [https://docs.google.com/document/d/1mp4egSbNs8jFYRbPVbm_l0Y5GjH3HnoqCmOpMTR_g4w/edit]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-7098) File Metadata Metastore Plugin
[ https://issues.apache.org/jira/browse/DRILL-7098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Volodymyr Vysotskyi updated DRILL-7098: --- Reviewer: Volodymyr Vysotskyi (was: Aman Sinha) > File Metadata Metastore Plugin > -- > > Key: DRILL-7098 > URL: https://issues.apache.org/jira/browse/DRILL-7098 > Project: Apache Drill > Issue Type: Sub-task > Components: Server, Metadata >Reporter: Vitalii Diravka >Assignee: Vitalii Diravka >Priority: Major > Labels: Metastore, ready-to-commit > Fix For: 2.0.0 > > > DRILL-6852 introduces Drill Metastore API. > The second step is to create internal Drill Metastore mechanism (and File > Metastore Plugin), which will involve Metastore API and can be extended for > using by other Storage Plugins. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-7098) File Metadata Metastore Plugin
[ https://issues.apache.org/jira/browse/DRILL-7098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Volodymyr Vysotskyi updated DRILL-7098: --- Labels: Metastore ready-to-commit (was: Metastore) > File Metadata Metastore Plugin > -- > > Key: DRILL-7098 > URL: https://issues.apache.org/jira/browse/DRILL-7098 > Project: Apache Drill > Issue Type: Sub-task > Components: Server, Metadata >Reporter: Vitalii Diravka >Assignee: Vitalii Diravka >Priority: Major > Labels: Metastore, ready-to-commit > Fix For: 2.0.0 > > > DRILL-6852 introduces Drill Metastore API. > The second step is to create internal Drill Metastore mechanism (and File > Metastore Plugin), which will involve Metastore API and can be extended for > using by other Storage Plugins. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-7226) Compilation error on Windows when building from the release tarball sources
[
https://issues.apache.org/jira/browse/DRILL-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16830957#comment-16830957
]
Volodymyr Vysotskyi commented on DRILL-7226:
One more difference is that my output contains the following line:
{noformat}
[INFO] Compiling 119 source files to
D:\Work\apache-drill-1.16.0-src\protocol\target\classes
{noformat}
but your is the following:
{noformat}
[INFO] Compiling 73 source files to
D:\src\rc2\apache-drill-1.16.0-src\protocol\target\classes
{noformat}
Could you please confirm that you have unpacked the archive correctly, and all
files are present, for example, this class is present:
{noformat}
apache-drill-1.16.0-src/protocol/src/main/java/org/apache/drill/exec/proto/beans/DrillbitEndpoint.java
{noformat}
> Compilation error on Windows when building from the release tarball sources
> ---
>
> Key: DRILL-7226
> URL: https://issues.apache.org/jira/browse/DRILL-7226
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.15.0
>Reporter: Denys Ordynskiy
>Assignee: Kunal Khatua
>Priority: Major
> Fix For: 1.17.0
>
> Attachments: tarball_building.log
>
>
> *Description:*
> OS - Windows.
> Downloaded tarball with sources for the
> [1.15|http://home.apache.org/~vitalii/drill/releases/1.15.0/rc2/apache-drill-1.15.0-src.tar.gz]
> or
> [1.16|http://home.apache.org/~sorabh/drill/releases/1.16.0/rc2/apache-drill-1.16.0-src.tar.gz]
> Drill release.
> Extracted the sources.
> Built sources using the following command:
> {noformat}
> mvn clean install -DskipTests -Pmapr
> {noformat}
> *Expected result:*
> BUILD SUCCESS
> *Actual result:*
> {noformat}
> ...
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR]
> D:\src\rc2\apache-drill-1.16.0-src\protocol\src\main\java\org\apache\drill\exec\proto\beans\RecordBatchDef.java:[53,17]
> error: cannot find symbol
> symbol: class SerializedField
> location: class RecordBatchDef
> ...
> BUILD FAILURE
> {noformat}
> See "tarball_building.log"
> There are no errors when building sources on Windows from the GitHub release
> [branch|https://github.com/sohami/drill/commits/drill-1.16.0].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-7226) Compilation error on Windows when building from the release tarball sources
[
https://issues.apache.org/jira/browse/DRILL-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16830952#comment-16830952
]
Volodymyr Vysotskyi commented on DRILL-7226:
[~denysord88], could you please share more info about your system, for example,
JDK and maven versions? Also, it would be good to see the output with enabled
debug.
Also, could you please confirm that you have used clean, just unpacked version,
since I didn't observe the following line in the output:
{noformat}
Deleting D:\src\rc2\apache-drill-1.16.0-src\protocol\target
{noformat}
And it should be mentioned that all further tries also were successful for me.
> Compilation error on Windows when building from the release tarball sources
> ---
>
> Key: DRILL-7226
> URL: https://issues.apache.org/jira/browse/DRILL-7226
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.15.0
>Reporter: Denys Ordynskiy
>Assignee: Kunal Khatua
>Priority: Major
> Fix For: 1.17.0
>
> Attachments: tarball_building.log
>
>
> *Description:*
> OS - Windows.
> Downloaded tarball with sources for the
> [1.15|http://home.apache.org/~vitalii/drill/releases/1.15.0/rc2/apache-drill-1.15.0-src.tar.gz]
> or
> [1.16|http://home.apache.org/~sorabh/drill/releases/1.16.0/rc2/apache-drill-1.16.0-src.tar.gz]
> Drill release.
> Extracted the sources.
> Built sources using the following command:
> {noformat}
> mvn clean install -DskipTests -Pmapr
> {noformat}
> *Expected result:*
> BUILD SUCCESS
> *Actual result:*
> {noformat}
> ...
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR]
> D:\src\rc2\apache-drill-1.16.0-src\protocol\src\main\java\org\apache\drill\exec\proto\beans\RecordBatchDef.java:[53,17]
> error: cannot find symbol
> symbol: class SerializedField
> location: class RecordBatchDef
> ...
> BUILD FAILURE
> {noformat}
> See "tarball_building.log"
> There are no errors when building sources on Windows from the GitHub release
> [branch|https://github.com/sohami/drill/commits/drill-1.16.0].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
