[jira] [Commented] (DRILL-6429) SortImpl Should Not Use BufferAllocator.setLenient()
[ https://issues.apache.org/jira/browse/DRILL-6429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16481446#comment-16481446 ] Paul Rogers commented on DRILL-6429: [~timothyfarkas], good plan. BTW, forgot to mention, leniency is only turned on in production, it is off by default in testing (so we can catch errors.) (Assertion state is used as the trigger.) Just go ahead and add a config option or something to force leniency off in your tests. One suggestion: memory planning must be done up front as reactive planning will fail when dealing with queries with a sort: sorts start with little memory use, but grow to maximum use just before spilling, then drop their use. Better to budget for the full amount allocated to each operator. Drill makes a very strong assumption: all nodes are identical. If we use up-front planning, we already know the amount of memory available on each node. Choosing a memory-per fragment, and allocating memory per fragment (uniformly across nodes) avoids the need to track per-node memory fluctuations. Then, we know that, statistically, not all fragments will use their full memory allotment at all times. This provides some wiggle-room. To achieve your goal, just do testing with leniency turned off. Since tests do pass, we know that, at least for our test cases, the memory management in sort does work. It will be much better once the work is done to control batch sizes elsewhere. You can decide later if you still need the margin of safety provided by leniency. Would be great if not. Looking forward to the result of all this great work! > SortImpl Should Not Use BufferAllocator.setLenient() > > > Key: DRILL-6429 > URL: https://issues.apache.org/jira/browse/DRILL-6429 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > > The Sort operator should always obey its memory limit. This is required for > resource management. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-5846) Improve Parquet Reader Performance for Flat Data types
[ https://issues.apache.org/jira/browse/DRILL-5846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16481405#comment-16481405 ] ASF GitHub Bot commented on DRILL-5846: --- sachouche commented on a change in pull request #1060: DRILL-5846: Improve parquet performance for Flat Data Types URL: https://github.com/apache/drill/pull/1060#discussion_r189421067 ## File path: exec/memory/pom.xml ## @@ -34,4 +34,5 @@ base + Review comment: Removed the new empty line (introduced by accident). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Improve Parquet Reader Performance for Flat Data types > --- > > Key: DRILL-5846 > URL: https://issues.apache.org/jira/browse/DRILL-5846 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Parquet >Affects Versions: 1.11.0 >Reporter: salim achouche >Assignee: salim achouche >Priority: Major > Labels: performance > Fix For: 1.14.0 > > Attachments: 2542d447-9837-3924-dd12-f759108461e5.sys.drill, > 2542d49b-88ef-38e3-a02b-b441c1295817.sys.drill > > > The Parquet Reader is a key use-case for Drill. This JIRA is an attempt to > further improve the Parquet Reader performance as several users reported that > Parquet parsing represents the lion share of the overall query execution. It > tracks Flat Data types only as Nested DTs might involve functional and > processing enhancements (e.g., a nested column can be seen as a Document; > user might want to perform operations scoped at the document level that is no > need to span all rows). Another JIRA will be created to handle the nested > columns use-case. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6429) SortImpl Should Not Use BufferAllocator.setLenient()
[ https://issues.apache.org/jira/browse/DRILL-6429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16481353#comment-16481353 ] Timothy Farkas commented on DRILL-6429: --- [~paul-rogers] We want to start implementing true resource management for memory. This would entail a resource monitor that keeps track of all available memory on each node, and planning queries to fit within available memory. In order to do this, we need to be sure that when we tell an operator to run within X amount of memory it will do so. If we tell an operator to run within X and it decides to use more memory, then our resource monitor would be inaccurate and we can get queries that unexpectedly OOM. It is currently possibly to do this reliable with heuristics. I have done it for HashJoin, which withstood a battery of tests without using setLenient(). See https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/HashJoinMemoryCalculatorImpl.java . Furthermore as the batch sizing project you and Padma have been working on progresses, we will eventually be able to move away from heuristics and deal with certainties. The goal here is to look at SortImpl and see if it can be augmented to not require setLenient, so our resource management system can make the assumption that all memory limits are obeyed and are absolute. If we can make sure that assumption is true, the difficulty of implementing a reliable resource management system drastically reduces. I suspect it is possible to do since it has been done for HashJoin, and I suspect it would not involve any major changes to Sort. > SortImpl Should Not Use BufferAllocator.setLenient() > > > Key: DRILL-6429 > URL: https://issues.apache.org/jira/browse/DRILL-6429 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > > The Sort operator should always obey its memory limit. This is required for > resource management. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6429) SortImpl Should Not Use BufferAllocator.setLenient()
[ https://issues.apache.org/jira/browse/DRILL-6429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16481337#comment-16481337 ] Paul Rogers commented on DRILL-6429: Bit of the back-story here. Originally, there was no `setLenient()`. We added it just for sort. Drill's allocator model enforces strict per-operator limits. Until a year ago, operators made no real attempt to honor the limit; so the limit was the only tool available to prevent run-away operator usage. Of course, the limit was of magical value because, for most operators, it was set at 10 GB... For sort, we did our best to modify the code to honor those limits for spilling, measuring input batch size and so on. As a result, the operator itself enforces its own limit. Now, it is nearly impossible to get everything right since the Sort has no control over its input batch size. The result is that when the moon is full and the starts are aligned just right, the Sort will exceed the hard-coded limit by some few percent. Should we punish the user by killing their query? We decided not to. So, we introduced the lenient mode. The thought is, since most operators made no attempt at all to limit memory, it makes no sense to punish the user because Sort happened to be off by a few percent. Given all this, what is the reason to change our thinking now? Can the sort do a better job of predicting memory usage? Does sort now have control over its input batch size? > SortImpl Should Not Use BufferAllocator.setLenient() > > > Key: DRILL-6429 > URL: https://issues.apache.org/jira/browse/DRILL-6429 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > > The Sort operator should always obey its memory limit. This is required for > resource management. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-5671) Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
[ https://issues.apache.org/jira/browse/DRILL-5671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pritesh Maker updated DRILL-5671: - Reviewer: (was: Aditya Kishore) > Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster > > > Key: DRILL-5671 > URL: https://issues.apache.org/jira/browse/DRILL-5671 > Project: Apache Drill > Issue Type: New Feature > Components: Server >Reporter: Karthikeyan Manivannan >Assignee: Karthikeyan Manivannan >Priority: Major > > All Drill ZK nodes, currently, are assigned a default [world:all] ACL i.e. > anyone gets to do CDRWA(create, delete, read, write, admin access). This > means that even on a secure cluster anyone can perform all CRDWA actions on > the znodes. > This should be changed such that: > - In a non-secure cluster, Drill will continue using the current default > [world:all] ACL > - In a secure cluster, all nodes should have an [authid: all] ACL i.e. the > authenticated user that created the znode gets full access. The discovery > znodes i.e. the znodes with the list of Drillbits will have an additional > [world:read] ACL, i.e. the list of Drillbits will be readable by anyone. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-5846) Improve Parquet Reader Performance for Flat Data types
[
https://issues.apache.org/jira/browse/DRILL-5846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16481301#comment-16481301
]
ASF GitHub Bot commented on DRILL-5846:
---
vrozov commented on a change in pull request #1060: DRILL-5846: Improve parquet
performance for Flat Data Types
URL: https://github.com/apache/drill/pull/1060#discussion_r189409447
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java
##
@@ -289,6 +289,10 @@ private ExecConstants() {
public static final OptionValidator COMPILE_SCALAR_REPLACEMENT = new
BooleanValidator("exec.compile.scalar_replacement");
+ // Controls whether to enable bulk parquet reader processing
+ public static final String PARQUET_FLAT_READER_BULK=
"store.parquet.flat.reader.bulk";
Review comment:
While it marginally improves code readability, maintaining the code style
that aligns on the longest line is not sustainable as it requires modifications
to existing code when a new line is introduced or otherwise code becomes
unreadable. My vote is to avoid newly introduced code formatting as well and
keep existing Drill style.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Improve Parquet Reader Performance for Flat Data types
> ---
>
> Key: DRILL-5846
> URL: https://issues.apache.org/jira/browse/DRILL-5846
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - Parquet
>Affects Versions: 1.11.0
>Reporter: salim achouche
>Assignee: salim achouche
>Priority: Major
> Labels: performance
> Fix For: 1.14.0
>
> Attachments: 2542d447-9837-3924-dd12-f759108461e5.sys.drill,
> 2542d49b-88ef-38e3-a02b-b441c1295817.sys.drill
>
>
> The Parquet Reader is a key use-case for Drill. This JIRA is an attempt to
> further improve the Parquet Reader performance as several users reported that
> Parquet parsing represents the lion share of the overall query execution. It
> tracks Flat Data types only as Nested DTs might involve functional and
> processing enhancements (e.g., a nested column can be seen as a Document;
> user might want to perform operations scoped at the document level that is no
> need to span all rows). Another JIRA will be created to handle the nested
> columns use-case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-5846) Improve Parquet Reader Performance for Flat Data types
[ https://issues.apache.org/jira/browse/DRILL-5846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16481291#comment-16481291 ] ASF GitHub Bot commented on DRILL-5846: --- vrozov commented on a change in pull request #1060: DRILL-5846: Improve parquet performance for Flat Data Types URL: https://github.com/apache/drill/pull/1060#discussion_r189407919 ## File path: exec/memory/pom.xml ## @@ -34,4 +34,5 @@ base + Review comment: Avoid formatting changes This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Improve Parquet Reader Performance for Flat Data types > --- > > Key: DRILL-5846 > URL: https://issues.apache.org/jira/browse/DRILL-5846 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Parquet >Affects Versions: 1.11.0 >Reporter: salim achouche >Assignee: salim achouche >Priority: Major > Labels: performance > Fix For: 1.14.0 > > Attachments: 2542d447-9837-3924-dd12-f759108461e5.sys.drill, > 2542d49b-88ef-38e3-a02b-b441c1295817.sys.drill > > > The Parquet Reader is a key use-case for Drill. This JIRA is an attempt to > further improve the Parquet Reader performance as several users reported that > Parquet parsing represents the lion share of the overall query execution. It > tracks Flat Data types only as Nested DTs might involve functional and > processing enhancements (e.g., a nested column can be seen as a Document; > user might want to perform operations scoped at the document level that is no > need to span all rows). Another JIRA will be created to handle the nested > columns use-case. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-5846) Improve Parquet Reader Performance for Flat Data types
[ https://issues.apache.org/jira/browse/DRILL-5846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16481288#comment-16481288 ] ASF GitHub Bot commented on DRILL-5846: --- vrozov commented on a change in pull request #1060: DRILL-5846: Improve parquet performance for Flat Data Types URL: https://github.com/apache/drill/pull/1060#discussion_r189407797 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/VarLenAbstractEntryReader.java ## @@ -0,0 +1,266 @@ +/*** Review comment: It should be disabled by default, IMO. There is no need to check license headers all the time. Please either run license check explicitly or rely on Travis. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Improve Parquet Reader Performance for Flat Data types > --- > > Key: DRILL-5846 > URL: https://issues.apache.org/jira/browse/DRILL-5846 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Parquet >Affects Versions: 1.11.0 >Reporter: salim achouche >Assignee: salim achouche >Priority: Major > Labels: performance > Fix For: 1.14.0 > > Attachments: 2542d447-9837-3924-dd12-f759108461e5.sys.drill, > 2542d49b-88ef-38e3-a02b-b441c1295817.sys.drill > > > The Parquet Reader is a key use-case for Drill. This JIRA is an attempt to > further improve the Parquet Reader performance as several users reported that > Parquet parsing represents the lion share of the overall query execution. It > tracks Flat Data types only as Nested DTs might involve functional and > processing enhancements (e.g., a nested column can be seen as a Document; > user might want to perform operations scoped at the document level that is no > need to span all rows). Another JIRA will be created to handle the nested > columns use-case. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6430) Drill Should Not Fail If It Sees Deprecated Options Stored In Zookeeper Or Locally
Timothy Farkas created DRILL-6430: - Summary: Drill Should Not Fail If It Sees Deprecated Options Stored In Zookeeper Or Locally Key: DRILL-6430 URL: https://issues.apache.org/jira/browse/DRILL-6430 Project: Apache Drill Issue Type: Improvement Reporter: Timothy Farkas Assignee: Timothy Farkas This is required for resource management since we will likely remove many options. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6429) SortImpl Should Not Use BufferAllocator.setLenient()
[ https://issues.apache.org/jira/browse/DRILL-6429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timothy Farkas updated DRILL-6429: -- Description: The Sort operator should always obey its memory limit. This is required for resource management. > SortImpl Should Not Use BufferAllocator.setLenient() > > > Key: DRILL-6429 > URL: https://issues.apache.org/jira/browse/DRILL-6429 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > > The Sort operator should always obey its memory limit. This is required for > resource management. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6429) SortImpl Should Not Use BufferAllocator.setLenient()
Timothy Farkas created DRILL-6429: - Summary: SortImpl Should Not Use BufferAllocator.setLenient() Key: DRILL-6429 URL: https://issues.apache.org/jira/browse/DRILL-6429 Project: Apache Drill Issue Type: Improvement Reporter: Timothy Farkas Assignee: Timothy Farkas -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-5846) Improve Parquet Reader Performance for Flat Data types
[ https://issues.apache.org/jira/browse/DRILL-5846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16481271#comment-16481271 ] ASF GitHub Bot commented on DRILL-5846: --- sachouche commented on issue #1060: DRILL-5846: Improve parquet performance for Flat Data Types URL: https://github.com/apache/drill/pull/1060#issuecomment-390348355 @arina-ielchiieva I have addressed all requested changes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Improve Parquet Reader Performance for Flat Data types > --- > > Key: DRILL-5846 > URL: https://issues.apache.org/jira/browse/DRILL-5846 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Parquet >Affects Versions: 1.11.0 >Reporter: salim achouche >Assignee: salim achouche >Priority: Major > Labels: performance > Fix For: 1.14.0 > > Attachments: 2542d447-9837-3924-dd12-f759108461e5.sys.drill, > 2542d49b-88ef-38e3-a02b-b441c1295817.sys.drill > > > The Parquet Reader is a key use-case for Drill. This JIRA is an attempt to > further improve the Parquet Reader performance as several users reported that > Parquet parsing represents the lion share of the overall query execution. It > tracks Flat Data types only as Nested DTs might involve functional and > processing enhancements (e.g., a nested column can be seen as a Document; > user might want to perform operations scoped at the document level that is no > need to span all rows). Another JIRA will be created to handle the nested > columns use-case. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6428) HashAgg should not use BufferAllocator.setLenient()
Timothy Farkas created DRILL-6428: - Summary: HashAgg should not use BufferAllocator.setLenient() Key: DRILL-6428 URL: https://issues.apache.org/jira/browse/DRILL-6428 Project: Apache Drill Issue Type: Improvement Reporter: Timothy Farkas Assignee: Timothy Farkas Memory limits should not be lenient, they need to be explicit and deterministic. This is required for resource management. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6389) Fix building javadocs
[
https://issues.apache.org/jira/browse/DRILL-6389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timothy Farkas updated DRILL-6389:
--
Reviewer: Arina Ielchiieva
> Fix building javadocs
> -
>
> Key: DRILL-6389
> URL: https://issues.apache.org/jira/browse/DRILL-6389
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>
> Javadocs don't build when running
> {code}
> mvn javadoc:aggregate
> {code}
> Get the javadocs for all the classes to build. Fix some warnings.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6389) Fix building javadocs
[
https://issues.apache.org/jira/browse/DRILL-6389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16481264#comment-16481264
]
ASF GitHub Bot commented on DRILL-6389:
---
ilooner opened a new pull request #1276: DRILL-6389: Generate Javadoc for all
class and fix some warnings.
URL: https://github.com/apache/drill/pull/1276
# Fixes For Javadoc Generation
This patch accomplishes the following:
- Javadoc is generated for all classes (except for a few excluded classes).
- Javadoc warnings about unknown classes and packages are fixed.
- Some (but not all) easy javadoc warnings are fixed.
- Documentation was added on how to add and view javadocs
## Implementation Details
The javadoc plugin can only compile and link all the javadocs if `mvn
javadoc:aggregate` is called in the project root directory. The javadoc plugin
is also unaware of the test dependencies, so they must be explicitly added to
the plugin in order for javadoc to be able to resolve test classes.
Also since test dependencies must be explicitly added to the javadoc plugin,
we must also create test jars for drill-common and drill-java-exec which
contain common test classes used by other modules, and add these test-jars to
the javadoc plugin dependency list.
Some classes were excluded from javadoc generation due to weird package
linking issues I could not figure out resulting in javadoc warnings.
There was also a bug with the maven-license-plugin. The plugin definition
was included in the plugins section not the pluginManagement section of the
pom. This caused a strange issue where an extra license header was being added
in storage-kudu/src/main/codegen/templates/KuduRecordWriter.java . This caused
the javadoc tool to throw an error.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Fix building javadocs
> -
>
> Key: DRILL-6389
> URL: https://issues.apache.org/jira/browse/DRILL-6389
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>
> Javadocs don't build when running
> {code}
> mvn javadoc:aggregate
> {code}
> Get the javadocs for all the classes to build. Fix some warnings.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6389) Fix building javadocs
[
https://issues.apache.org/jira/browse/DRILL-6389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16481265#comment-16481265
]
ASF GitHub Bot commented on DRILL-6389:
---
ilooner commented on issue #1276: DRILL-6389: Generate Javadoc for all class
and fix some warnings.
URL: https://github.com/apache/drill/pull/1276#issuecomment-390347251
@arina-ielchiieva
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Fix building javadocs
> -
>
> Key: DRILL-6389
> URL: https://issues.apache.org/jira/browse/DRILL-6389
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>
> Javadocs don't build when running
> {code}
> mvn javadoc:aggregate
> {code}
> Get the javadocs for all the classes to build. Fix some warnings.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6389) Fix building javadocs
[
https://issues.apache.org/jira/browse/DRILL-6389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timothy Farkas updated DRILL-6389:
--
Description:
Javadocs don't build when running
{code}
mvn javadoc:aggregate
{code}
Get the javadocs for all the classes to build. Fix some warnings.
was:
Javadocs don't build when running
{code}
mvn javadoc:aggregate
{code}
Get the javadocs for all the classes to build
> Fix building javadocs
> -
>
> Key: DRILL-6389
> URL: https://issues.apache.org/jira/browse/DRILL-6389
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>
> Javadocs don't build when running
> {code}
> mvn javadoc:aggregate
> {code}
> Get the javadocs for all the classes to build. Fix some warnings.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-5846) Improve Parquet Reader Performance for Flat Data types
[ https://issues.apache.org/jira/browse/DRILL-5846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16481246#comment-16481246 ] ASF GitHub Bot commented on DRILL-5846: --- sachouche commented on a change in pull request #1060: DRILL-5846: Improve parquet performance for Flat Data Types URL: https://github.com/apache/drill/pull/1060#discussion_r189400447 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/VarLenAbstractEntryReader.java ## @@ -0,0 +1,266 @@ +/*** Review comment: I have fixed the headers and modified my maven script to always set "-Dlicense.skip" to false; shouldn't be unset by default? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Improve Parquet Reader Performance for Flat Data types > --- > > Key: DRILL-5846 > URL: https://issues.apache.org/jira/browse/DRILL-5846 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Parquet >Affects Versions: 1.11.0 >Reporter: salim achouche >Assignee: salim achouche >Priority: Major > Labels: performance > Fix For: 1.14.0 > > Attachments: 2542d447-9837-3924-dd12-f759108461e5.sys.drill, > 2542d49b-88ef-38e3-a02b-b441c1295817.sys.drill > > > The Parquet Reader is a key use-case for Drill. This JIRA is an attempt to > further improve the Parquet Reader performance as several users reported that > Parquet parsing represents the lion share of the overall query execution. It > tracks Flat Data types only as Nested DTs might involve functional and > processing enhancements (e.g., a nested column can be seen as a Document; > user might want to perform operations scoped at the document level that is no > need to span all rows). Another JIRA will be created to handle the nested > columns use-case. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-5846) Improve Parquet Reader Performance for Flat Data types
[
https://issues.apache.org/jira/browse/DRILL-5846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16481242#comment-16481242
]
ASF GitHub Bot commented on DRILL-5846:
---
sachouche commented on a change in pull request #1060: DRILL-5846: Improve
parquet performance for Flat Data Types
URL: https://github.com/apache/drill/pull/1060#discussion_r189400501
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/VarLenAbstractEntryReader.java
##
@@ -0,0 +1,266 @@
+/***
+
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+
**/
+package org.apache.drill.exec.store.parquet.columnreaders;
+
+import java.nio.ByteBuffer;
+
+import
org.apache.drill.exec.store.parquet.columnreaders.VarLenColumnBulkInput.ColumnPrecisionInfo;
+import
org.apache.drill.exec.store.parquet.columnreaders.VarLenColumnBulkInput.PageDataInfo;
+
+import io.netty.buffer.DrillBuf;
+
+/** Abstract class for sub-classes implementing several algorithms for loading
a Bulk Entry */
+abstract class VarLenAbstractEntryReader {
+
+ /** byte buffer used for buffering page data */
+ protected final ByteBuffer buffer;
+ /** Page Data Information */
+ protected final PageDataInfo pageInfo;
+ /** expected precision type: fixed or variable length */
+ protected final ColumnPrecisionInfo columnPrecInfo;
+ /** Bulk entry */
+ protected final VarLenColumnBulkEntry entry;
+
+ /**
+ * CTOR.
+ * @param _buffer byte buffer for data buffering (within CPU cache)
+ * @param _pageInfo page being processed information
+ * @param _columnPrecInfo column precision information
+ * @param _entry reusable bulk entry object
+ */
+ VarLenAbstractEntryReader(ByteBuffer _buffer,
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Improve Parquet Reader Performance for Flat Data types
> ---
>
> Key: DRILL-5846
> URL: https://issues.apache.org/jira/browse/DRILL-5846
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - Parquet
>Affects Versions: 1.11.0
>Reporter: salim achouche
>Assignee: salim achouche
>Priority: Major
> Labels: performance
> Fix For: 1.14.0
>
> Attachments: 2542d447-9837-3924-dd12-f759108461e5.sys.drill,
> 2542d49b-88ef-38e3-a02b-b441c1295817.sys.drill
>
>
> The Parquet Reader is a key use-case for Drill. This JIRA is an attempt to
> further improve the Parquet Reader performance as several users reported that
> Parquet parsing represents the lion share of the overall query execution. It
> tracks Flat Data types only as Nested DTs might involve functional and
> processing enhancements (e.g., a nested column can be seen as a Document;
> user might want to perform operations scoped at the document level that is no
> need to span all rows). Another JIRA will be created to handle the nested
> columns use-case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-5846) Improve Parquet Reader Performance for Flat Data types
[ https://issues.apache.org/jira/browse/DRILL-5846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16481240#comment-16481240 ] ASF GitHub Bot commented on DRILL-5846: --- sachouche commented on a change in pull request #1060: DRILL-5846: Improve parquet performance for Flat Data Types URL: https://github.com/apache/drill/pull/1060#discussion_r189400447 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/VarLenAbstractEntryReader.java ## @@ -0,0 +1,266 @@ +/*** Review comment: I have fixed the headers and modified my maven script to always set "-Dlicense.skip" to true; shouldn't be set by default? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Improve Parquet Reader Performance for Flat Data types > --- > > Key: DRILL-5846 > URL: https://issues.apache.org/jira/browse/DRILL-5846 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Parquet >Affects Versions: 1.11.0 >Reporter: salim achouche >Assignee: salim achouche >Priority: Major > Labels: performance > Fix For: 1.14.0 > > Attachments: 2542d447-9837-3924-dd12-f759108461e5.sys.drill, > 2542d49b-88ef-38e3-a02b-b441c1295817.sys.drill > > > The Parquet Reader is a key use-case for Drill. This JIRA is an attempt to > further improve the Parquet Reader performance as several users reported that > Parquet parsing represents the lion share of the overall query execution. It > tracks Flat Data types only as Nested DTs might involve functional and > processing enhancements (e.g., a nested column can be seen as a Document; > user might want to perform operations scoped at the document level that is no > need to span all rows). Another JIRA will be created to handle the nested > columns use-case. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-5846) Improve Parquet Reader Performance for Flat Data types
[
https://issues.apache.org/jira/browse/DRILL-5846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16481236#comment-16481236
]
ASF GitHub Bot commented on DRILL-5846:
---
sachouche commented on a change in pull request #1060: DRILL-5846: Improve
parquet performance for Flat Data Types
URL: https://github.com/apache/drill/pull/1060#discussion_r189399881
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java
##
@@ -289,6 +289,10 @@ private ExecConstants() {
public static final OptionValidator COMPILE_SCALAR_REPLACEMENT = new
BooleanValidator("exec.compile.scalar_replacement");
+ // Controls whether to enable bulk parquet reader processing
+ public static final String PARQUET_FLAT_READER_BULK=
"store.parquet.flat.reader.bulk";
Review comment:
A mentor of mine insisted on code formatting to improve the overall code
readability; it is very hard for me to forgo such a practice.. but I did. I
went through the changes and removed all formatting around the equal sign.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Improve Parquet Reader Performance for Flat Data types
> ---
>
> Key: DRILL-5846
> URL: https://issues.apache.org/jira/browse/DRILL-5846
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - Parquet
>Affects Versions: 1.11.0
>Reporter: salim achouche
>Assignee: salim achouche
>Priority: Major
> Labels: performance
> Fix For: 1.14.0
>
> Attachments: 2542d447-9837-3924-dd12-f759108461e5.sys.drill,
> 2542d49b-88ef-38e3-a02b-b441c1295817.sys.drill
>
>
> The Parquet Reader is a key use-case for Drill. This JIRA is an attempt to
> further improve the Parquet Reader performance as several users reported that
> Parquet parsing represents the lion share of the overall query execution. It
> tracks Flat Data types only as Nested DTs might involve functional and
> processing enhancements (e.g., a nested column can be seen as a Document;
> user might want to perform operations scoped at the document level that is no
> need to span all rows). Another JIRA will be created to handle the nested
> columns use-case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-5305) Query Profile must display Query ID
[
https://issues.apache.org/jira/browse/DRILL-5305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16481078#comment-16481078
]
ASF GitHub Bot commented on DRILL-5305:
---
kkhatua commented on a change in pull request #1265: DRILL-5305: Query Profile
must display Query ID
URL: https://github.com/apache/drill/pull/1265#discussion_r189368967
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/profile/ProfileWrapper.java
##
@@ -63,7 +63,7 @@
public ProfileWrapper(final QueryProfile profile, DrillConfig drillConfig) {
this.profile = profile;
-this.id = QueryIdHelper.getQueryId(profile.getId());
+this.id = profile.getId().hasText() ? profile.getId().getText() :
QueryIdHelper.getQueryId(profile.getId());
Review comment:
The JIRA's goal is to embed the query ID text within the profile. Since it
is within the profile, it would be more efficient to use that value versus
decoding repeatedly. Hence, it is an optional part of the QueryID (see proto
file), so that we can fall back to the decoding approach.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Query Profile must display Query ID
> ---
>
> Key: DRILL-5305
> URL: https://issues.apache.org/jira/browse/DRILL-5305
> Project: Apache Drill
> Issue Type: Improvement
> Components: Client - HTTP, Web Server
>Affects Versions: 1.10.0
>Reporter: Abhishek Girish
>Assignee: Kunal Khatua
>Priority: Minor
> Labels: newbie
> Fix For: 1.14.0
>
>
> It would be good to have the query ID, profile ID listed as part of the
> profile itself. Currently there is no easy way to map a profile with the
> query.
> I understand that the profile ID in the URL can be used, but often when we
> have profile dumps shared for analysis, they may not contain the query IDs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6423) Export query result as a CSV file
[ https://issues.apache.org/jira/browse/DRILL-6423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16481077#comment-16481077 ] ASF GitHub Bot commented on DRILL-6423: --- kkhatua commented on issue #1266: DRILL-6423: Export query result as a CSV file URL: https://github.com/apache/drill/pull/1266#issuecomment-390305729 @arina-ielchiieva I've done the changes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Export query result as a CSV file > - > > Key: DRILL-6423 > URL: https://issues.apache.org/jira/browse/DRILL-6423 > Project: Apache Drill > Issue Type: New Feature > Components: Web Server >Reporter: Kunal Khatua >Assignee: Kunal Khatua >Priority: Major > Fix For: 1.14.0 > > > This feature request is based on a question posted in the user mailing list: > [is there any way to download the data through Drill Web > UI?|https://lists.apache.org/thread.html/bd6b2453c83d818b618ec6334891309f42556d4a7de34d84024a43cd@] > Since the results of the WebUI allows for sorting and filtering as well, it > can be very useful if there is a way for a user to download the results of an > executed query for further processing in other tools (like Excel) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6321) Lateral Join: Planning changes - enable submitting physical plan
[
https://issues.apache.org/jira/browse/DRILL-6321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16481070#comment-16481070
]
ASF GitHub Bot commented on DRILL-6321:
---
amansinha100 commented on a change in pull request #1275: DRILL-6321: Lateral
Join and Unnest - star column prefix
URL: https://github.com/apache/drill/pull/1275#discussion_r189366569
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/lateraljoin/TestLateralPlans.java
##
@@ -90,8 +90,50 @@ public void testLateralSqlPlainCol() throws Exception {
@Test
public void testLateralSqlStar() throws Exception {
-String Sql = "select * from cp.`lateraljoin/nested-customer.json` t,
unnest(t.orders) t2 limit 1";
-test(Sql);
+String Sql = "select * from cp.`lateraljoin/nested-customer.json` t,
unnest(t.orders) t2 limit 0";
+
+testBuilder()
+.unOrdered()
+.sqlQuery(Sql)
+.baselineColumns("c_name", "c_id", "c_phone", "orders",
"c_address", "o_id", "o_shop", "o_amount", "items")
Review comment:
I think this set of columns is not right. The UNNEST should produce only
the top level column since that's what the planner knows about.
De-constructing the underlying map in the data means that we will not be able
to specify a column alias for the unnest output: e.g UNNEST(t.orders) as
t2(o).
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Lateral Join: Planning changes - enable submitting physical plan
>
>
> Key: DRILL-6321
> URL: https://issues.apache.org/jira/browse/DRILL-6321
> Project: Apache Drill
> Issue Type: Task
>Reporter: Parth Chandra
>Assignee: Chunhui Shi
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> Implement changes to enable submitting a physical plan containing lateral and
> unnest.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6418) Handle Schema change in Unnest And Lateral for unnest field / non-unnest field
[ https://issues.apache.org/jira/browse/DRILL-6418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sorabh Hamirwasia updated DRILL-6418: - Labels: ready-to-commit (was: ) > Handle Schema change in Unnest And Lateral for unnest field / non-unnest field > -- > > Key: DRILL-6418 > URL: https://issues.apache.org/jira/browse/DRILL-6418 > Project: Apache Drill > Issue Type: Improvement >Reporter: Sorabh Hamirwasia >Assignee: Sorabh Hamirwasia >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > Handling the schema change scenarios for LATERAL and UNNEST when schema > change is observed for non-unnest field and unnest field. It should also > handle the scenario when UNNEST field is Repeated Map Type and schema change > happens only in the children field of Map component. Following issues were > found: > 1) There were issues found in how scan treats that kind of data where it > scans 2 files, one has Map (let say cutomer_order) children type (custKey) as > integer and other has custKey as string type. Then Scan just replaces the > ValueVector from its output container for custKey but in the schema it has 2 > fields with same name but different types. > 2) Unnest and Lateral check for schema change across batches based on the > MaterializedField and BatchSchema reference they store. But since it's a > reference any change in pointed value changes their reference as well and > hence comparison always returns true. So for Unnest it actually keeps a clone > of the Materialized Field instead of reference and use that to determine for > schema change. For LATERAL any OK_NEW_SCHEMA from upstream is treated as > actual schema change and setup is done again. > 3) Since the MaterializedField is mutable the isEquivalent method checks for > object equality which is removed as that is not correct for mutable objects. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6425) Upgrade org.ojai:ojai version
[
https://issues.apache.org/jira/browse/DRILL-6425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16481048#comment-16481048
]
ASF GitHub Bot commented on DRILL-6425:
---
vrozov commented on issue #1270: DRILL-6425: Upgrade org.ojai:ojai version
URL: https://github.com/apache/drill/pull/1270#issuecomment-390299485
@arina-ielchiieva I guess [mapr maven
repository](http://repository.mapr.com/maven/) was down when you tried to
build. Can you try one more time.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Upgrade org.ojai:ojai version
> -
>
> Key: DRILL-6425
> URL: https://issues.apache.org/jira/browse/DRILL-6425
> Project: Apache Drill
> Issue Type: Task
>Reporter: Vlad Rozov
>Assignee: Vlad Rozov
>Priority: Major
> Fix For: 1.14.0
>
>
> Upgrade from {{1.1}} to {{2.0.1-mapr-1804}} or the most recent.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6425) Upgrade org.ojai:ojai version
[
https://issues.apache.org/jira/browse/DRILL-6425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16481043#comment-16481043
]
ASF GitHub Bot commented on DRILL-6425:
---
vrozov commented on a change in pull request #1270: DRILL-6425: Upgrade
org.ojai:ojai version
URL: https://github.com/apache/drill/pull/1270#discussion_r189361374
##
File path: pom.xml
##
@@ -52,7 +52,7 @@
2.7.9
2.7.9.1
5.2.1-mapr
-1.1
+2.0.1-mapr-1804
Review comment:
Please see
http://repository.mapr.com/nexus/content/groups/mapr-public/org/ojai/ojai/2.0.1-mapr-1804/.
The latest version is required to avoid `2.0.1-SNAPSHOT` dependency used by
maprdb 6.0.1 otherwise Drill can simply drop that dependency from the
`DependencyManagement`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Upgrade org.ojai:ojai version
> -
>
> Key: DRILL-6425
> URL: https://issues.apache.org/jira/browse/DRILL-6425
> Project: Apache Drill
> Issue Type: Task
>Reporter: Vlad Rozov
>Assignee: Vlad Rozov
>Priority: Major
> Fix For: 1.14.0
>
>
> Upgrade from {{1.1}} to {{2.0.1-mapr-1804}} or the most recent.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6421) Refactor DecimalUtility and CoreDecimalUtility classes
[
https://issues.apache.org/jira/browse/DRILL-6421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16481027#comment-16481027
]
ASF GitHub Bot commented on DRILL-6421:
---
asfgit closed pull request #1267: DRILL-6421: Refactor DecimalUtility and
CoreDecimalUtility classes
URL: https://github.com/apache/drill/pull/1267
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/common/src/main/java/org/apache/drill/common/types/Types.java
b/common/src/main/java/org/apache/drill/common/types/Types.java
index 5c6e7f3b97..e66a340f1f 100644
--- a/common/src/main/java/org/apache/drill/common/types/Types.java
+++ b/common/src/main/java/org/apache/drill/common/types/Types.java
@@ -27,12 +27,11 @@
import org.apache.drill.common.types.TypeProtos.DataMode;
import org.apache.drill.common.types.TypeProtos.MajorType;
import org.apache.drill.common.types.TypeProtos.MinorType;
-import org.apache.drill.common.util.CoreDecimalUtility;
import com.google.protobuf.TextFormat;
+@SuppressWarnings("WeakerAccess")
public class Types {
- static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(Types.class);
public static final int MAX_VARCHAR_LENGTH = 65535;
public static final int UNDEFINED = 0;
@@ -47,10 +46,6 @@ public static boolean isUnion(MajorType toType) {
return toType.getMinorType() == MinorType.UNION;
}
- public enum Comparability {
-UNKNOWN, NONE, EQUAL, ORDERED
- }
-
public static boolean isComplex(final MajorType type) {
switch(type.getMinorType()) {
case LIST:
@@ -90,6 +85,37 @@ public static boolean isNumericType(final MajorType type) {
case UINT4:
case UINT8:
return true;
+default:
+ return false;
+}
+ }
+
+ /**
+ * Returns true if specified type is decimal data type.
+ *
+ * @param type type to check
+ * @return true if specified type is decimal data type.
+ */
+ public static boolean isDecimalType(MajorType type) {
+return isDecimalType(type.getMinorType());
+ }
+
+ /**
+ * Returns true if specified type is decimal data type.
+ *
+ * @param minorType type to check
+ * @return true if specified type is decimal data type.
+ */
+ public static boolean isDecimalType(MinorType minorType) {
+switch(minorType) {
+ case VARDECIMAL:
+ case DECIMAL38SPARSE:
+ case DECIMAL38DENSE:
+ case DECIMAL28SPARSE:
+ case DECIMAL28DENSE:
+ case DECIMAL18:
+ case DECIMAL9:
+return true;
default:
return false;
}
@@ -182,11 +208,9 @@ public static String getBaseSqlTypeName(final MajorType
type) {
* Extend decimal type with precision and scale.
*
* @param type major type
- * @param typeName type converted to a string
* @return type name augmented with precision and scale,
* if type is a decimal
*/
-
public static String getExtendedSqlTypeName(MajorType type) {
String typeName = getSqlTypeName(type);
@@ -266,7 +290,7 @@ public static int getJdbcTypeCode(final String sqlTypeName)
{
/**
* Reports whether given RPC-level type is a signed type (per semantics of
- * {@link ResultSetMetaData#isSigned(int)}).
+ * {@link java.sql.ResultSetMetaData#isSigned(int)}).
*/
public static boolean isJdbcSignedType( final MajorType type ) {
final boolean isSigned;
@@ -477,41 +501,6 @@ public static boolean isScalarStringType(final MajorType
type) {
}
}
- public static boolean isBytesScalarType(final MajorType type) {
-if (type.getMode() == REPEATED) {
- return false;
-}
-switch(type.getMinorType()) {
-case FIXEDBINARY:
-case VARBINARY:
- return true;
-default:
- return false;
-}
- }
-
- public static Comparability getComparability(final MajorType type) {
-if (type.getMode() == REPEATED) {
- return Comparability.NONE;
-}
-if (type.getMinorType() == MinorType.LATE) {
- return Comparability.UNKNOWN;
-}
-
-switch(type.getMinorType()) {
-case LATE:
- return Comparability.UNKNOWN;
-case MAP:
- return Comparability.NONE;
-case BIT:
- return Comparability.EQUAL;
-default:
- return Comparability.ORDERED;
-}
-
- }
-
-
public static boolean softEquals(final MajorType a, final MajorType b, final
boolean allowNullSwap) {
if (a.getMinorType() != b.getMinorType()) {
return false;
@@ -532,10 +521,6 @@ public static boolean softEquals(final MajorType a, final
MajorType b, final boo
return a.getMode() == b.getMode();
}
- public static boolean isLateBind(final MajorType type) {
-return type.getMinorType() == MinorType.LATE;
- }
-
public static boolean isUntypedNull(final
[jira] [Commented] (DRILL-6413) Specific query returns an exception if filter a boolean column by "equals" operator
[
https://issues.apache.org/jira/browse/DRILL-6413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16481029#comment-16481029
]
ASF GitHub Bot commented on DRILL-6413:
---
asfgit closed pull request #1269: DRILL-6413: Update
ParquetFilterBuilder.visitBooleanOperator to handle simplied boolean expression
URL: https://github.com/apache/drill/pull/1269
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/AbstractParquetGroupScan.java
b/exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/AbstractParquetGroupScan.java
index 1f8c535a91..33472bb654 100644
---
a/exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/AbstractParquetGroupScan.java
+++
b/exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/AbstractParquetGroupScan.java
@@ -255,8 +255,7 @@ public GroupScan applyFilter(LogicalExpression filterExpr,
UdfUtilities udfUtili
logger.debug("materializedFilter : {}",
ExpressionStringBuilder.toString(materializedFilter));
Set constantBoundaries =
ConstantExpressionIdentifier.getConstantExpressionSet(materializedFilter);
-filterPredicate = (ParquetFilterPredicate)
ParquetFilterBuilder.buildParquetFilterPredicate(
-materializedFilter, constantBoundaries, udfUtilities);
+filterPredicate =
ParquetFilterBuilder.buildParquetFilterPredicate(materializedFilter,
constantBoundaries, udfUtilities);
if (filterPredicate == null) {
return null;
diff --git
a/exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetFilterBuilder.java
b/exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetFilterBuilder.java
index e55425e948..a8e101d6e0 100644
---
a/exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetFilterBuilder.java
+++
b/exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetFilterBuilder.java
@@ -42,6 +42,7 @@
import org.apache.drill.exec.expr.holders.VarDecimalHolder;
import org.apache.drill.exec.expr.stat.ParquetBooleanPredicates;
import org.apache.drill.exec.expr.stat.ParquetComparisonPredicates;
+import org.apache.drill.exec.expr.stat.ParquetFilterPredicate;
import org.apache.drill.exec.expr.stat.ParquetIsPredicates;
import org.apache.drill.exec.ops.UdfUtilities;
import org.apache.drill.exec.util.DecimalUtility;
@@ -54,7 +55,7 @@
/**
* A visitor which visits a materialized logical expression, and build
ParquetFilterPredicate
- * If a visitXXX method returns null, that means the corresponding filter
branch is not qualified for pushdown.
+ * If a visitXXX method returns null, that means the corresponding filter
branch is not qualified for push down.
*/
public class ParquetFilterBuilder extends
AbstractExprVisitor {
static final Logger logger =
LoggerFactory.getLogger(ParquetFilterBuilder.class);
@@ -66,12 +67,18 @@
* @param constantBoundaries set of constant expressions
* @param udfUtilities udf utilities
*
- * @return logical expression
+ * @return parquet filter predicate
*/
- public static LogicalExpression
buildParquetFilterPredicate(LogicalExpression expr, final
Set constantBoundaries, UdfUtilities udfUtilities) {
-return expr.accept(new ParquetFilterBuilder(udfUtilities),
constantBoundaries);
+ public static ParquetFilterPredicate
buildParquetFilterPredicate(LogicalExpression expr, final
Set constantBoundaries, UdfUtilities udfUtilities) {
+LogicalExpression logicalExpression = expr.accept(new
ParquetFilterBuilder(udfUtilities), constantBoundaries);
+if (logicalExpression instanceof ParquetFilterPredicate) {
+ return (ParquetFilterPredicate) logicalExpression;
+}
+logger.debug("Logical expression {} was not qualified for filter push
down", logicalExpression);
+return null;
}
+
private ParquetFilterBuilder(UdfUtilities udfUtilities) {
this.udfUtilities = udfUtilities;
}
@@ -150,6 +157,10 @@ public LogicalExpression
visitBooleanOperator(BooleanOperator op, Set 1");
+final String sql = "select * from dfs.`parquet/multirowgroup.parquet`
where a > 1";
final String[] expectedPlan = {"numRowGroups=1"};
-final String[] excludedPlan = {};
-PlanTestBase.testPlanMatchingPatterns(sql, expectedPlan, excludedPlan);
+PlanTestBase.testPlanMatchingPatterns(sql, expectedPlan);
}
//
@@ -461,23 +464,20 @@ private void testParquetFilterPD(final String query, int
expectedRowCount, int e
String numFilesPattern =
[jira] [Commented] (DRILL-6426) Refactor TestFunctionsWithTypeExpoQueries test to be independent on limit 0 optimization option
[
https://issues.apache.org/jira/browse/DRILL-6426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16481028#comment-16481028
]
ASF GitHub Bot commented on DRILL-6426:
---
asfgit closed pull request #1273: DRILL-6426: Refactor
TestFunctionsWithTypeExpoQueries test to be independent on limit 0 optimization
option
URL: https://github.com/apache/drill/pull/1273
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/exec/java-exec/src/test/java/org/apache/drill/TestFunctionsWithTypeExpoQueries.java
b/exec/java-exec/src/test/java/org/apache/drill/TestFunctionsWithTypeExpoQueries.java
index 88558ae000..d298acc98f 100644
---
a/exec/java-exec/src/test/java/org/apache/drill/TestFunctionsWithTypeExpoQueries.java
+++
b/exec/java-exec/src/test/java/org/apache/drill/TestFunctionsWithTypeExpoQueries.java
@@ -22,7 +22,10 @@
import org.apache.drill.categories.SqlFunctionTest;
import org.apache.drill.common.expression.SchemaPath;
import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.ExecConstants;
import org.apache.drill.test.BaseTestQuery;
+import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.Test;
import org.junit.experimental.categories.Category;
@@ -32,9 +35,16 @@
@Category(SqlFunctionTest.class)
public class TestFunctionsWithTypeExpoQueries extends BaseTestQuery {
+
@BeforeClass
- public static void setupFiles() {
+ public static void setup() {
dirTestWatcher.copyResourceToRoot(Paths.get("typeExposure"));
+alterSession(ExecConstants.EARLY_LIMIT0_OPT_KEY, true);
+ }
+
+ @AfterClass
+ public static void tearDown() {
+resetSessionOption(ExecConstants.EARLY_LIMIT0_OPT_KEY);
}
@Test
@@ -44,9 +54,10 @@ public void testConcatWithMoreThanTwoArgs() throws Exception
{
List> expectedSchema =
Lists.newArrayList();
TypeProtos.MajorType majorType = TypeProtos.MajorType.newBuilder()
-.setMinorType(TypeProtos.MinorType.VARCHAR)
-.setMode(TypeProtos.DataMode.REQUIRED)
-.build();
+.setMinorType(TypeProtos.MinorType.VARCHAR)
+.setMode(TypeProtos.DataMode.REQUIRED)
+.setPrecision(Types.MAX_VARCHAR_LENGTH)
+.build();
expectedSchema.add(Pair.of(SchemaPath.getSimplePath("col"), majorType));
testBuilder()
@@ -95,6 +106,7 @@ public void testLRBTrimOneArg() throws Exception {
TypeProtos.MajorType majorType = TypeProtos.MajorType.newBuilder()
.setMinorType(TypeProtos.MinorType.VARCHAR)
.setMode(TypeProtos.DataMode.REQUIRED)
+.setPrecision(Types.MAX_VARCHAR_LENGTH)
.build();
expectedSchema.add(Pair.of(SchemaPath.getSimplePath("col"), majorType));
@@ -125,9 +137,10 @@ public void testTrim() throws Exception {
List> expectedSchema =
Lists.newArrayList();
TypeProtos.MajorType majorType = TypeProtos.MajorType.newBuilder()
-.setMinorType(TypeProtos.MinorType.VARCHAR)
-.setMode(TypeProtos.DataMode.REQUIRED)
-.build();
+.setMinorType(TypeProtos.MinorType.VARCHAR)
+.setMode(TypeProtos.DataMode.REQUIRED)
+.setPrecision(Types.MAX_VARCHAR_LENGTH)
+.build();
expectedSchema.add(Pair.of(SchemaPath.getSimplePath("col"), majorType));
testBuilder()
@@ -159,6 +172,7 @@ public void testTrimOneArg() throws Exception {
TypeProtos.MajorType majorType = TypeProtos.MajorType.newBuilder()
.setMinorType(TypeProtos.MinorType.VARCHAR)
.setMode(TypeProtos.DataMode.REQUIRED)
+.setPrecision(Types.MAX_VARCHAR_LENGTH)
.build();
expectedSchema.add(Pair.of(SchemaPath.getSimplePath("col"), majorType));
@@ -191,6 +205,7 @@ public void testTrimTwoArg() throws Exception {
TypeProtos.MajorType majorType = TypeProtos.MajorType.newBuilder()
.setMinorType(TypeProtos.MinorType.VARCHAR)
.setMode(TypeProtos.DataMode.REQUIRED)
+.setPrecision(Types.MAX_VARCHAR_LENGTH)
.build();
expectedSchema.add(Pair.of(SchemaPath.getSimplePath("col"), majorType));
@@ -214,12 +229,12 @@ public void testTrimTwoArg() throws Exception {
}
@Test
- public void tesIsNull() throws Exception {
+ public void testIsNull() throws Exception {
final String query = "select r_name is null as col from
cp.`tpch/region.parquet` limit 0";
List> expectedSchema =
Lists.newArrayList();
TypeProtos.MajorType majorType = TypeProtos.MajorType.newBuilder()
.setMinorType(TypeProtos.MinorType.BIT)
-
[jira] [Commented] (DRILL-6255) Drillbit while sending control message to itself creates a connection instead of submitting locally
[
https://issues.apache.org/jira/browse/DRILL-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16481026#comment-16481026
]
ASF GitHub Bot commented on DRILL-6255:
---
asfgit closed pull request #1253: DRILL-6255: Drillbit while sending control
message to itself creates …
URL: https://github.com/apache/drill/pull/1253
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/exec/java-exec/src/main/java/org/apache/drill/exec/rpc/BitRpcUtility.java
b/exec/java-exec/src/main/java/org/apache/drill/exec/rpc/BitRpcUtility.java
index c3980336f7..0bc01f5367 100644
--- a/exec/java-exec/src/main/java/org/apache/drill/exec/rpc/BitRpcUtility.java
+++ b/exec/java-exec/src/main/java/org/apache/drill/exec/rpc/BitRpcUtility.java
@@ -103,6 +103,24 @@ void prepareSaslHandshake(final RpcConnectionHandler
connectionHandler, List
}
}
+ /**
+ * Verifies if local and remote Drillbit Endpoint has same control server by
using address and control port
+ * information. This method is used instead of equals in {@link
DrillbitEndpoint} because DrillbitEndpoint stores
+ * state information in it.
+ * For local Drillbit a reference is stored in {@link
org.apache.drill.exec.server.DrillbitContext} as soon as
+ * Drillbit is started in {@link
org.apache.drill.exec.service.ServiceEngine#start} with state as STARTUP, but
+ * while planning minor fragment the assignment list is used from active
list of Drillbits in which state for local
+ * Drillbit will not be STARTUP
+ * @param local - DrillbitEndpoint instance for local bit
+ * @param remote - DrillbitEndpoint instance for remote bit
+ * @return true if address and control port for local and remote are same.
+ * false - otherwise
+ */
+ public static boolean isLocalControlServer(DrillbitEndpoint local,
DrillbitEndpoint remote) {
+return local.hasAddress() && local.hasControlPort() && remote.hasAddress()
&& remote.hasControlPort() &&
+ local.getAddress().equals(remote.getAddress()) && local.getControlPort()
== remote.getControlPort();
+ }
+
// Suppress default constructor
private BitRpcUtility() {
}
diff --git
a/exec/java-exec/src/main/java/org/apache/drill/exec/rpc/control/ConnectionManagerRegistry.java
b/exec/java-exec/src/main/java/org/apache/drill/exec/rpc/control/ConnectionManagerRegistry.java
index 800cf3cc85..e27729c1f4 100644
---
a/exec/java-exec/src/main/java/org/apache/drill/exec/rpc/control/ConnectionManagerRegistry.java
+++
b/exec/java-exec/src/main/java/org/apache/drill/exec/rpc/control/ConnectionManagerRegistry.java
@@ -23,6 +23,7 @@
import org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint;
import com.google.common.collect.Maps;
+import org.apache.drill.exec.rpc.BitRpcUtility;
public class ConnectionManagerRegistry implements
Iterable {
static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(ConnectionManagerRegistry.class);
@@ -40,9 +41,12 @@ public ConnectionManagerRegistry(ControlConnectionConfig
config) {
public ControlConnectionManager getConnectionManager(DrillbitEndpoint
remoteEndpoint) {
assert localEndpoint != null :
"DrillbitEndpoint must be set before a connection manager can be
retrieved";
+
+final boolean isLocalServer =
BitRpcUtility.isLocalControlServer(localEndpoint, remoteEndpoint);
ControlConnectionManager m = registry.get(remoteEndpoint);
if (m == null) {
- m = new ControlConnectionManager(config, localEndpoint, remoteEndpoint);
+ m = (isLocalServer) ? new LocalControlConnectionManager(config,
remoteEndpoint)
+: new RemoteControlConnectionManager(config,
localEndpoint, remoteEndpoint);
final ControlConnectionManager m2 = registry.putIfAbsent(remoteEndpoint,
m);
if (m2 != null) {
m = m2;
diff --git
a/exec/java-exec/src/main/java/org/apache/drill/exec/rpc/control/ControlConnectionConfig.java
b/exec/java-exec/src/main/java/org/apache/drill/exec/rpc/control/ControlConnectionConfig.java
index b19fb8bb75..f114af44de 100644
---
a/exec/java-exec/src/main/java/org/apache/drill/exec/rpc/control/ControlConnectionConfig.java
+++
b/exec/java-exec/src/main/java/org/apache/drill/exec/rpc/control/ControlConnectionConfig.java
@@ -17,6 +17,7 @@
*/
package org.apache.drill.exec.rpc.control;
+import com.google.common.annotations.VisibleForTesting;
import org.apache.drill.exec.exception.DrillbitStartupException;
import org.apache.drill.exec.memory.BufferAllocator;
import org.apache.drill.exec.rpc.BitConnectionConfig;
@@ -24,8 +25,8 @@
import org.apache.drill.exec.work.batch.ControlMessageHandler;
// config for bit to bit connection
-// package private
[jira] [Commented] (DRILL-4525) Query with BETWEEN clause on Date and Timestamp values fails with Validation Error
[
https://issues.apache.org/jira/browse/DRILL-4525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16481031#comment-16481031
]
ASF GitHub Bot commented on DRILL-4525:
---
asfgit closed pull request #1268: DRILL-4525: Allow SqlBetweenOperator to
accept LOWER_OPERAND and UPPER_OPERAND with different types
URL: https://github.com/apache/drill/pull/1268
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DrillCalciteSqlBetweenOperatorWrapper.java
b/exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DrillCalciteSqlBetweenOperatorWrapper.java
new file mode 100644
index 00..f112e57908
--- /dev/null
+++
b/exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DrillCalciteSqlBetweenOperatorWrapper.java
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql;
+
+import org.apache.calcite.sql.fun.SqlBetweenOperator;
+import org.apache.calcite.sql.SqlCallBinding;
+import org.apache.calcite.sql.SqlOperator;
+
+import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.exec.resolver.TypeCastRules;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * This class serves as a wrapper class for SqlBetweenOperator. The motivation
is to plug-in the return type inference and operand
+ * type check algorithms of Drill into Calcite's sql validation procedure.
+ *
+ * Except for the methods which are relevant to the return type inference and
operand type check algorithms, the wrapper
+ * simply forwards the method calls to the wrapped SqlOperator.
+ *
+ * Note that SqlBetweenOperator cannot be wrapped in {@link
DrillCalciteSqlOperatorWrapper}. The reason is when RexNode
+ * conversion is happening, StandardConvertletTable.convertBetween expects the
SqlOperator to be a subclass of SqlBetweenOperator.
+ */
+public class DrillCalciteSqlBetweenOperatorWrapper extends SqlBetweenOperator
implements DrillCalciteSqlWrapper {
+ private final SqlBetweenOperator operator;
+
+ public DrillCalciteSqlBetweenOperatorWrapper(SqlBetweenOperator
sqlBetweenOperator) {
+super(sqlBetweenOperator.flag, sqlBetweenOperator.isNegated());
+operator = sqlBetweenOperator;
+ }
+
+ @Override
+ public SqlOperator getOperator() {
+return operator;
+ }
+
+ /**
+ * Since Calcite has its rule for type compatibility
+ * (see {@link
org.apache.calcite.sql.type.SqlTypeUtil#isComparable(org.apache.calcite.rel.type.RelDataType,
+ * org.apache.calcite.rel.type.RelDataType)}), which is usually different
from Drill's, this method is overridden here to avoid
+ * Calcite early terminating the queries.
+ */
+ @Override
+ public boolean checkOperandTypes(SqlCallBinding callBinding, boolean
throwOnFailure) {
+final List types = new ArrayList<>();
+for (int i = 0; i < callBinding.getOperandCount(); i++) {
+ final TypeProtos.MinorType inMinorType =
TypeInferenceUtils.getDrillTypeFromCalciteType(callBinding.getOperandType(i));
+ if (inMinorType == TypeProtos.MinorType.LATE) {
+return true;
+ }
+ types.add(inMinorType);
+}
+
+final boolean isCompatible = TypeCastRules.getLeastRestrictiveType(types)
!= null;
+if (!isCompatible && throwOnFailure) {
+ throw callBinding.newValidationSignatureError();
+}
+return isCompatible;
+ }
+}
diff --git
a/exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DrillOperatorTable.java
b/exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DrillOperatorTable.java
index cf858d3e27..aea73d80db 100644
---
a/exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DrillOperatorTable.java
+++
b/exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DrillOperatorTable.java
@@ -22,6 +22,7 @@
import com.google.common.collect.Maps;
import org.apache.calcite.sql.SqlAggFunction;
import
[jira] [Commented] (DRILL-6424) Updating FasterXML Jackson libraries
[
https://issues.apache.org/jira/browse/DRILL-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16481030#comment-16481030
]
ASF GitHub Bot commented on DRILL-6424:
---
asfgit closed pull request #1274: DRILL-6424: Updating FasterXML Jackson
libraries
URL: https://github.com/apache/drill/pull/1274
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/contrib/storage-hive/hive-exec-shade/pom.xml
b/contrib/storage-hive/hive-exec-shade/pom.xml
index ad572efaff..6f511adf71 100644
--- a/contrib/storage-hive/hive-exec-shade/pom.xml
+++ b/contrib/storage-hive/hive-exec-shade/pom.xml
@@ -167,23 +167,4 @@
-
-
- mapr
-
-
-
2.4.2
-
-
-
-
-com.fasterxml.jackson.core
-jackson-databind
-${jackson.databind.mapr.hive.version}
-
-
-
-
-
diff --git
a/exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/json/JsonRecordWriter.java
b/exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/json/JsonRecordWriter.java
index b350d57783..bd81578870 100644
---
a/exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/json/JsonRecordWriter.java
+++
b/exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/json/JsonRecordWriter.java
@@ -18,6 +18,7 @@
package org.apache.drill.exec.store.easy.json;
import java.io.IOException;
+import java.io.OutputStream;
import java.util.List;
import java.util.Map;
@@ -33,7 +34,6 @@
import org.apache.drill.exec.vector.complex.fn.JsonWriter;
import org.apache.drill.exec.vector.complex.reader.FieldReader;
import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.fs.FSDataOutputStream;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
@@ -56,7 +56,7 @@
private int index;
private FileSystem fs = null;
- private FSDataOutputStream stream = null;
+ private OutputStream stream = null;
private final JsonFactory factory = new JsonFactory();
private final StorageStrategy storageStrategy;
diff --git
a/exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/metadata/Metadata.java
b/exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/metadata/Metadata.java
index cdf98e605b..ab655e9217 100644
---
a/exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/metadata/Metadata.java
+++
b/exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/metadata/Metadata.java
@@ -40,8 +40,6 @@
import org.apache.drill.exec.util.DrillFileSystemUtil;
import org.apache.drill.exec.util.ImpersonationUtil;
import org.apache.hadoop.fs.BlockLocation;
-import org.apache.hadoop.fs.FSDataInputStream;
-import org.apache.hadoop.fs.FSDataOutputStream;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
@@ -60,6 +58,8 @@
import javax.annotation.Nullable;
import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
import java.security.PrivilegedExceptionAction;
import java.util.ArrayList;
import java.util.Iterator;
@@ -542,7 +542,7 @@ private void writeFile(ParquetTableMetadata_v3
parquetTableMetadata, Path p, Fil
SimpleModule module = new SimpleModule();
module.addSerializer(ColumnMetadata_v3.class, new
ColumnMetadata_v3.Serializer());
mapper.registerModule(module);
-FSDataOutputStream os = fs.create(p);
+OutputStream os = fs.create(p);
mapper.writerWithDefaultPrettyPrinter().writeValue(os,
parquetTableMetadata);
os.flush();
os.close();
@@ -555,7 +555,7 @@ private void writeFile(ParquetTableMetadataDirs
parquetTableMetadataDirs, Path p
ObjectMapper mapper = new ObjectMapper(jsonFactory);
SimpleModule module = new SimpleModule();
mapper.registerModule(module);
-FSDataOutputStream os = fs.create(p);
+OutputStream os = fs.create(p);
mapper.writerWithDefaultPrettyPrinter().writeValue(os,
parquetTableMetadataDirs);
os.flush();
os.close();
@@ -586,7 +586,7 @@ private void readBlockMeta(Path path, boolean dirsOnly,
MetadataContext metaCont
mapper.registerModule(serialModule);
mapper.registerModule(module);
mapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
-try (FSDataInputStream is = fs.open(path)) {
+try (InputStream is = fs.open(path)) {
boolean alreadyCheckedModification;
boolean newMetadata = false;
alreadyCheckedModification =
metaContext.getStatus(metadataParentDirPath);
diff --git
a/logical/src/main/java/org/apache/drill/common/logical/data/Sequence.java
[jira] [Commented] (DRILL-6420) Add Lateral and Unnest Keyword for highlighting on WebUI
[
https://issues.apache.org/jira/browse/DRILL-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16481025#comment-16481025
]
ASF GitHub Bot commented on DRILL-6420:
---
asfgit closed pull request #1261: DRILL-6420: Add Lateral and Unnest Keyword
for highlighting on WebUI
URL: https://github.com/apache/drill/pull/1261
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/exec/java-exec/src/main/resources/rest/static/js/ace-code-editor/mode-sql.js
b/exec/java-exec/src/main/resources/rest/static/js/ace-code-editor/mode-sql.js
index 992fdb6842..ceb47c8f04 100644
---
a/exec/java-exec/src/main/resources/rest/static/js/ace-code-editor/mode-sql.js
+++
b/exec/java-exec/src/main/resources/rest/static/js/ace-code-editor/mode-sql.js
@@ -15,11 +15,11 @@ var SqlHighlightRules = function() {
//Covered: https://drill.apache.org/docs/supported-sql-commands/
var keywords = (
"select|insert|update|delete|from|where|and|or|group|by|order|limit|offset|having|as|case|"
+
-
"when|else|end|type|left|right|join|on|outer|desc|asc|union|create|table|key|if|"
+
+
"when|else|end|type|left|right|join|on|outer|desc|asc|union|create|table|key|if|lateral|apply|unnest|"
+
"not|default|null|inner|database|drop|" +
"flatten|kvgen|columns|" +
"set|reset|alter|session|system|" +
-"temporary|function|using|jar|between|distinct" +
+"temporary|function|using|jar|between|distinct|" +
"partition|view|schema|files|" +
"explain|plan|with|without|implementation|" +
"show|describe|use"
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add Lateral and Unnest Keyword for highlighting on WebUI
>
>
> Key: DRILL-6420
> URL: https://issues.apache.org/jira/browse/DRILL-6420
> Project: Apache Drill
> Issue Type: Task
> Components: Web Server
>Reporter: Sorabh Hamirwasia
>Assignee: Sorabh Hamirwasia
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.14.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6427) outputBatchSize is missing from the DEBUG output for HashJoinBatch operator
Khurram Faraaz created DRILL-6427:
-
Summary: outputBatchSize is missing from the DEBUG output for
HashJoinBatch operator
Key: DRILL-6427
URL: https://issues.apache.org/jira/browse/DRILL-6427
Project: Apache Drill
Issue Type: Bug
Reporter: Khurram Faraaz
Assignee: Padma Penumarthy
Drill 1.14.0-SNAPSHOT commit : f99d1f1323c0a5bac99842d6283d3025f3cb527f
outputBatchSize is missing from the DEBUG output for HashJoinBatch operator
Query used in test,
{noformat}
select count(*) from `twovarchar_asc_128MB.parquet` t1,
`twovarchar_asc_16MB.parquet` t2 WHERE t1.Varbinaryvalue = t2.Varbinaryvalue
{noformat}
Snippet from drillbit.log
{noformat}
2018-05-18 11:23:59,655 [2500e5c3-8f54-1f92-6eeb-7a81499a8abd:frag:0:0] DEBUG
o.a.d.e.p.impl.join.HashJoinBatch - left input: batch count : 1, avg batch
bytes : 90145920, avg row bytes : 8166, record count : 11040
2018-05-18 11:23:59,655 [2500e5c3-8f54-1f92-6eeb-7a81499a8abd:frag:0:0] DEBUG
o.a.d.e.p.impl.join.HashJoinBatch - right input: batch count : 1, avg batch
bytes : 10567808, avg row bytes : 7506, record count : 1408
2018-05-18 11:23:59,655 [2500e5c3-8f54-1f92-6eeb-7a81499a8abd:frag:0:0] DEBUG
o.a.d.e.p.impl.join.HashJoinBatch - output: batch count : 166, avg batch bytes
: 15951453, avg row bytes : 15672, record count : 168960
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-5305) Query Profile must display Query ID
[
https://issues.apache.org/jira/browse/DRILL-5305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480992#comment-16480992
]
ASF GitHub Bot commented on DRILL-5305:
---
ppadma commented on a change in pull request #1265: DRILL-5305: Query Profile
must display Query ID
URL: https://github.com/apache/drill/pull/1265#discussion_r189349466
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/profile/ProfileWrapper.java
##
@@ -63,7 +63,7 @@
public ProfileWrapper(final QueryProfile profile, DrillConfig drillConfig) {
this.profile = profile;
-this.id = QueryIdHelper.getQueryId(profile.getId());
+this.id = profile.getId().hasText() ? profile.getId().getText() :
QueryIdHelper.getQueryId(profile.getId());
Review comment:
why not decode always instead of doing the check ? It is not a big deal
either way.
LGTM +1.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Query Profile must display Query ID
> ---
>
> Key: DRILL-5305
> URL: https://issues.apache.org/jira/browse/DRILL-5305
> Project: Apache Drill
> Issue Type: Improvement
> Components: Client - HTTP, Web Server
>Affects Versions: 1.10.0
>Reporter: Abhishek Girish
>Assignee: Kunal Khatua
>Priority: Minor
> Labels: newbie
> Fix For: 1.14.0
>
>
> It would be good to have the query ID, profile ID listed as part of the
> profile itself. Currently there is no easy way to map a profile with the
> query.
> I understand that the profile ID in the URL can be used, but often when we
> have profile dumps shared for analysis, they may not contain the query IDs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6348) Unordered Receiver does not report its memory usage
[ https://issues.apache.org/jira/browse/DRILL-6348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480991#comment-16480991 ] ASF GitHub Bot commented on DRILL-6348: --- sachouche commented on issue #1237: DRILL-6348: Fixed code so that Unordered Receiver reports its memory … URL: https://github.com/apache/drill/pull/1237#issuecomment-390286281 @arina-ielchiieva I have fixed the Travis issue. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Unordered Receiver does not report its memory usage > --- > > Key: DRILL-6348 > URL: https://issues.apache.org/jira/browse/DRILL-6348 > Project: Apache Drill > Issue Type: Improvement > Components: Execution - Flow >Reporter: salim achouche >Assignee: salim achouche >Priority: Major > Fix For: 1.14.0 > > > The Drill Profile functionality doesn't show any memory usage for the > Unordered Receiver operator. This is problematic when analyzing OOM > conditions since we cannot account for all of a query memory usage. This Jira > is to fix memory reporting for the Unordered Receiver operator. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-5305) Query Profile must display Query ID
[
https://issues.apache.org/jira/browse/DRILL-5305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480930#comment-16480930
]
ASF GitHub Bot commented on DRILL-5305:
---
kkhatua commented on a change in pull request #1265: DRILL-5305: Query Profile
must display Query ID
URL: https://github.com/apache/drill/pull/1265#discussion_r189334169
##
File path: exec/java-exec/src/main/resources/rest/profile/profile.ftl
##
@@ -165,7 +165,7 @@ table.sortable thead .sorting_desc { background-image:
url("/static/img/black-de
<#assign queued = queueName != "" && queueName != "-" />
- Query Profile
+ Query Profile : ${model.getQueryId()}
Review comment:
Yes. The backing ProfileWrapper returns the queryID as a text.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Query Profile must display Query ID
> ---
>
> Key: DRILL-5305
> URL: https://issues.apache.org/jira/browse/DRILL-5305
> Project: Apache Drill
> Issue Type: Improvement
> Components: Client - HTTP, Web Server
>Affects Versions: 1.10.0
>Reporter: Abhishek Girish
>Assignee: Kunal Khatua
>Priority: Minor
> Labels: newbie
> Fix For: 1.14.0
>
>
> It would be good to have the query ID, profile ID listed as part of the
> profile itself. Currently there is no easy way to map a profile with the
> query.
> I understand that the profile ID in the URL can be used, but often when we
> have profile dumps shared for analysis, they may not contain the query IDs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-5305) Query Profile must display Query ID
[
https://issues.apache.org/jira/browse/DRILL-5305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480929#comment-16480929
]
ASF GitHub Bot commented on DRILL-5305:
---
kkhatua commented on a change in pull request #1265: DRILL-5305: Query Profile
must display Query ID
URL: https://github.com/apache/drill/pull/1265#discussion_r189333907
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/profile/ProfileWrapper.java
##
@@ -63,7 +63,7 @@
public ProfileWrapper(final QueryProfile profile, DrillConfig drillConfig) {
this.profile = profile;
-this.id = QueryIdHelper.getQueryId(profile.getId());
+this.id = profile.getId().hasText() ? profile.getId().getText() :
QueryIdHelper.getQueryId(profile.getId());
Review comment:
Yes. But if the profile is of an older format, ``hasText()` will ensure that
the query ID text is decoded from the ID components: part1 & part2.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Query Profile must display Query ID
> ---
>
> Key: DRILL-5305
> URL: https://issues.apache.org/jira/browse/DRILL-5305
> Project: Apache Drill
> Issue Type: Improvement
> Components: Client - HTTP, Web Server
>Affects Versions: 1.10.0
>Reporter: Abhishek Girish
>Assignee: Kunal Khatua
>Priority: Minor
> Labels: newbie
> Fix For: 1.14.0
>
>
> It would be good to have the query ID, profile ID listed as part of the
> profile itself. Currently there is no easy way to map a profile with the
> query.
> I understand that the profile ID in the URL can be used, but often when we
> have profile dumps shared for analysis, they may not contain the query IDs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6373) Refactor the Result Set Loader to prepare for Union, List support
[
https://issues.apache.org/jira/browse/DRILL-6373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva updated DRILL-6373:
Labels: (was: ready-to-commit)
> Refactor the Result Set Loader to prepare for Union, List support
> -
>
> Key: DRILL-6373
> URL: https://issues.apache.org/jira/browse/DRILL-6373
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.13.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Fix For: 1.14.0
>
>
> As the next step in merging the "batch sizing" enhancements, refactor the
> {{ResultSetLoader}} and related classes to prepare for Union and List
> support. This fix follows the refactoring of the column accessors for the
> same purpose. Actual Union and List support is to follow in a separate PR.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-5735) UI options grouping and filtering & Metrics hints
[ https://issues.apache.org/jira/browse/DRILL-5735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Khatua updated DRILL-5735: Fix Version/s: 1.14.0 > UI options grouping and filtering & Metrics hints > - > > Key: DRILL-5735 > URL: https://issues.apache.org/jira/browse/DRILL-5735 > Project: Apache Drill > Issue Type: Improvement > Components: Web Server >Affects Versions: 1.9.0, 1.10.0, 1.11.0 >Reporter: Muhammad Gelbana >Assignee: Kunal Khatua >Priority: Major > Fix For: 1.14.0 > > > I'm thinking of some UI improvements that could make all the difference for > users trying to optimize low-performing queries. > h2. Options > h3. Grouping > We can organize the options to be grouped by their scope of effect, this will > help users easily locate the options they may need to tune. > h3. Filtering > Since the options are a lot, we can add a filtering mechanism (i.e. string > search or group\scope filtering) so the user can filter out the options he's > not interested in. To provide more benefit than the grouping idea mentioned > above, filtering may include keywords also and not just the option name, > since the user may not be aware of the name of the option he's looking for. > h2. Metrics > I'm referring here to the metrics page and the query execution plan page that > displays the overview section and major\minor fragments metrics. We can show > hints for each metric such as: > # What does it represent in more details. > # What option\scope-of-options to tune (increase ? decrease ?) to improve the > performance reported by this metric. > # May be even provide a small dialog to quickly allow the modification of the > related option(s) to that metric -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (DRILL-5735) UI options grouping and filtering & Metrics hints
[ https://issues.apache.org/jira/browse/DRILL-5735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Khatua reassigned DRILL-5735: --- Assignee: Kunal Khatua > UI options grouping and filtering & Metrics hints > - > > Key: DRILL-5735 > URL: https://issues.apache.org/jira/browse/DRILL-5735 > Project: Apache Drill > Issue Type: Improvement > Components: Web Server >Affects Versions: 1.9.0, 1.10.0, 1.11.0 >Reporter: Muhammad Gelbana >Assignee: Kunal Khatua >Priority: Major > Fix For: 1.14.0 > > > I'm thinking of some UI improvements that could make all the difference for > users trying to optimize low-performing queries. > h2. Options > h3. Grouping > We can organize the options to be grouped by their scope of effect, this will > help users easily locate the options they may need to tune. > h3. Filtering > Since the options are a lot, we can add a filtering mechanism (i.e. string > search or group\scope filtering) so the user can filter out the options he's > not interested in. To provide more benefit than the grouping idea mentioned > above, filtering may include keywords also and not just the option name, > since the user may not be aware of the name of the option he's looking for. > h2. Metrics > I'm referring here to the metrics page and the query execution plan page that > displays the overview section and major\minor fragments metrics. We can show > hints for each metric such as: > # What does it represent in more details. > # What option\scope-of-options to tune (increase ? decrease ?) to improve the > performance reported by this metric. > # May be even provide a small dialog to quickly allow the modification of the > related option(s) to that metric -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6425) Upgrade org.ojai:ojai version
[
https://issues.apache.org/jira/browse/DRILL-6425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480888#comment-16480888
]
ASF GitHub Bot commented on DRILL-6425:
---
parthchandra commented on a change in pull request #1270: DRILL-6425: Upgrade
org.ojai:ojai version
URL: https://github.com/apache/drill/pull/1270#discussion_r189328526
##
File path: pom.xml
##
@@ -52,7 +52,7 @@
2.7.9
2.7.9.1
5.2.1-mapr
-1.1
+2.0.1-mapr-1804
Review comment:
maven repository has a ojai:2.0 and a ojai:2.0-mapr-1710. I don't see the
2.0.1 version. That apart, what is the difference between these two and why
should Drill prefer one over the other?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Upgrade org.ojai:ojai version
> -
>
> Key: DRILL-6425
> URL: https://issues.apache.org/jira/browse/DRILL-6425
> Project: Apache Drill
> Issue Type: Task
>Reporter: Vlad Rozov
>Assignee: Vlad Rozov
>Priority: Major
> Fix For: 1.14.0
>
>
> Upgrade from {{1.1}} to {{2.0.1-mapr-1804}} or the most recent.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6425) Upgrade org.ojai:ojai version
[
https://issues.apache.org/jira/browse/DRILL-6425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva updated DRILL-6425:
Labels: (was: ready-to-commit)
> Upgrade org.ojai:ojai version
> -
>
> Key: DRILL-6425
> URL: https://issues.apache.org/jira/browse/DRILL-6425
> Project: Apache Drill
> Issue Type: Task
>Reporter: Vlad Rozov
>Assignee: Vlad Rozov
>Priority: Major
> Fix For: 1.14.0
>
>
> Upgrade from {{1.1}} to {{2.0.1-mapr-1804}} or the most recent.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6425) Upgrade org.ojai:ojai version
[
https://issues.apache.org/jira/browse/DRILL-6425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480831#comment-16480831
]
ASF GitHub Bot commented on DRILL-6425:
---
arina-ielchiieva commented on issue #1270: DRILL-6425: Upgrade org.ojai:ojai
version
URL: https://github.com/apache/drill/pull/1270#issuecomment-390252156
@vrozov looks like build fails with mapr profile, please check.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Upgrade org.ojai:ojai version
> -
>
> Key: DRILL-6425
> URL: https://issues.apache.org/jira/browse/DRILL-6425
> Project: Apache Drill
> Issue Type: Task
>Reporter: Vlad Rozov
>Assignee: Vlad Rozov
>Priority: Major
> Fix For: 1.14.0
>
>
> Upgrade from {{1.1}} to {{2.0.1-mapr-1804}} or the most recent.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6425) Upgrade org.ojai:ojai version
[
https://issues.apache.org/jira/browse/DRILL-6425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva updated DRILL-6425:
Fix Version/s: 1.14.0
> Upgrade org.ojai:ojai version
> -
>
> Key: DRILL-6425
> URL: https://issues.apache.org/jira/browse/DRILL-6425
> Project: Apache Drill
> Issue Type: Task
>Reporter: Vlad Rozov
>Assignee: Vlad Rozov
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> Upgrade from {{1.1}} to {{2.0.1-mapr-1804}} or the most recent.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6425) Upgrade org.ojai:ojai version
[
https://issues.apache.org/jira/browse/DRILL-6425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480776#comment-16480776
]
ASF GitHub Bot commented on DRILL-6425:
---
arina-ielchiieva commented on issue #1270: DRILL-6425: Upgrade org.ojai:ojai
version
URL: https://github.com/apache/drill/pull/1270#issuecomment-390235753
+1
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Upgrade org.ojai:ojai version
> -
>
> Key: DRILL-6425
> URL: https://issues.apache.org/jira/browse/DRILL-6425
> Project: Apache Drill
> Issue Type: Task
>Reporter: Vlad Rozov
>Assignee: Vlad Rozov
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> Upgrade from {{1.1}} to {{2.0.1-mapr-1804}} or the most recent.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6424) Updating FasterXML Jackson libraries
[ https://issues.apache.org/jira/browse/DRILL-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480775#comment-16480775 ] ASF GitHub Bot commented on DRILL-6424: --- arina-ielchiieva commented on issue #1274: DRILL-6424: Updating FasterXML Jackson libraries URL: https://github.com/apache/drill/pull/1274#issuecomment-390235591 Looks good, thanks for the upgrade. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Updating FasterXML Jackson libraries > > > Key: DRILL-6424 > URL: https://issues.apache.org/jira/browse/DRILL-6424 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Hive, Tools, Build Test >Affects Versions: 1.13.0 >Reporter: Vitalii Diravka >Assignee: Vitalii Diravka >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > Drill uses jackson 2.7.9 and jackson-databind 2.7.9.1 versions. > There are newer versions of theses libraries, which are more stable and > involves different improvements. > hive-exec-shaded for mapr profile leverages even older 2.4.2 jackson-databind > library. It can be updated too. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6424) Updating FasterXML Jackson libraries
[ https://issues.apache.org/jira/browse/DRILL-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva updated DRILL-6424: Labels: ready-to-commit (was: ) > Updating FasterXML Jackson libraries > > > Key: DRILL-6424 > URL: https://issues.apache.org/jira/browse/DRILL-6424 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Hive, Tools, Build Test >Affects Versions: 1.13.0 >Reporter: Vitalii Diravka >Assignee: Vitalii Diravka >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > Drill uses jackson 2.7.9 and jackson-databind 2.7.9.1 versions. > There are newer versions of theses libraries, which are more stable and > involves different improvements. > hive-exec-shaded for mapr profile leverages even older 2.4.2 jackson-databind > library. It can be updated too. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6424) Updating FasterXML Jackson libraries
[ https://issues.apache.org/jira/browse/DRILL-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480761#comment-16480761 ] ASF GitHub Bot commented on DRILL-6424: --- vdiravka commented on issue #1274: DRILL-6424: Updating FasterXML Jackson libraries URL: https://github.com/apache/drill/pull/1274#issuecomment-390232028 @arina-ielchiieva It is necessary to resolve conflicts there with a new Jackson API. But this class is deprecated and nowhere used. I have tried to submit old logical plan with `sequence` and it was failed, since the LogicalPlan has newer structure to be deserialized by Jackson. Therefore no need to keep this class anymore. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Updating FasterXML Jackson libraries > > > Key: DRILL-6424 > URL: https://issues.apache.org/jira/browse/DRILL-6424 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Hive, Tools, Build Test >Affects Versions: 1.13.0 >Reporter: Vitalii Diravka >Assignee: Vitalii Diravka >Priority: Major > Fix For: 1.14.0 > > > Drill uses jackson 2.7.9 and jackson-databind 2.7.9.1 versions. > There are newer versions of theses libraries, which are more stable and > involves different improvements. > hive-exec-shaded for mapr profile leverages even older 2.4.2 jackson-databind > library. It can be updated too. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-4525) Query with BETWEEN clause on Date and Timestamp values fails with Validation Error
[
https://issues.apache.org/jira/browse/DRILL-4525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi updated DRILL-4525:
---
Labels: ready-to-commit (was: )
> Query with BETWEEN clause on Date and Timestamp values fails with Validation
> Error
> --
>
> Key: DRILL-4525
> URL: https://issues.apache.org/jira/browse/DRILL-4525
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning Optimization
>Reporter: Abhishek Girish
>Assignee: Bohdan Kazydub
>Priority: Critical
> Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> Query: (simplified variant of TPC-DS Query37)
> {code}
> SELECT
>*
> FROM
>date_dim
> WHERE
>d_date BETWEEN Cast('1999-03-06' AS DATE) AND (
> Cast('1999-03-06' AS DATE) + INTERVAL '60' day)
> LIMIT 10;
> {code}
> Error:
> {code}
> Error: VALIDATION ERROR: From line 6, column 8 to line 7, column 64: Cannot
> apply 'BETWEEN ASYMMETRIC' to arguments of type ' BETWEEN ASYMMETRIC
> AND '. Supported form(s): ' BETWEEN
> AND '
> SQL Query null
> [Error Id: 223fb37c-f561-4a37-9283-871dc6f4d6d0 on abhi2:31010]
> (state=,code=0)
> {code}
> This is a regression from 1.6.0.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-4525) Query with BETWEEN clause on Date and Timestamp values fails with Validation Error
[
https://issues.apache.org/jira/browse/DRILL-4525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480759#comment-16480759
]
ASF GitHub Bot commented on DRILL-4525:
---
vvysotskyi commented on a change in pull request #1268: DRILL-4525: Allow
SqlBetweenOperator to accept LOWER_OPERAND and UPPER_OPERAND with different
types
URL: https://github.com/apache/drill/pull/1268#discussion_r189292990
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DrillOperatorTable.java
##
@@ -158,26 +159,39 @@ private void
populateFromWithoutTypeInference(SqlIdentifier opName, SqlFunctionC
// Get the list of SqlOperator's with the given name.
public List getSqlOperator(String name) {
-if(isInferenceEnabled()) {
+if (isInferenceEnabled()) {
return drillOperatorsWithInferenceMap.get(name.toLowerCase());
} else {
return drillOperatorsWithoutInferenceMap.get(name.toLowerCase());
}
}
private void populateWrappedCalciteOperators() {
-for(SqlOperator calciteOperator : inner.getOperatorList()) {
+for (SqlOperator calciteOperator : inner.getOperatorList()) {
final SqlOperator wrapper;
- if(calciteOperator instanceof SqlAggFunction) {
+ if (calciteOperator instanceof SqlAggFunction) {
wrapper = new DrillCalciteSqlAggFunctionWrapper((SqlAggFunction)
calciteOperator,
getFunctionListWithInference(calciteOperator.getName()));
- } else if(calciteOperator instanceof SqlFunction) {
+ } else if (calciteOperator instanceof SqlFunction) {
wrapper = new DrillCalciteSqlFunctionWrapper((SqlFunction)
calciteOperator,
getFunctionListWithInference(calciteOperator.getName()));
+ } else if (calciteOperator instanceof SqlBetweenOperator) {
+// During the procedure of converting to RexNode,
+// StandardConvertletTable.convertBetween expects the SqlOperator to
be a subclass of SqlBetweenOperator
+final SqlBetweenOperator sqlBetweenOperator = (SqlBetweenOperator)
calciteOperator;
+wrapper = new
DrillCalciteSqlBetweenOperatorWrapper(sqlBetweenOperator);
} else {
-final String drillOpName =
FunctionCallFactory.replaceOpWithFuncName(calciteOperator.getName());
+final String drillOpName;
+// For UNARY_MINUS (-) or UNARY_PLUS (+), we do not rename them as
function_add or function_subtract.
Review comment:
Thanks for the explanation.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Query with BETWEEN clause on Date and Timestamp values fails with Validation
> Error
> --
>
> Key: DRILL-4525
> URL: https://issues.apache.org/jira/browse/DRILL-4525
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning Optimization
>Reporter: Abhishek Girish
>Assignee: Bohdan Kazydub
>Priority: Critical
> Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> Query: (simplified variant of TPC-DS Query37)
> {code}
> SELECT
>*
> FROM
>date_dim
> WHERE
>d_date BETWEEN Cast('1999-03-06' AS DATE) AND (
> Cast('1999-03-06' AS DATE) + INTERVAL '60' day)
> LIMIT 10;
> {code}
> Error:
> {code}
> Error: VALIDATION ERROR: From line 6, column 8 to line 7, column 64: Cannot
> apply 'BETWEEN ASYMMETRIC' to arguments of type ' BETWEEN ASYMMETRIC
> AND '. Supported form(s): ' BETWEEN
> AND '
> SQL Query null
> [Error Id: 223fb37c-f561-4a37-9283-871dc6f4d6d0 on abhi2:31010]
> (state=,code=0)
> {code}
> This is a regression from 1.6.0.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6424) Updating FasterXML Jackson libraries
[ https://issues.apache.org/jira/browse/DRILL-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480758#comment-16480758 ] ASF GitHub Bot commented on DRILL-6424: --- arina-ielchiieva commented on issue #1274: DRILL-6424: Updating FasterXML Jackson libraries URL: https://github.com/apache/drill/pull/1274#issuecomment-390230307 @vdiravka why did you removed sequence class? How it affected lib version upgrade? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Updating FasterXML Jackson libraries > > > Key: DRILL-6424 > URL: https://issues.apache.org/jira/browse/DRILL-6424 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Hive, Tools, Build Test >Affects Versions: 1.13.0 >Reporter: Vitalii Diravka >Assignee: Vitalii Diravka >Priority: Major > Fix For: 1.14.0 > > > Drill uses jackson 2.7.9 and jackson-databind 2.7.9.1 versions. > There are newer versions of theses libraries, which are more stable and > involves different improvements. > hive-exec-shaded for mapr profile leverages even older 2.4.2 jackson-databind > library. It can be updated too. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6424) Updating FasterXML Jackson libraries
[ https://issues.apache.org/jira/browse/DRILL-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480757#comment-16480757 ] ASF GitHub Bot commented on DRILL-6424: --- arina-ielchiieva commented on issue #1274: DRILL-6424: Updating FasterXML Jackson libraries URL: https://github.com/apache/drill/pull/1274#issuecomment-390230307 @vdiravka why you need to remove sequence class, how it affected lib version upgrade? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Updating FasterXML Jackson libraries > > > Key: DRILL-6424 > URL: https://issues.apache.org/jira/browse/DRILL-6424 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Hive, Tools, Build Test >Affects Versions: 1.13.0 >Reporter: Vitalii Diravka >Assignee: Vitalii Diravka >Priority: Major > Fix For: 1.14.0 > > > Drill uses jackson 2.7.9 and jackson-databind 2.7.9.1 versions. > There are newer versions of theses libraries, which are more stable and > involves different improvements. > hive-exec-shaded for mapr profile leverages even older 2.4.2 jackson-databind > library. It can be updated too. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6424) Updating FasterXML Jackson libraries
[ https://issues.apache.org/jira/browse/DRILL-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva updated DRILL-6424: Reviewer: Arina Ielchiieva > Updating FasterXML Jackson libraries > > > Key: DRILL-6424 > URL: https://issues.apache.org/jira/browse/DRILL-6424 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Hive, Tools, Build Test >Affects Versions: 1.13.0 >Reporter: Vitalii Diravka >Assignee: Vitalii Diravka >Priority: Major > Fix For: 1.14.0 > > > Drill uses jackson 2.7.9 and jackson-databind 2.7.9.1 versions. > There are newer versions of theses libraries, which are more stable and > involves different improvements. > hive-exec-shaded for mapr profile leverages even older 2.4.2 jackson-databind > library. It can be updated too. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-4525) Query with BETWEEN clause on Date and Timestamp values fails with Validation Error
[
https://issues.apache.org/jira/browse/DRILL-4525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480737#comment-16480737
]
ASF GitHub Bot commented on DRILL-4525:
---
KazydubB commented on a change in pull request #1268: DRILL-4525: Allow
SqlBetweenOperator to accept LOWER_OPERAND and UPPER_OPERAND with different
types
URL: https://github.com/apache/drill/pull/1268#discussion_r189289783
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DrillOperatorTable.java
##
@@ -43,7 +44,7 @@
* {@link #inner SqlStdOperatorTable}, and Drill User Defined Functions.
*/
public class DrillOperatorTable extends SqlStdOperatorTable {
-// private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(DrillOperatorTable.class);
+ // private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(DrillOperatorTable.class);
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Query with BETWEEN clause on Date and Timestamp values fails with Validation
> Error
> --
>
> Key: DRILL-4525
> URL: https://issues.apache.org/jira/browse/DRILL-4525
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning Optimization
>Reporter: Abhishek Girish
>Assignee: Bohdan Kazydub
>Priority: Critical
> Fix For: 1.14.0
>
>
> Query: (simplified variant of TPC-DS Query37)
> {code}
> SELECT
>*
> FROM
>date_dim
> WHERE
>d_date BETWEEN Cast('1999-03-06' AS DATE) AND (
> Cast('1999-03-06' AS DATE) + INTERVAL '60' day)
> LIMIT 10;
> {code}
> Error:
> {code}
> Error: VALIDATION ERROR: From line 6, column 8 to line 7, column 64: Cannot
> apply 'BETWEEN ASYMMETRIC' to arguments of type ' BETWEEN ASYMMETRIC
> AND '. Supported form(s): ' BETWEEN
> AND '
> SQL Query null
> [Error Id: 223fb37c-f561-4a37-9283-871dc6f4d6d0 on abhi2:31010]
> (state=,code=0)
> {code}
> This is a regression from 1.6.0.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-4525) Query with BETWEEN clause on Date and Timestamp values fails with Validation Error
[
https://issues.apache.org/jira/browse/DRILL-4525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480725#comment-16480725
]
ASF GitHub Bot commented on DRILL-4525:
---
KazydubB commented on a change in pull request #1268: DRILL-4525: Allow
SqlBetweenOperator to accept LOWER_OPERAND and UPPER_OPERAND with different
types
URL: https://github.com/apache/drill/pull/1268#discussion_r189286094
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DrillOperatorTable.java
##
@@ -158,26 +159,39 @@ private void
populateFromWithoutTypeInference(SqlIdentifier opName, SqlFunctionC
// Get the list of SqlOperator's with the given name.
public List getSqlOperator(String name) {
-if(isInferenceEnabled()) {
+if (isInferenceEnabled()) {
return drillOperatorsWithInferenceMap.get(name.toLowerCase());
} else {
return drillOperatorsWithoutInferenceMap.get(name.toLowerCase());
}
}
private void populateWrappedCalciteOperators() {
-for(SqlOperator calciteOperator : inner.getOperatorList()) {
+for (SqlOperator calciteOperator : inner.getOperatorList()) {
final SqlOperator wrapper;
- if(calciteOperator instanceof SqlAggFunction) {
+ if (calciteOperator instanceof SqlAggFunction) {
wrapper = new DrillCalciteSqlAggFunctionWrapper((SqlAggFunction)
calciteOperator,
getFunctionListWithInference(calciteOperator.getName()));
- } else if(calciteOperator instanceof SqlFunction) {
+ } else if (calciteOperator instanceof SqlFunction) {
wrapper = new DrillCalciteSqlFunctionWrapper((SqlFunction)
calciteOperator,
getFunctionListWithInference(calciteOperator.getName()));
+ } else if (calciteOperator instanceof SqlBetweenOperator) {
+// During the procedure of converting to RexNode,
+// StandardConvertletTable.convertBetween expects the SqlOperator to
be a subclass of SqlBetweenOperator
+final SqlBetweenOperator sqlBetweenOperator = (SqlBetweenOperator)
calciteOperator;
+wrapper = new
DrillCalciteSqlBetweenOperatorWrapper(sqlBetweenOperator);
} else {
-final String drillOpName =
FunctionCallFactory.replaceOpWithFuncName(calciteOperator.getName());
+final String drillOpName;
+// For UNARY_MINUS (-) or UNARY_PLUS (+), we do not rename them as
function_add or function_subtract.
Review comment:
If left unchanged, for UNARY_MINUS and UNARY_PLUS operators Calcite's
behaviour will be applied. The wrapper, on the other hand, ensures that
customized behaviour will be adopted.
(These two operators are treated differently because as described in the
comment found in file their drillOpName collides with binary MINUS and PLUS
respectively.)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Query with BETWEEN clause on Date and Timestamp values fails with Validation
> Error
> --
>
> Key: DRILL-4525
> URL: https://issues.apache.org/jira/browse/DRILL-4525
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning Optimization
>Reporter: Abhishek Girish
>Assignee: Bohdan Kazydub
>Priority: Critical
> Fix For: 1.14.0
>
>
> Query: (simplified variant of TPC-DS Query37)
> {code}
> SELECT
>*
> FROM
>date_dim
> WHERE
>d_date BETWEEN Cast('1999-03-06' AS DATE) AND (
> Cast('1999-03-06' AS DATE) + INTERVAL '60' day)
> LIMIT 10;
> {code}
> Error:
> {code}
> Error: VALIDATION ERROR: From line 6, column 8 to line 7, column 64: Cannot
> apply 'BETWEEN ASYMMETRIC' to arguments of type ' BETWEEN ASYMMETRIC
> AND '. Supported form(s): ' BETWEEN
> AND '
> SQL Query null
> [Error Id: 223fb37c-f561-4a37-9283-871dc6f4d6d0 on abhi2:31010]
> (state=,code=0)
> {code}
> This is a regression from 1.6.0.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6321) Lateral Join: Planning changes - enable submitting physical plan
[ https://issues.apache.org/jira/browse/DRILL-6321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480714#comment-16480714 ] ASF GitHub Bot commented on DRILL-6321: --- chunhui-shi opened a new pull request #1275: DRILL-6321: Lateral Join and Unnest - star column prefix URL: https://github.com/apache/drill/pull/1275 * Add prefix to unnest's star column * add tests for output column verification This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Lateral Join: Planning changes - enable submitting physical plan > > > Key: DRILL-6321 > URL: https://issues.apache.org/jira/browse/DRILL-6321 > Project: Apache Drill > Issue Type: Task >Reporter: Parth Chandra >Assignee: Chunhui Shi >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > Implement changes to enable submitting a physical plan containing lateral and > unnest. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6424) Updating FasterXML Jackson libraries
[ https://issues.apache.org/jira/browse/DRILL-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480695#comment-16480695 ] ASF GitHub Bot commented on DRILL-6424: --- vdiravka opened a new pull request #1274: DRILL-6424: Updating FasterXML Jackson libraries URL: https://github.com/apache/drill/pull/1274 * all jackson libraries are updated to the latest version * using of old jackson library by hive-exec-shaded for mapr profile is removed * redundant Sequence class is removed This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Updating FasterXML Jackson libraries > > > Key: DRILL-6424 > URL: https://issues.apache.org/jira/browse/DRILL-6424 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Hive, Tools, Build Test >Affects Versions: 1.13.0 >Reporter: Vitalii Diravka >Assignee: Vitalii Diravka >Priority: Major > Fix For: 1.14.0 > > > Drill uses jackson 2.7.9 and jackson-databind 2.7.9.1 versions. > There are newer versions of theses libraries, which are more stable and > involves different improvements. > hive-exec-shaded for mapr profile leverages even older 2.4.2 jackson-databind > library. It can be updated too. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6373) Refactor the Result Set Loader to prepare for Union, List support
[
https://issues.apache.org/jira/browse/DRILL-6373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480637#comment-16480637
]
ASF GitHub Bot commented on DRILL-6373:
---
arina-ielchiieva commented on issue #1244: DRILL-6373: Refactor Result Set
Loader for Union, List support
URL: https://github.com/apache/drill/pull/1244#issuecomment-390205193
@paul-rogers there are some check styles failures, please take a look.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Refactor the Result Set Loader to prepare for Union, List support
> -
>
> Key: DRILL-6373
> URL: https://issues.apache.org/jira/browse/DRILL-6373
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.13.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> As the next step in merging the "batch sizing" enhancements, refactor the
> {{ResultSetLoader}} and related classes to prepare for Union and List
> support. This fix follows the refactoring of the column accessors for the
> same purpose. Actual Union and List support is to follow in a separate PR.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6423) Export query result as a CSV file
[
https://issues.apache.org/jira/browse/DRILL-6423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480551#comment-16480551
]
ASF GitHub Bot commented on DRILL-6423:
---
arina-ielchiieva commented on a change in pull request #1266: DRILL-6423:
Export query result as a CSV file
URL: https://github.com/apache/drill/pull/1266#discussion_r189245505
##
File path: exec/java-exec/src/main/resources/rest/query/result.ftl
##
@@ -59,11 +75,67 @@
$('#result').dataTable( {
"aaSorting": [],
"scrollX" : true,
+"lengthMenu": [[10, 25, 50, 100, -1], [10, 25, 50, 100, "All"]],
+"lengthChange": true,
"dom": '<"H"lCfr>t<"F"ip>',
"jQueryUI" : true
} );
} );
-
+
+//Pop out profile (needed to avoid losing query results)
+function popOutProfile(queryId) {
+ var profileUrl = location.protocol+'//'+
location.host+'/profiles/'+queryId;
+ var tgtWindow = '_blank';
+ window.open(profileUrl, tgtWindow);
+}
+
+//Ref: https://jsfiddle.net/gengns/j1jm2tjx/
+function download_csv(csvRecords, filename) {
Review comment:
`downloadCsv`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Export query result as a CSV file
> -
>
> Key: DRILL-6423
> URL: https://issues.apache.org/jira/browse/DRILL-6423
> Project: Apache Drill
> Issue Type: New Feature
> Components: Web Server
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Major
> Fix For: 1.14.0
>
>
> This feature request is based on a question posted in the user mailing list:
> [is there any way to download the data through Drill Web
> UI?|https://lists.apache.org/thread.html/bd6b2453c83d818b618ec6334891309f42556d4a7de34d84024a43cd@]
> Since the results of the WebUI allows for sorting and filtering as well, it
> can be very useful if there is a way for a user to download the results of an
> executed query for further processing in other tools (like Excel)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6423) Export query result as a CSV file
[
https://issues.apache.org/jira/browse/DRILL-6423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480552#comment-16480552
]
ASF GitHub Bot commented on DRILL-6423:
---
arina-ielchiieva commented on a change in pull request #1266: DRILL-6423:
Export query result as a CSV file
URL: https://github.com/apache/drill/pull/1266#discussion_r189245434
##
File path: exec/java-exec/src/main/resources/rest/query/result.ftl
##
@@ -59,11 +75,67 @@
$('#result').dataTable( {
"aaSorting": [],
"scrollX" : true,
+"lengthMenu": [[10, 25, 50, 100, -1], [10, 25, 50, 100, "All"]],
+"lengthChange": true,
"dom": '<"H"lCfr>t<"F"ip>',
"jQueryUI" : true
} );
} );
-
+
+//Pop out profile (needed to avoid losing query results)
+function popOutProfile(queryId) {
+ var profileUrl = location.protocol+'//'+
location.host+'/profiles/'+queryId;
+ var tgtWindow = '_blank';
+ window.open(profileUrl, tgtWindow);
+}
+
+//Ref: https://jsfiddle.net/gengns/j1jm2tjx/
+function download_csv(csvRecords, filename) {
+var csvFile;
+var downloadElem;
+
+//CSV File
+csvFile = new Blob([csvRecords], {type: "text/csv"});
+// Download link
+downloadElem = document.createElement("a");
+// File name
+downloadElem.download = filename;
+
+// We have to create a link to the file
+downloadElem.href = window.URL.createObjectURL(csvFile);
+
+// Make sure that the link is not displayed
+downloadElem.style.display = "none";
+
+// Add the link to your DOM
+document.body.appendChild(downloadElem);
+
+// Launch the download prompt
+downloadElem.click();
+}
+
+function exportTableAsCsv(queryId) {
+var filename = queryId+'.csv';
Review comment:
space after `queryId`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Export query result as a CSV file
> -
>
> Key: DRILL-6423
> URL: https://issues.apache.org/jira/browse/DRILL-6423
> Project: Apache Drill
> Issue Type: New Feature
> Components: Web Server
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Major
> Fix For: 1.14.0
>
>
> This feature request is based on a question posted in the user mailing list:
> [is there any way to download the data through Drill Web
> UI?|https://lists.apache.org/thread.html/bd6b2453c83d818b618ec6334891309f42556d4a7de34d84024a43cd@]
> Since the results of the WebUI allows for sorting and filtering as well, it
> can be very useful if there is a way for a user to download the results of an
> executed query for further processing in other tools (like Excel)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6423) Export query result as a CSV file
[
https://issues.apache.org/jira/browse/DRILL-6423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480553#comment-16480553
]
ASF GitHub Bot commented on DRILL-6423:
---
arina-ielchiieva commented on a change in pull request #1266: DRILL-6423:
Export query result as a CSV file
URL: https://github.com/apache/drill/pull/1266#discussion_r189245944
##
File path: exec/java-exec/src/main/resources/rest/query/result.ftl
##
@@ -30,6 +30,22 @@
back
+
+
+
+
+ Query Profile: ${model.getQueryId()}
+
+
+ Delimiter
+
+
+
Review comment:
Maybe we can add tooltip that explains that we export only visible table? If
user needs to export all results, then should chose all rows to display?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Export query result as a CSV file
> -
>
> Key: DRILL-6423
> URL: https://issues.apache.org/jira/browse/DRILL-6423
> Project: Apache Drill
> Issue Type: New Feature
> Components: Web Server
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Major
> Fix For: 1.14.0
>
>
> This feature request is based on a question posted in the user mailing list:
> [is there any way to download the data through Drill Web
> UI?|https://lists.apache.org/thread.html/bd6b2453c83d818b618ec6334891309f42556d4a7de34d84024a43cd@]
> Since the results of the WebUI allows for sorting and filtering as well, it
> can be very useful if there is a way for a user to download the results of an
> executed query for further processing in other tools (like Excel)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6426) Refactor TestFunctionsWithTypeExpoQueries test to be independent on limit 0 optimization option
[ https://issues.apache.org/jira/browse/DRILL-6426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Volodymyr Vysotskyi updated DRILL-6426: --- Labels: ready-to-commit (was: ) > Refactor TestFunctionsWithTypeExpoQueries test to be independent on limit 0 > optimization option > --- > > Key: DRILL-6426 > URL: https://issues.apache.org/jira/browse/DRILL-6426 > Project: Apache Drill > Issue Type: Task >Reporter: Arina Ielchiieva >Assignee: Arina Ielchiieva >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > TestFunctionsWithTypeExpoQueries depends on limit 0 optimization option. > Currently by default it is off and tests pass. If option by default is on, > tests fail. Need to refactor tests to ensure they pass disregarding on > default option value. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6426) Refactor TestFunctionsWithTypeExpoQueries test to be independent on limit 0 optimization option
[ https://issues.apache.org/jira/browse/DRILL-6426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480540#comment-16480540 ] ASF GitHub Bot commented on DRILL-6426: --- vvysotskyi commented on issue #1273: DRILL-6426: Refactor TestFunctionsWithTypeExpoQueries test to be independent on limit 0 optimization option URL: https://github.com/apache/drill/pull/1273#issuecomment-390184220 It is fine to make these tests independent of the default value of limit 0 optimization option. LGTM, +1 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Refactor TestFunctionsWithTypeExpoQueries test to be independent on limit 0 > optimization option > --- > > Key: DRILL-6426 > URL: https://issues.apache.org/jira/browse/DRILL-6426 > Project: Apache Drill > Issue Type: Task >Reporter: Arina Ielchiieva >Assignee: Arina Ielchiieva >Priority: Major > Fix For: 1.14.0 > > > TestFunctionsWithTypeExpoQueries depends on limit 0 optimization option. > Currently by default it is off and tests pass. If option by default is on, > tests fail. Need to refactor tests to ensure they pass disregarding on > default option value. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6415) Unit test TestGracefulShutdown.testRestApiShutdown times out
[
https://issues.apache.org/jira/browse/DRILL-6415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480542#comment-16480542
]
ASF GitHub Bot commented on DRILL-6415:
---
arina-ielchiieva commented on issue #1263: DRILL-6415: Fixed
TestGracefulShutdown.TestRestApi test from timing out
URL: https://github.com/apache/drill/pull/1263#issuecomment-390184317
@dvjyothsna could you please address code review comments? The faster we fix
failing test, the better :)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Unit test TestGracefulShutdown.testRestApiShutdown times out
>
>
> Key: DRILL-6415
> URL: https://issues.apache.org/jira/browse/DRILL-6415
> Project: Apache Drill
> Issue Type: Bug
> Components: Tools, Build Test
>Reporter: Abhishek Girish
>Assignee: Venkata Jyothsna Donapati
>Priority: Major
> Fix For: 1.14.0
>
>
> {code}
> 16:03:40.415 [main] ERROR org.apache.drill.TestReporter - Test Failed (d:
> -18.3 KiB(72.9 KiB), h: -335.3 MiB(1.3 GiB), nh: 1.1 MiB(335.9 MiB)):
> testRestApiShutdown(org.apache.drill.test.TestGracefulShutdown)
> org.junit.runners.model.TestTimedOutException: test timed out after 18
> milliseconds
> at sun.misc.Unsafe.park(Native Method) ~[na:1.8.0_161]
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> ~[na:1.8.0_161]
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUninterruptibly(AbstractQueuedSynchronizer.java:1976)
> ~[na:1.8.0_161]
> at
> org.apache.drill.exec.work.WorkManager.waitToExit(WorkManager.java:203)
> ~[classes/:na]
> at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:242)
> ~[classes/:na]
> at
> org.apache.drill.test.ClusterFixture.safeClose(ClusterFixture.java:454)
> ~[test-classes/:1.14.0-SNAPSHOT]
> at org.apache.drill.test.ClusterFixture.close(ClusterFixture.java:405)
> ~[test-classes/:1.14.0-SNAPSHOT]
> at
> org.apache.drill.test.TestGracefulShutdown.testRestApiShutdown(TestGracefulShutdown.java:294)
> ~[test-classes/:1.14.0-SNAPSHOT]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[na:1.8.0_161]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[na:1.8.0_161]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[na:1.8.0_161]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_161]
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> ~[junit-4.12.jar:4.12]
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> ~[junit-4.12.jar:4.12]
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> ~[junit-4.12.jar:4.12]
> at
> mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.executeTestMethod(JUnit4TestRunnerDecorator.java:154)
> ~[jmockit-1.39.jar:1.39]
> at
> mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.invokeExplosively(JUnit4TestRunnerDecorator.java:70)
> ~[jmockit-1.39.jar:1.39]
> at
> mockit.integration.junit4.internal.FakeFrameworkMethod.invokeExplosively(FakeFrameworkMethod.java:34)
> ~[jmockit-1.39.jar:1.39]
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java)
> ~[junit-4.12.jar:4.12]
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> ~[junit-4.12.jar:4.12]
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> ~[junit-4.12.jar:4.12]
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> ~[junit-4.12.jar:4.12]
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> ~[junit-4.12.jar:4.12]
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> ~[junit-4.12.jar:4.12]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_161]
> at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_161]
> {code}
> {code}
> testRestApiShutdown(org.apache.drill.test.TestGracefulShutdown) Time
> elapsed: 180.028 sec <<< ERROR!
> org.junit.runners.model.TestTimedOutException: test timed out after 18
> milliseconds
> at sun.misc.Unsafe.park(Native Method)
> at
[jira] [Commented] (DRILL-5846) Improve Parquet Reader Performance for Flat Data types
[ https://issues.apache.org/jira/browse/DRILL-5846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480539#comment-16480539 ] ASF GitHub Bot commented on DRILL-5846: --- arina-ielchiieva commented on a change in pull request #1060: DRILL-5846: Improve parquet performance for Flat Data Types URL: https://github.com/apache/drill/pull/1060#discussion_r189244567 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/VarLenAbstractEntryReader.java ## @@ -0,0 +1,266 @@ +/*** Review comment: Please license headers, Travis fails. Consider other classes as well. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Improve Parquet Reader Performance for Flat Data types > --- > > Key: DRILL-5846 > URL: https://issues.apache.org/jira/browse/DRILL-5846 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Parquet >Affects Versions: 1.11.0 >Reporter: salim achouche >Assignee: salim achouche >Priority: Major > Labels: performance > Fix For: 1.14.0 > > Attachments: 2542d447-9837-3924-dd12-f759108461e5.sys.drill, > 2542d49b-88ef-38e3-a02b-b441c1295817.sys.drill > > > The Parquet Reader is a key use-case for Drill. This JIRA is an attempt to > further improve the Parquet Reader performance as several users reported that > Parquet parsing represents the lion share of the overall query execution. It > tracks Flat Data types only as Nested DTs might involve functional and > processing enhancements (e.g., a nested column can be seen as a Document; > user might want to perform operations scoped at the document level that is no > need to span all rows). Another JIRA will be created to handle the nested > columns use-case. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-5846) Improve Parquet Reader Performance for Flat Data types
[
https://issues.apache.org/jira/browse/DRILL-5846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480538#comment-16480538
]
ASF GitHub Bot commented on DRILL-5846:
---
arina-ielchiieva commented on a change in pull request #1060: DRILL-5846:
Improve parquet performance for Flat Data Types
URL: https://github.com/apache/drill/pull/1060#discussion_r189243992
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java
##
@@ -289,6 +289,10 @@ private ExecConstants() {
public static final OptionValidator COMPILE_SCALAR_REPLACEMENT = new
BooleanValidator("exec.compile.scalar_replacement");
+ // Controls whether to enable bulk parquet reader processing
+ public static final String PARQUET_FLAT_READER_BULK=
"store.parquet.flat.reader.bulk";
Review comment:
Could you please fix shift between variable name and equals? Looks like it
correspond to the general code style in Drill project. Please consider the same
in other changed classes.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Improve Parquet Reader Performance for Flat Data types
> ---
>
> Key: DRILL-5846
> URL: https://issues.apache.org/jira/browse/DRILL-5846
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - Parquet
>Affects Versions: 1.11.0
>Reporter: salim achouche
>Assignee: salim achouche
>Priority: Major
> Labels: performance
> Fix For: 1.14.0
>
> Attachments: 2542d447-9837-3924-dd12-f759108461e5.sys.drill,
> 2542d49b-88ef-38e3-a02b-b441c1295817.sys.drill
>
>
> The Parquet Reader is a key use-case for Drill. This JIRA is an attempt to
> further improve the Parquet Reader performance as several users reported that
> Parquet parsing represents the lion share of the overall query execution. It
> tracks Flat Data types only as Nested DTs might involve functional and
> processing enhancements (e.g., a nested column can be seen as a Document;
> user might want to perform operations scoped at the document level that is no
> need to span all rows). Another JIRA will be created to handle the nested
> columns use-case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-5846) Improve Parquet Reader Performance for Flat Data types
[
https://issues.apache.org/jira/browse/DRILL-5846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480537#comment-16480537
]
ASF GitHub Bot commented on DRILL-5846:
---
arina-ielchiieva commented on a change in pull request #1060: DRILL-5846:
Improve parquet performance for Flat Data Types
URL: https://github.com/apache/drill/pull/1060#discussion_r189244757
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/VarLenAbstractEntryReader.java
##
@@ -0,0 +1,266 @@
+/***
+
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+
**/
+package org.apache.drill.exec.store.parquet.columnreaders;
+
+import java.nio.ByteBuffer;
+
+import
org.apache.drill.exec.store.parquet.columnreaders.VarLenColumnBulkInput.ColumnPrecisionInfo;
+import
org.apache.drill.exec.store.parquet.columnreaders.VarLenColumnBulkInput.PageDataInfo;
+
+import io.netty.buffer.DrillBuf;
+
+/** Abstract class for sub-classes implementing several algorithms for loading
a Bulk Entry */
+abstract class VarLenAbstractEntryReader {
+
+ /** byte buffer used for buffering page data */
+ protected final ByteBuffer buffer;
+ /** Page Data Information */
+ protected final PageDataInfo pageInfo;
+ /** expected precision type: fixed or variable length */
+ protected final ColumnPrecisionInfo columnPrecInfo;
+ /** Bulk entry */
+ protected final VarLenColumnBulkEntry entry;
+
+ /**
+ * CTOR.
+ * @param _buffer byte buffer for data buffering (within CPU cache)
+ * @param _pageInfo page being processed information
+ * @param _columnPrecInfo column precision information
+ * @param _entry reusable bulk entry object
+ */
+ VarLenAbstractEntryReader(ByteBuffer _buffer,
Review comment:
Not sure naming parameters with underscore is according to Java standard.
Please consider renaming here and in other classes.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Improve Parquet Reader Performance for Flat Data types
> ---
>
> Key: DRILL-5846
> URL: https://issues.apache.org/jira/browse/DRILL-5846
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - Parquet
>Affects Versions: 1.11.0
>Reporter: salim achouche
>Assignee: salim achouche
>Priority: Major
> Labels: performance
> Fix For: 1.14.0
>
> Attachments: 2542d447-9837-3924-dd12-f759108461e5.sys.drill,
> 2542d49b-88ef-38e3-a02b-b441c1295817.sys.drill
>
>
> The Parquet Reader is a key use-case for Drill. This JIRA is an attempt to
> further improve the Parquet Reader performance as several users reported that
> Parquet parsing represents the lion share of the overall query execution. It
> tracks Flat Data types only as Nested DTs might involve functional and
> processing enhancements (e.g., a nested column can be seen as a Document;
> user might want to perform operations scoped at the document level that is no
> need to span all rows). Another JIRA will be created to handle the nested
> columns use-case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6348) Unordered Receiver does not report its memory usage
[ https://issues.apache.org/jira/browse/DRILL-6348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480511#comment-16480511 ] ASF GitHub Bot commented on DRILL-6348: --- arina-ielchiieva commented on issue #1237: DRILL-6348: Fixed code so that Unordered Receiver reports its memory … URL: https://github.com/apache/drill/pull/1237#issuecomment-390179279 @sachouche travis build fails with check style errors. Please fix. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Unordered Receiver does not report its memory usage > --- > > Key: DRILL-6348 > URL: https://issues.apache.org/jira/browse/DRILL-6348 > Project: Apache Drill > Issue Type: Improvement > Components: Execution - Flow >Reporter: salim achouche >Assignee: salim achouche >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > The Drill Profile functionality doesn't show any memory usage for the > Unordered Receiver operator. This is problematic when analyzing OOM > conditions since we cannot account for all of a query memory usage. This Jira > is to fix memory reporting for the Unordered Receiver operator. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6348) Unordered Receiver does not report its memory usage
[ https://issues.apache.org/jira/browse/DRILL-6348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva updated DRILL-6348: Issue Type: Improvement (was: Task) > Unordered Receiver does not report its memory usage > --- > > Key: DRILL-6348 > URL: https://issues.apache.org/jira/browse/DRILL-6348 > Project: Apache Drill > Issue Type: Improvement > Components: Execution - Flow >Reporter: salim achouche >Assignee: salim achouche >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > The Drill Profile functionality doesn't show any memory usage for the > Unordered Receiver operator. This is problematic when analyzing OOM > conditions since we cannot account for all of a query memory usage. This Jira > is to fix memory reporting for the Unordered Receiver operator. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6426) Refactor TestFunctionsWithTypeExpoQueries test to be independent on limit 0 optimization option
[ https://issues.apache.org/jira/browse/DRILL-6426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480459#comment-16480459 ] ASF GitHub Bot commented on DRILL-6426: --- arina-ielchiieva commented on issue #1273: DRILL-6426: Refactor TestFunctionsWithTypeExpoQueries test to be independent on limit 0 optimization option URL: https://github.com/apache/drill/pull/1273#issuecomment-390162305 @vvysotskyi please review. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Refactor TestFunctionsWithTypeExpoQueries test to be independent on limit 0 > optimization option > --- > > Key: DRILL-6426 > URL: https://issues.apache.org/jira/browse/DRILL-6426 > Project: Apache Drill > Issue Type: Task >Reporter: Arina Ielchiieva >Assignee: Arina Ielchiieva >Priority: Major > Fix For: 1.14.0 > > > TestFunctionsWithTypeExpoQueries depends on limit 0 optimization option. > Currently by default it is off and tests pass. If option by default is on, > tests fail. Need to refactor tests to ensure they pass disregarding on > default option value. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6426) Refactor TestFunctionsWithTypeExpoQueries test to be independent on limit 0 optimization option
[ https://issues.apache.org/jira/browse/DRILL-6426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480458#comment-16480458 ] ASF GitHub Bot commented on DRILL-6426: --- arina-ielchiieva opened a new pull request #1273: DRILL-6426: Refactor TestFunctionsWithTypeExpoQueries test to be independent on limit 0 optimization option URL: https://github.com/apache/drill/pull/1273 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Refactor TestFunctionsWithTypeExpoQueries test to be independent on limit 0 > optimization option > --- > > Key: DRILL-6426 > URL: https://issues.apache.org/jira/browse/DRILL-6426 > Project: Apache Drill > Issue Type: Task >Reporter: Arina Ielchiieva >Assignee: Arina Ielchiieva >Priority: Major > Fix For: 1.14.0 > > > TestFunctionsWithTypeExpoQueries depends on limit 0 optimization option. > Currently by default it is off and tests pass. If option by default is on, > tests fail. Need to refactor tests to ensure they pass disregarding on > default option value. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6413) Specific query returns an exception if filter a boolean column by "equals" operator
[
https://issues.apache.org/jira/browse/DRILL-6413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi updated DRILL-6413:
---
Labels: ready-to-commit (was: )
> Specific query returns an exception if filter a boolean column by "equals"
> operator
> ---
>
> Key: DRILL-6413
> URL: https://issues.apache.org/jira/browse/DRILL-6413
> Project: Apache Drill
> Issue Type: Bug
>Reporter: Anton Gozhiy
>Assignee: Arina Ielchiieva
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.14.0
>
> Attachments: Test_data.tar.gz
>
>
> *Data:*
> Use the attached dataset
> *Query:*
> {code:sql}
> select *
> from dfs.tmp.`Test_data`
> where bool_col = true and part_col in ('Partition_two')
> {code}
> *Expected result:*
> The query should return result normally
> *Actual result:*
> Exception happens:
> {noformat}
> Error: SYSTEM ERROR: ClassCastException:
> org.apache.drill.common.expression.TypedFieldExpr cannot be cast to
> org.apache.drill.exec.expr.stat.ParquetFilterPredicate
> {noformat}
> *Notes:*
> It works OK if use "is" operator or if not use "*" in the select statement
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6413) Specific query returns an exception if filter a boolean column by "equals" operator
[
https://issues.apache.org/jira/browse/DRILL-6413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480437#comment-16480437
]
ASF GitHub Bot commented on DRILL-6413:
---
vvysotskyi commented on issue #1269: DRILL-6413: Update
ParquetFilterBuilder.visitBooleanOperator to handle simplied boolean expression
URL: https://github.com/apache/drill/pull/1269#issuecomment-390157202
LGTM, +1
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Specific query returns an exception if filter a boolean column by "equals"
> operator
> ---
>
> Key: DRILL-6413
> URL: https://issues.apache.org/jira/browse/DRILL-6413
> Project: Apache Drill
> Issue Type: Bug
>Reporter: Anton Gozhiy
>Assignee: Arina Ielchiieva
>Priority: Major
> Fix For: 1.14.0
>
> Attachments: Test_data.tar.gz
>
>
> *Data:*
> Use the attached dataset
> *Query:*
> {code:sql}
> select *
> from dfs.tmp.`Test_data`
> where bool_col = true and part_col in ('Partition_two')
> {code}
> *Expected result:*
> The query should return result normally
> *Actual result:*
> Exception happens:
> {noformat}
> Error: SYSTEM ERROR: ClassCastException:
> org.apache.drill.common.expression.TypedFieldExpr cannot be cast to
> org.apache.drill.exec.expr.stat.ParquetFilterPredicate
> {noformat}
> *Notes:*
> It works OK if use "is" operator or if not use "*" in the select statement
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6426) Refactor TestFunctionsWithTypeExpoQueries test to be independent on limit 0 optimization option
Arina Ielchiieva created DRILL-6426: --- Summary: Refactor TestFunctionsWithTypeExpoQueries test to be independent on limit 0 optimization option Key: DRILL-6426 URL: https://issues.apache.org/jira/browse/DRILL-6426 Project: Apache Drill Issue Type: Task Reporter: Arina Ielchiieva Assignee: Arina Ielchiieva Fix For: 1.14.0 TestFunctionsWithTypeExpoQueries depends on limit 0 optimization option. Currently by default it is off and tests pass. If option by default is on, tests fail. Need to refactor tests to ensure they pass disregarding on default option value. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DRILL-4552) During inference, treat decimal literals as Double when planner.enable_decimal_data_type is off
[ https://issues.apache.org/jira/browse/DRILL-4552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva closed DRILL-4552. --- Resolution: Not A Problem > During inference, treat decimal literals as Double when > planner.enable_decimal_data_type is off > --- > > Key: DRILL-4552 > URL: https://issues.apache.org/jira/browse/DRILL-4552 > Project: Apache Drill > Issue Type: Improvement > Components: Query Planning Optimization >Reporter: Sean Hsuan-Yi Chu >Assignee: Sean Hsuan-Yi Chu >Priority: Major > > In SQL standard, decimal literals (e.g., 1.2, 2.5, etc) are Decimal type. In > Drill, when `planner.enable_decimal_data_type` is off, they are treated as > Double. > However, the current inference mechanism is "not to do any inference if the > operand is Decimal type". (To be more exact, treat them as ANY type and defer > to the execution to make the decision.) > The mechanism can be improved by treating decimal literals as Double when > `planner.enable_decimal_data_type` is off. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-5977) predicate pushdown support kafkaMsgOffset
[
https://issues.apache.org/jira/browse/DRILL-5977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480249#comment-16480249
]
ASF GitHub Bot commented on DRILL-5977:
---
aravi5 commented on issue #1272: DRILL-5977: Filter Pushdown in Drill-Kafka
plugin
URL: https://github.com/apache/drill/pull/1272#issuecomment-390079802
@akumarb2010, @kameshb, @kkhatua, @arina-ielchiieva please review the
changes for filter pushdown implementation.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> predicate pushdown support kafkaMsgOffset
> -
>
> Key: DRILL-5977
> URL: https://issues.apache.org/jira/browse/DRILL-5977
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: B Anil Kumar
>Assignee: Abhishek Ravi
>Priority: Major
> Fix For: 1.14.0
>
>
> As part of Kafka storage plugin review, below is the suggestion from Paul.
> {noformat}
> Does it make sense to provide a way to select a range of messages: a starting
> point or a count? Perhaps I want to run my query every five minutes, scanning
> only those messages since the previous scan. Or, I want to limit my take to,
> say, the next 1000 messages. Could we use a pseudo-column such as
> "kafkaMsgOffset" for that purpose? Maybe
> SELECT * FROM WHERE kafkaMsgOffset > 12345
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
