[jira] [Created] (DRILL-6999) Queries failed with "Failed to aggregate or route the RFW" due to "java.lang.ArrayIndexOutOfBoundsException"
Sorabh Hamirwasia created DRILL-6999:
Summary: Queries failed with "Failed to aggregate or route the
RFW" due to "java.lang.ArrayIndexOutOfBoundsException"
Key: DRILL-6999
URL: https://issues.apache.org/jira/browse/DRILL-6999
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.16.0
Reporter: Sorabh Hamirwasia
Assignee: weijie.tong
Fix For: 1.16.0
Following query joins 2 tables on *two* (>1) fields.
{noformat}
select count(*) from lineitem l inner join partsupp p on l.l_partkey =
p.ps_partkey AND l.l_suppkey = p.ps_suppkey
{noformat}
This is because AsyncAggregateWorker exits due to the following exception when
multiple fields are used in the join condition
{code:java}
2019-01-22 16:01:18,773 [drill-executor-1301] ERROR
o.a.d.e.w.filter.RuntimeFilterSink - Failed to aggregate or route the RFW
java.lang.ArrayIndexOutOfBoundsException: 1
at
org.apache.drill.exec.work.filter.RuntimeFilterWritable.unwrap(RuntimeFilterWritable.java:67)
~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
at
org.apache.drill.exec.work.filter.RuntimeFilterWritable.aggregate(RuntimeFilterWritable.java:78)
~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
at
org.apache.drill.exec.work.filter.RuntimeFilterSink.aggregate(RuntimeFilterSink.java:140)
~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
at
org.apache.drill.exec.work.filter.RuntimeFilterSink.access$600(RuntimeFilterSink.java:52)
~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
at
org.apache.drill.exec.work.filter.RuntimeFilterSink$AsyncAggregateWorker.run(RuntimeFilterSink.java:246)
~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_151]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[na:1.8.0_151]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[na:1.8.0_151]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[na:1.8.0_151]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151]
{code}
Hit the issue with latest changes in the PR ->
https://github.com/apache/drill/pull/1600
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6998) Queries failing with "Failed to aggregate or route the RFW" due to "java.lang.ArrayIndexOutOfBoundsException" do not complete
[
https://issues.apache.org/jira/browse/DRILL-6998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16750598#comment-16750598
]
Sorabh Hamirwasia commented on DRILL-6998:
--
There are 2 issues here:
1) IOB when multiple fields is used in join condition.
2) Foreman thread deadlocks when aggregator thread hits an exception while
aggregation. For this I looked into it more and just setting over.set(true) in
catch block will not be sufficient. Since with that the producer thread from
Netty layer might have already produced a RFW in the queue which will be leaked.
I will open 2 separate child JIRA's for these issues and post a PR for the
second case.
> Queries failing with "Failed to aggregate or route the RFW" due to
> "java.lang.ArrayIndexOutOfBoundsException" do not complete
> -
>
> Key: DRILL-6998
> URL: https://issues.apache.org/jira/browse/DRILL-6998
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.16.0
>Reporter: Abhishek Ravi
>Assignee: weijie.tong
>Priority: Major
> Fix For: 1.16.0
>
>
> Following query joins 2 tables on *two* (>1) fields.
> {noformat}
> select count(*) from lineitem l inner join partsupp p on l.l_partkey =
> p.ps_partkey AND l.l_suppkey = p.ps_suppkey
> {noformat}
> The query does not return even though Fragment 0:0 reports a state change
> from {{RUNNING}} -> {{FINISHED}}
> Following is the jstack output of the {{Frag0:0}}.
> {noformat}
> "23b85137-b102-39a9-70d9-72381c5fb93b:frag:0:0" #16037 daemon prio=10
> os_prio=0 tid=0x7f5f48d415d0 nid=0x1a61 waiting on condition
> [0x7f61b32b2000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.drill.exec.work.filter.RuntimeFilterSink.close(RuntimeFilterSink.java:116)
> at
> org.apache.drill.exec.work.filter.RuntimeFilterRouter.waitForComplete(RuntimeFilterRouter.java:113)
> at
> org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:738)
> at
> org.apache.drill.exec.work.foreman.QueryStateProcessor.wrapUpCompletion(QueryStateProcessor.java:315)
> at
> org.apache.drill.exec.work.foreman.QueryStateProcessor.running(QueryStateProcessor.java:276)
> at
> org.apache.drill.exec.work.foreman.QueryStateProcessor.moveToState(QueryStateProcessor.java:92)
> - locked <0x00055f9a7468> (a
> org.apache.drill.exec.work.foreman.QueryStateProcessor)
> at
> org.apache.drill.exec.work.foreman.QueryStateProcessor$StateSwitch.processEvent(QueryStateProcessor.java:349)
> at
> org.apache.drill.exec.work.foreman.QueryStateProcessor$StateSwitch.processEvent(QueryStateProcessor.java:342)
> at
> org.apache.drill.common.EventProcessor.processEvents(EventProcessor.java:107)
> at
> org.apache.drill.common.EventProcessor.sendEvent(EventProcessor.java:65)
> at
> org.apache.drill.exec.work.foreman.QueryStateProcessor$StateSwitch.addEvent(QueryStateProcessor.java:344)
> at
> org.apache.drill.exec.work.foreman.QueryStateProcessor.addToEventQueue(QueryStateProcessor.java:155)
> at
> org.apache.drill.exec.work.foreman.Foreman.addToEventQueue(Foreman.java:213)
> at
> org.apache.drill.exec.work.foreman.QueryManager.nodeComplete(QueryManager.java:519)
> at
> org.apache.drill.exec.work.foreman.QueryManager.access$100(QueryManager.java:65)
> at
> org.apache.drill.exec.work.foreman.QueryManager$NodeTracker.fragmentComplete(QueryManager.java:483)
> at
> org.apache.drill.exec.work.foreman.QueryManager.fragmentDone(QueryManager.java:155)
> at
> org.apache.drill.exec.work.foreman.QueryManager.access$400(QueryManager.java:65)
> at
> org.apache.drill.exec.work.foreman.QueryManager$1.statusUpdate(QueryManager.java:546)
> at
> org.apache.drill.exec.rpc.control.WorkEventBus.statusUpdate(WorkEventBus.java:63)
> at
> org.apache.drill.exec.work.batch.ControlMessageHandler.requestFragmentStatus(ControlMessageHandler.java:253)
> at
> org.apache.drill.exec.rpc.control.LocalControlConnectionManager.runCommand(LocalControlConnectionManager.java:130)
> at
> org.apache.drill.exec.rpc.control.ControlTunnel.sendFragmentStatus(ControlTunnel.java:89)
> at
> org.apache.drill.exec.work.fragment.FragmentStatusReporter.sendStatus(FragmentStatusReporter.java:122)
> at
> org.apache.drill.exec.work.fragment.FragmentStatusReporter.stateChanged(FragmentStatusReporter.java:91)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:367)
> at
>
[jira] [Created] (DRILL-7000) Queries failing with "Failed to aggregate or route the RFW" do not complete
Sorabh Hamirwasia created DRILL-7000:
Summary: Queries failing with "Failed to aggregate or route the
RFW" do not complete
Key: DRILL-7000
URL: https://issues.apache.org/jira/browse/DRILL-7000
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.16.0
Reporter: Sorabh Hamirwasia
Assignee: Sorabh Hamirwasia
Fix For: 1.16.0
The query does not return even though Fragment 0:0 reports a state change from
RUNNING -> FINISHED
{code:java}
Following is the jstack output of the Frag0:0.
"23b85137-b102-39a9-70d9-72381c5fb93b:frag:0:0" #16037 daemon prio=10 os_prio=0
tid=0x7f5f48d415d0 nid=0x1a61 waiting on condition [0x7f61b32b2000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.drill.exec.work.filter.RuntimeFilterSink.close(RuntimeFilterSink.java:116)
at
org.apache.drill.exec.work.filter.RuntimeFilterRouter.waitForComplete(RuntimeFilterRouter.java:113)
at
org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:738)
at
org.apache.drill.exec.work.foreman.QueryStateProcessor.wrapUpCompletion(QueryStateProcessor.java:315)
at
org.apache.drill.exec.work.foreman.QueryStateProcessor.running(QueryStateProcessor.java:276)
at
org.apache.drill.exec.work.foreman.QueryStateProcessor.moveToState(QueryStateProcessor.java:92)
locked <0x00055f9a7468> (a
org.apache.drill.exec.work.foreman.QueryStateProcessor)
at
org.apache.drill.exec.work.foreman.QueryStateProcessor$StateSwitch.processEvent(QueryStateProcessor.java:349)
at
org.apache.drill.exec.work.foreman.QueryStateProcessor$StateSwitch.processEvent(QueryStateProcessor.java:342)
at
org.apache.drill.common.EventProcessor.processEvents(EventProcessor.java:107)
at org.apache.drill.common.EventProcessor.sendEvent(EventProcessor.java:65)
at
org.apache.drill.exec.work.foreman.QueryStateProcessor$StateSwitch.addEvent(QueryStateProcessor.java:344)
at
org.apache.drill.exec.work.foreman.QueryStateProcessor.addToEventQueue(QueryStateProcessor.java:155)
at org.apache.drill.exec.work.foreman.Foreman.addToEventQueue(Foreman.java:213)
at
org.apache.drill.exec.work.foreman.QueryManager.nodeComplete(QueryManager.java:519)
at
org.apache.drill.exec.work.foreman.QueryManager.access$100(QueryManager.java:65)
at
org.apache.drill.exec.work.foreman.QueryManager$NodeTracker.fragmentComplete(QueryManager.java:483)
at
org.apache.drill.exec.work.foreman.QueryManager.fragmentDone(QueryManager.java:155)
at
org.apache.drill.exec.work.foreman.QueryManager.access$400(QueryManager.java:65)
at
org.apache.drill.exec.work.foreman.QueryManager$1.statusUpdate(QueryManager.java:546)
at
org.apache.drill.exec.rpc.control.WorkEventBus.statusUpdate(WorkEventBus.java:63)
at
org.apache.drill.exec.work.batch.ControlMessageHandler.requestFragmentStatus(ControlMessageHandler.java:253)
at
org.apache.drill.exec.rpc.control.LocalControlConnectionManager.runCommand(LocalControlConnectionManager.java:130)
at
org.apache.drill.exec.rpc.control.ControlTunnel.sendFragmentStatus(ControlTunnel.java:89)
at
org.apache.drill.exec.work.fragment.FragmentStatusReporter.sendStatus(FragmentStatusReporter.java:122)
at
org.apache.drill.exec.work.fragment.FragmentStatusReporter.stateChanged(FragmentStatusReporter.java:91)
at
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:367)
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:219)
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:330)
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748) {code}
- From the code, it seems that RuntimeFilterSink.close is stuck at
{code:java}
while (!asyncAggregateWorker.over.get()) {
try
{ Thread.sleep(100); }
catch (InterruptedException e)
{ logger.error("interrupted while sleeping to wait for the aggregating worker
thread to exit", e); }
}
{code}
This is because AsyncAggregateWorker exits due to the following exception,
before it could set asyncAggregateWorker.over is set to false.
{code:java}
2019-01-22 16:01:18,773 [drill-executor-1301] ERROR
o.a.d.e.w.filter.RuntimeFilterSink - Failed to aggregate or route the RFW
java.lang.ArrayIndexOutOfBoundsException: 1
at
org.apache.drill.exec.work.filter.RuntimeFilterWritable.unwrap(RuntimeFilterWritable.java:67)
~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
at
org.apache.drill.exec.work.filter.RuntimeFilterWritable.aggregate(RuntimeFilterWritable.java:78)
~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
[jira] [Commented] (DRILL-6983) PAM Auth Enabled on Drill-On-YARN only works on YARN user
[
https://issues.apache.org/jira/browse/DRILL-6983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16750586#comment-16750586
]
Kunal Khatua commented on DRILL-6983:
-
[~mikehomee] this *might* be a config/setup issue. Can you try asking in the
User mailing list?
[https://drill.apache.org/mailinglists/]
Also, take a look in the archives as this might have been addressed previously.
> PAM Auth Enabled on Drill-On-YARN only works on YARN user
> -
>
> Key: DRILL-6983
> URL: https://issues.apache.org/jira/browse/DRILL-6983
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - HTTP
>Affects Versions: 1.14.0, 1.15.0
>Reporter: Michael Dennis Uanang
>Priority: Major
> Attachments: Selection_999(203).png, Selection_999(204).png,
> Selection_999(205).png
>
>
> Hi,
> I'm having problem running Drill-on-YARN with PAM authentication enabled. PAM
> auth is working, BUT only accepting login via WEBUI for YARN user.
> _drill-override.conf_
>
> {code:java}
> drill.exec: {
> cluster-id: "drillbits2",
> zk.connect: "app40:2181,app41:2181,app42:2181",
> impersonation: {
> enabled: true
> },
> security: {
> auth.mechanisms: [ "PLAIN" ],
> user.auth.enabled: true,
> user.auth.packages += "org.apache.drill.exec.rpc.user.security",
> user.auth.impl: "pam",
> user.auth.pam_profiles: [ "login", "sshd" ]
> }
> }
> {code}
>
>
> SEE errors below:
> !Selection_999(204).png!
>
> !Selection_999(203).png!
> As you can see from the screenshot, when trying to login via WEBUI using
> infra or drill user, I'm having error 'password check failed for user
> (USER)`. But you'll also notice that it's giving me authentication failure
> for UID=1018 which is YARN
> !Selection_999(205).png!
>
> Please help me to right direction or if I'm missing something.
> Thank you.
>
> MD
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6998) Queries failing with "Failed to aggregate or route the RFW" due to "java.lang.ArrayIndexOutOfBoundsException" do not complete
[
https://issues.apache.org/jira/browse/DRILL-6998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Ravi updated DRILL-6998:
-
Description:
Following query joins 2 tables on *two* (>1) fields.
{noformat}
select count(*) from lineitem l inner join partsupp p on l.l_partkey =
p.ps_partkey AND l.l_suppkey = p.ps_suppkey
{noformat}
The query does not return even though Fragment 0:0 reports a state change from
{{RUNNING}} -> {{FINISHED}}
Following is the jstack output of the {{Frag0:0}}.
{noformat}
"23b85137-b102-39a9-70d9-72381c5fb93b:frag:0:0" #16037 daemon prio=10 os_prio=0
tid=0x7f5f48d415d0 nid=0x1a61 waiting on condition [0x7f61b32b2000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.drill.exec.work.filter.RuntimeFilterSink.close(RuntimeFilterSink.java:116)
at
org.apache.drill.exec.work.filter.RuntimeFilterRouter.waitForComplete(RuntimeFilterRouter.java:113)
at
org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:738)
at
org.apache.drill.exec.work.foreman.QueryStateProcessor.wrapUpCompletion(QueryStateProcessor.java:315)
at
org.apache.drill.exec.work.foreman.QueryStateProcessor.running(QueryStateProcessor.java:276)
at
org.apache.drill.exec.work.foreman.QueryStateProcessor.moveToState(QueryStateProcessor.java:92)
- locked <0x00055f9a7468> (a
org.apache.drill.exec.work.foreman.QueryStateProcessor)
at
org.apache.drill.exec.work.foreman.QueryStateProcessor$StateSwitch.processEvent(QueryStateProcessor.java:349)
at
org.apache.drill.exec.work.foreman.QueryStateProcessor$StateSwitch.processEvent(QueryStateProcessor.java:342)
at
org.apache.drill.common.EventProcessor.processEvents(EventProcessor.java:107)
at
org.apache.drill.common.EventProcessor.sendEvent(EventProcessor.java:65)
at
org.apache.drill.exec.work.foreman.QueryStateProcessor$StateSwitch.addEvent(QueryStateProcessor.java:344)
at
org.apache.drill.exec.work.foreman.QueryStateProcessor.addToEventQueue(QueryStateProcessor.java:155)
at
org.apache.drill.exec.work.foreman.Foreman.addToEventQueue(Foreman.java:213)
at
org.apache.drill.exec.work.foreman.QueryManager.nodeComplete(QueryManager.java:519)
at
org.apache.drill.exec.work.foreman.QueryManager.access$100(QueryManager.java:65)
at
org.apache.drill.exec.work.foreman.QueryManager$NodeTracker.fragmentComplete(QueryManager.java:483)
at
org.apache.drill.exec.work.foreman.QueryManager.fragmentDone(QueryManager.java:155)
at
org.apache.drill.exec.work.foreman.QueryManager.access$400(QueryManager.java:65)
at
org.apache.drill.exec.work.foreman.QueryManager$1.statusUpdate(QueryManager.java:546)
at
org.apache.drill.exec.rpc.control.WorkEventBus.statusUpdate(WorkEventBus.java:63)
at
org.apache.drill.exec.work.batch.ControlMessageHandler.requestFragmentStatus(ControlMessageHandler.java:253)
at
org.apache.drill.exec.rpc.control.LocalControlConnectionManager.runCommand(LocalControlConnectionManager.java:130)
at
org.apache.drill.exec.rpc.control.ControlTunnel.sendFragmentStatus(ControlTunnel.java:89)
at
org.apache.drill.exec.work.fragment.FragmentStatusReporter.sendStatus(FragmentStatusReporter.java:122)
at
org.apache.drill.exec.work.fragment.FragmentStatusReporter.stateChanged(FragmentStatusReporter.java:91)
at
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:367)
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:219)
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:330)
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{noformat}
>From the code, it seems that {{RuntimeFilterSink.close}} is stuck at
{code:java}
while (!asyncAggregateWorker.over.get()) {
try {
Thread.sleep(100);
} catch (InterruptedException e) {
logger.error("interrupted while sleeping to wait for the aggregating
worker thread to exit", e);
}
}
{code}
This is because {{AsyncAggregateWorker}} exits due to the following exception,
before it could set asyncAggregateWorker.over is set to *false*.
{noformat}
2019-01-22 16:01:18,773 [drill-executor-1301] ERROR
o.a.d.e.w.filter.RuntimeFilterSink - Failed to aggregate or route the RFW
java.lang.ArrayIndexOutOfBoundsException: 1
at
org.apache.drill.exec.work.filter.RuntimeFilterWritable.unwrap(RuntimeFilterWritable.java:67)
[jira] [Created] (DRILL-6998) Queries failing with "Failed to aggregate or route the RFW" due to "java.lang.ArrayIndexOutOfBoundsException" do not complete
Abhishek Ravi created DRILL-6998:
Summary: Queries failing with "Failed to aggregate or route the
RFW" due to "java.lang.ArrayIndexOutOfBoundsException" do not complete
Key: DRILL-6998
URL: https://issues.apache.org/jira/browse/DRILL-6998
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.16.0
Reporter: Abhishek Ravi
Assignee: weijie.tong
Fix For: 1.16.0
Following query joins 2 tables on *two* (>1) fields.
{noformat}
select count(*) from lineitem l inner join partsupp p on l.l_partkey =
p.ps_partkey AND l.l_suppkey = p.ps_suppkey
{noformat}
The query does not return even though Fragment 0:0 reports a state change from
{{RUNNING}} -> {{FINISHED}}
Following is the jstack output of the {{Frag0:0}}.
{noformat}
"23b85137-b102-39a9-70d9-72381c5fb93b:frag:0:0" #16037 daemon prio=10 os_prio=0
tid=0x7f5f48d415d0 nid=0x1a61 waiting on condition [0x7f61b32b2000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.drill.exec.work.filter.RuntimeFilterSink.close(RuntimeFilterSink.java:116)
at
org.apache.drill.exec.work.filter.RuntimeFilterRouter.waitForComplete(RuntimeFilterRouter.java:113)
at
org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:738)
at
org.apache.drill.exec.work.foreman.QueryStateProcessor.wrapUpCompletion(QueryStateProcessor.java:315)
at
org.apache.drill.exec.work.foreman.QueryStateProcessor.running(QueryStateProcessor.java:276)
at
org.apache.drill.exec.work.foreman.QueryStateProcessor.moveToState(QueryStateProcessor.java:92)
- locked <0x00055f9a7468> (a
org.apache.drill.exec.work.foreman.QueryStateProcessor)
at
org.apache.drill.exec.work.foreman.QueryStateProcessor$StateSwitch.processEvent(QueryStateProcessor.java:349)
at
org.apache.drill.exec.work.foreman.QueryStateProcessor$StateSwitch.processEvent(QueryStateProcessor.java:342)
at
org.apache.drill.common.EventProcessor.processEvents(EventProcessor.java:107)
at
org.apache.drill.common.EventProcessor.sendEvent(EventProcessor.java:65)
at
org.apache.drill.exec.work.foreman.QueryStateProcessor$StateSwitch.addEvent(QueryStateProcessor.java:344)
at
org.apache.drill.exec.work.foreman.QueryStateProcessor.addToEventQueue(QueryStateProcessor.java:155)
at
org.apache.drill.exec.work.foreman.Foreman.addToEventQueue(Foreman.java:213)
at
org.apache.drill.exec.work.foreman.QueryManager.nodeComplete(QueryManager.java:519)
at
org.apache.drill.exec.work.foreman.QueryManager.access$100(QueryManager.java:65)
at
org.apache.drill.exec.work.foreman.QueryManager$NodeTracker.fragmentComplete(QueryManager.java:483)
at
org.apache.drill.exec.work.foreman.QueryManager.fragmentDone(QueryManager.java:155)
at
org.apache.drill.exec.work.foreman.QueryManager.access$400(QueryManager.java:65)
at
org.apache.drill.exec.work.foreman.QueryManager$1.statusUpdate(QueryManager.java:546)
at
org.apache.drill.exec.rpc.control.WorkEventBus.statusUpdate(WorkEventBus.java:63)
at
org.apache.drill.exec.work.batch.ControlMessageHandler.requestFragmentStatus(ControlMessageHandler.java:253)
at
org.apache.drill.exec.rpc.control.LocalControlConnectionManager.runCommand(LocalControlConnectionManager.java:130)
at
org.apache.drill.exec.rpc.control.ControlTunnel.sendFragmentStatus(ControlTunnel.java:89)
at
org.apache.drill.exec.work.fragment.FragmentStatusReporter.sendStatus(FragmentStatusReporter.java:122)
at
org.apache.drill.exec.work.fragment.FragmentStatusReporter.stateChanged(FragmentStatusReporter.java:91)
at
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:367)
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:219)
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:330)
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{noformat}
>From the code, it seems that {{RuntimeFilterSink.close}} is stuck at
{code:java}
while (!asyncAggregateWorker.over.get()) {
try {
Thread.sleep(100);
} catch (InterruptedException e) {
logger.error("interrupted while sleeping to wait for the aggregating
worker thread to exit", e);
}
}
{code}
This is because {{AsyncAggregateWorker}} exits due to the following exception,
before it could set
[jira] [Commented] (DRILL-6997) Semijoin is changing the join ordering for some tpcds queries.
[
https://issues.apache.org/jira/browse/DRILL-6997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16750380#comment-16750380
]
Hanumath Rao Maduri commented on DRILL-6997:
The following query shows the difference in the plan when semijoin is true and
false.
{code:java}
0: jdbc:drill:zk=local> alter session set `planner.enable_semijoin` = true;
+---+---+
| ok | summary |
+---+---+
| true | planner.enable_semijoin updated. |
+---+---+
1 row selected (0.073 seconds)
0: jdbc:drill:zk=local> select * from
dfs.`/home/mapr/data/sf1/parquet/web_sales` ws1,
dfs.`/home/mapr/data/sf1/parquet/web_sales` ws2 where ws1.ws_ship_date_sk in
(select dd.d_date_sk from dfs.`/home/mapr/data/sf1/parquet/date_dim` dd) and
ws1.ws_order_number = ws2.ws_order_number;
+--+--+
| text
| json
|
+--+--+
| DrillScreenRel
DrillProjectRel(**=[$0], **0=[$3])
DrillSemiJoinRel(condition=[=($1, $6)], joinType=[inner])
DrillJoinRel(condition=[=($2, $5)], joinType=[inner])
DrillScanRel(table=[[dfs, /home/mapr/data/sf1/parquet/web_sales]],
groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath
[path=file:/home/mapr/data/sf1/parquet/web_sales]],
selectionRoot=file:/home/mapr/data/sf1/parquet/web_sales, numFiles=1,
numRowGroups=1, usedMetadataFile=false, columns=[`**`, `ws_ship_date_sk`,
`ws_order_number`]]])
DrillScanRel(table=[[dfs, /home/mapr/data/sf1/parquet/web_sales]],
groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath
[path=file:/home/mapr/data/sf1/parquet/web_sales]],
selectionRoot=file:/home/mapr/data/sf1/parquet/web_sales, numFiles=1,
numRowGroups=1, usedMetadataFile=false, columns=[`**`, `ws_ship_date_sk`,
`ws_order_number`]]])
DrillScanRel(table=[[dfs, /home/mapr/data/sf1/parquet/date_dim]],
groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath
[path=file:/home/mapr/data/sf1/parquet/date_dim]],
selectionRoot=file:/home/mapr/data/sf1/parquet/date_dim, numFiles=1,
numRowGroups=1, usedMetadataFile=false, columns=[`d_date_sk`]]])
0: jdbc:drill:zk=local> alter session set `planner.enable_semijoin` = false;
+---+---+
| ok | summary |
+---+---+
| true | planner.enable_semijoin updated. |
+---+---+
1 row selected (0.077 seconds)
0: jdbc:drill:zk=local> select * from
dfs.`/home/mapr/data/sf1/parquet/web_sales` ws1,
dfs.`/home/mapr/data/sf1/parquet/web_sales` ws2 where ws1.ws_ship_date_sk in
(select dd.d_date_sk from dfs.`/home/mapr/data/sf1/parquet/date_dim` dd) and
ws1.ws_order_number = ws2.ws_order_number;
+--+--+
| text
| json
|
+--+--+
| DrillScreenRel
DrillProjectRel(**=[$0], **0=[$3])
DrillProjectRel(**=[$0], ws_ship_date_sk=[$1], ws_order_number=[$2],
**0=[$4], ws_ship_date_sk0=[$5], ws_order_number0=[$6], d_date_sk=[$3])
DrillJoinRel(condition=[=($2, $6)], joinType=[inner])
DrillJoinRel(condition=[=($1, $3)], joinType=[inner])
DrillScanRel(table=[[dfs, /home/mapr/data/sf1/parquet/web_sales]],
groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath
[path=file:/home/mapr/data/sf1/parquet/web_sales]],
selectionRoot=file:/home/mapr/data/sf1/parquet/web_sales, numFiles=1,
numRowGroups=1, usedMetadataFile=false, columns=[`**`, `ws_ship_date_sk`,
`ws_order_number`]]])
DrillAggregateRel(group=[{0}])
DrillScanRel(table=[[dfs, /home/mapr/data/sf1/parquet/date_dim]],
groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath
[path=file:/home/mapr/data/sf1/parquet/date_dim]],
selectionRoot=file:/home/mapr/data/sf1/parquet/date_dim, numFiles=1,
numRowGroups=1, usedMetadataFile=false, columns=[`d_date_sk`]]])
DrillScanRel(table=[[dfs, /home/mapr/data/sf1/parquet/web_sales]],
groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath
[jira] [Updated] (DRILL-6997) Semijoin is changing the join ordering for some tpcds queries.
[
https://issues.apache.org/jira/browse/DRILL-6997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hanumath Rao Maduri updated DRILL-6997:
---
Attachment: 240abc6d-b816-5320-93b1-2a07d850e734.sys.drill
> Semijoin is changing the join ordering for some tpcds queries.
> --
>
> Key: DRILL-6997
> URL: https://issues.apache.org/jira/browse/DRILL-6997
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning Optimization
>Affects Versions: 1.15.0
>Reporter: Hanumath Rao Maduri
>Assignee: Hanumath Rao Maduri
>Priority: Major
> Fix For: 1.16.0
>
> Attachments: 240aa5f8-24c4-e678-8d42-0fc06e5d2465.sys.drill,
> 240abc6d-b816-5320-93b1-2a07d850e734.sys.drill
>
>
> TPCDS query 95 runs 50% slower with semi-join enabled compared to semi-join
> disabled at scale factor 100. It runs 100% slower at scale factor 1000. This
> issue was introduced with commit 71809ca6216d95540b2a41ce1ab2ebb742888671.
> DRILL-6798: Planner changes to support semi-join.
> {code:java}
> with ws_wh as
> (select ws1.ws_order_number,ws1.ws_warehouse_sk wh1,ws2.ws_warehouse_sk wh2
> from web_sales ws1,web_sales ws2
> where ws1.ws_order_number = ws2.ws_order_number
> and ws1.ws_warehouse_sk <> ws2.ws_warehouse_sk)
> [_LIMITA] select [_LIMITB]
> count(distinct ws_order_number) as "order count"
> ,sum(ws_ext_ship_cost) as "total shipping cost"
> ,sum(ws_net_profit) as "total net profit"
> from
> web_sales ws1
> ,date_dim
> ,customer_address
> ,web_site
> where
> d_date between '[YEAR]-[MONTH]-01' and
> (cast('[YEAR]-[MONTH]-01' as date) + 60 days)
> and ws1.ws_ship_date_sk = d_date_sk
> and ws1.ws_ship_addr_sk = ca_address_sk
> and ca_state = '[STATE]'
> and ws1.ws_web_site_sk = web_site_sk
> and web_company_name = 'pri'
> and ws1.ws_order_number in (select ws_order_number
> from ws_wh)
> and ws1.ws_order_number in (select wr_order_number
> from web_returns,ws_wh
> where wr_order_number = ws_wh.ws_order_number)
> order by count(distinct ws_order_number)
> [_LIMITC];
> {code}
> I have attached two profiles. 240abc6d-b816-5320-93b1-2a07d850e734 has
> semi-join enabled. 240aa5f8-24c4-e678-8d42-0fc06e5d2465 has semi-join
> disabled. Both are executed with commit id
> 6267185823c4c50ab31c029ee5b4d9df2fc94d03 and scale factor 100.
> The plan with semi-join enabled has moved the first hash join:
> and ws1.ws_order_number in (select ws_order_number
> from ws_wh)
> It used to be on the build side of the first HJ on the left hand side
> (04-05). It is now on the build side of the fourth HJ on the left hand side
> (01-13).
> The plan with semi-join enabled has a hash_partition_sender (operator 05-00)
> that takes 10 seconds to execute. But all the fragments take about the same
> amount of time.
> The plan with semi-join enabled has two HJ that process 1B rows while the
> plan with semi-joins disabled has one HJ that processes 1B rows.
> The plan with semi-join enabled has several senders and receivers that wait
> more than 10 seconds, (00-07, 01-07, 03-00, 04-00, 07-00, 08-00, 14-00,
> 17-00). When disabled, there is no operator waiting more than 10 seconds.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6997) Semijoin is changing the join ordering for some tpcds queries.
[
https://issues.apache.org/jira/browse/DRILL-6997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hanumath Rao Maduri updated DRILL-6997:
---
Attachment: 240aa5f8-24c4-e678-8d42-0fc06e5d2465.sys.drill
> Semijoin is changing the join ordering for some tpcds queries.
> --
>
> Key: DRILL-6997
> URL: https://issues.apache.org/jira/browse/DRILL-6997
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning Optimization
>Affects Versions: 1.15.0
>Reporter: Hanumath Rao Maduri
>Assignee: Hanumath Rao Maduri
>Priority: Major
> Fix For: 1.16.0
>
> Attachments: 240aa5f8-24c4-e678-8d42-0fc06e5d2465.sys.drill,
> 240abc6d-b816-5320-93b1-2a07d850e734.sys.drill
>
>
> TPCDS query 95 runs 50% slower with semi-join enabled compared to semi-join
> disabled at scale factor 100. It runs 100% slower at scale factor 1000. This
> issue was introduced with commit 71809ca6216d95540b2a41ce1ab2ebb742888671.
> DRILL-6798: Planner changes to support semi-join.
> {code:java}
> with ws_wh as
> (select ws1.ws_order_number,ws1.ws_warehouse_sk wh1,ws2.ws_warehouse_sk wh2
> from web_sales ws1,web_sales ws2
> where ws1.ws_order_number = ws2.ws_order_number
> and ws1.ws_warehouse_sk <> ws2.ws_warehouse_sk)
> [_LIMITA] select [_LIMITB]
> count(distinct ws_order_number) as "order count"
> ,sum(ws_ext_ship_cost) as "total shipping cost"
> ,sum(ws_net_profit) as "total net profit"
> from
> web_sales ws1
> ,date_dim
> ,customer_address
> ,web_site
> where
> d_date between '[YEAR]-[MONTH]-01' and
> (cast('[YEAR]-[MONTH]-01' as date) + 60 days)
> and ws1.ws_ship_date_sk = d_date_sk
> and ws1.ws_ship_addr_sk = ca_address_sk
> and ca_state = '[STATE]'
> and ws1.ws_web_site_sk = web_site_sk
> and web_company_name = 'pri'
> and ws1.ws_order_number in (select ws_order_number
> from ws_wh)
> and ws1.ws_order_number in (select wr_order_number
> from web_returns,ws_wh
> where wr_order_number = ws_wh.ws_order_number)
> order by count(distinct ws_order_number)
> [_LIMITC];
> {code}
> I have attached two profiles. 240abc6d-b816-5320-93b1-2a07d850e734 has
> semi-join enabled. 240aa5f8-24c4-e678-8d42-0fc06e5d2465 has semi-join
> disabled. Both are executed with commit id
> 6267185823c4c50ab31c029ee5b4d9df2fc94d03 and scale factor 100.
> The plan with semi-join enabled has moved the first hash join:
> and ws1.ws_order_number in (select ws_order_number
> from ws_wh)
> It used to be on the build side of the first HJ on the left hand side
> (04-05). It is now on the build side of the fourth HJ on the left hand side
> (01-13).
> The plan with semi-join enabled has a hash_partition_sender (operator 05-00)
> that takes 10 seconds to execute. But all the fragments take about the same
> amount of time.
> The plan with semi-join enabled has two HJ that process 1B rows while the
> plan with semi-joins disabled has one HJ that processes 1B rows.
> The plan with semi-join enabled has several senders and receivers that wait
> more than 10 seconds, (00-07, 01-07, 03-00, 04-00, 07-00, 08-00, 14-00,
> 17-00). When disabled, there is no operator waiting more than 10 seconds.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6997) Semijoin is changing the join ordering for some tpcds queries.
Hanumath Rao Maduri created DRILL-6997:
--
Summary: Semijoin is changing the join ordering for some tpcds
queries.
Key: DRILL-6997
URL: https://issues.apache.org/jira/browse/DRILL-6997
Project: Apache Drill
Issue Type: Bug
Components: Query Planning Optimization
Affects Versions: 1.15.0
Reporter: Hanumath Rao Maduri
Assignee: Hanumath Rao Maduri
Fix For: 1.16.0
TPCDS query 95 runs 50% slower with semi-join enabled compared to semi-join
disabled at scale factor 100. It runs 100% slower at scale factor 1000. This
issue was introduced with commit 71809ca6216d95540b2a41ce1ab2ebb742888671.
DRILL-6798: Planner changes to support semi-join.
{code:java}
with ws_wh as
(select ws1.ws_order_number,ws1.ws_warehouse_sk wh1,ws2.ws_warehouse_sk wh2
from web_sales ws1,web_sales ws2
where ws1.ws_order_number = ws2.ws_order_number
and ws1.ws_warehouse_sk <> ws2.ws_warehouse_sk)
[_LIMITA] select [_LIMITB]
count(distinct ws_order_number) as "order count"
,sum(ws_ext_ship_cost) as "total shipping cost"
,sum(ws_net_profit) as "total net profit"
from
web_sales ws1
,date_dim
,customer_address
,web_site
where
d_date between '[YEAR]-[MONTH]-01' and
(cast('[YEAR]-[MONTH]-01' as date) + 60 days)
and ws1.ws_ship_date_sk = d_date_sk
and ws1.ws_ship_addr_sk = ca_address_sk
and ca_state = '[STATE]'
and ws1.ws_web_site_sk = web_site_sk
and web_company_name = 'pri'
and ws1.ws_order_number in (select ws_order_number
from ws_wh)
and ws1.ws_order_number in (select wr_order_number
from web_returns,ws_wh
where wr_order_number = ws_wh.ws_order_number)
order by count(distinct ws_order_number)
[_LIMITC];
{code}
I have attached two profiles. 240abc6d-b816-5320-93b1-2a07d850e734 has
semi-join enabled. 240aa5f8-24c4-e678-8d42-0fc06e5d2465 has semi-join disabled.
Both are executed with commit id 6267185823c4c50ab31c029ee5b4d9df2fc94d03 and
scale factor 100.
The plan with semi-join enabled has moved the first hash join:
and ws1.ws_order_number in (select ws_order_number
from ws_wh)
It used to be on the build side of the first HJ on the left hand side (04-05).
It is now on the build side of the fourth HJ on the left hand side (01-13).
The plan with semi-join enabled has a hash_partition_sender (operator 05-00)
that takes 10 seconds to execute. But all the fragments take about the same
amount of time.
The plan with semi-join enabled has two HJ that process 1B rows while the plan
with semi-joins disabled has one HJ that processes 1B rows.
The plan with semi-join enabled has several senders and receivers that wait
more than 10 seconds, (00-07, 01-07, 03-00, 04-00, 07-00, 08-00, 14-00, 17-00).
When disabled, there is no operator waiting more than 10 seconds.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6925) Unable to generate Protobuf
[
https://issues.apache.org/jira/browse/DRILL-6925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hanumath Rao Maduri updated DRILL-6925:
---
Labels: (was: ready-to-commit)
> Unable to generate Protobuf
> ---
>
> Key: DRILL-6925
> URL: https://issues.apache.org/jira/browse/DRILL-6925
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.15.0
>Reporter: Arina Ielchiieva
>Assignee: Vitalii Diravka
>Priority: Major
> Fix For: 1.15.0
>
>
> When generating protocol buffers from protocol dir using {{mvn clean
> process-sources -P proto-compile}} the following error occurs:
> {noformat}
> [ERROR] Failed to execute goal com.mycila:license-maven-plugin:3.0:format
> (proto-format) on project drill-protocol: Execution proto-format of goal
> com.mycila:license-maven-plugin:3.0:format failed: Cannot read header
> document header. Cause: Resource header not found in file system, classpath
> or URL: no protocol: header -> [Help 1]
> [ERROR]
> {noformat}
> Regression after DRILL-6895.
> Also need to check C++ protobuf generation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6996) Unauthorized command execution error received for Mongo
Kris Dalpiaz created DRILL-6996:
---
Summary: Unauthorized command execution error received for Mongo
Key: DRILL-6996
URL: https://issues.apache.org/jira/browse/DRILL-6996
Project: Apache Drill
Issue Type: Bug
Components: Storage - MongoDB
Affects Versions: 1.15.0
Environment: Running in embedded mode.
Storage:
{
"type": "mongo",
"connection":
"mongodb://[user]:[pass]@[host1]:[port1],[host2]:[port2],[host3]:[port3]/[db]?authMechanism=SCRAM-SHA-1=true=[set]",
"enabled": true
}
Query:
SELECT _id FROM [storage].[db].[collection];
Error:
Error: SYSTEM ERROR: MongoCommandException: Command failed with error 13
(Unauthorized): 'not authorized on [db] to execute command \{ count:
"[collection]", query: {} }' on server [host1]:[port1]. The full response is {
"ok" : 0.0, "errmsg" : "not authorized on [db] to execute command \{ count:
\"[collection]\", query: {} }", "code" : 13, "codeName" : "Unauthorized" }
Successful Robo command:
db.runCommand({
count: "[collection]",
query: {}
})
Reporter: Kris Dalpiaz
We recently upgraded from 1.13 to 1.15 for the support of Decimal128 between
Drill and some of our Mongo dbs. We are now receiving a MongoCommandException
error when trying to query a Mongo db. No auth settings have been changed for
the previously working login, and the same login is able to execute the command
through Robo 3T.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (DRILL-6540) Upgrade to HADOOP-3.1 libraries
[ https://issues.apache.org/jira/browse/DRILL-6540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva reassigned DRILL-6540: --- Assignee: (was: Vitalii Diravka) > Upgrade to HADOOP-3.1 libraries > > > Key: DRILL-6540 > URL: https://issues.apache.org/jira/browse/DRILL-6540 > Project: Apache Drill > Issue Type: Improvement > Components: Tools, Build Test >Affects Versions: 1.14.0 >Reporter: Vitalii Diravka >Priority: Major > Fix For: Future > > > Currently Drill uses 2.7.1 version of hadoop libraries (hadoop-common, > hadoop-hdfs, hadoop-annotations, hadoop-aws, hadoop-yarn-api, hadoop-client, > hadoop-yarn-client). > Half of year ago the [Hadoop > 3.0|https://hadoop.apache.org/docs/r3.0.0/index.html] was released and > recently it was an update - [Hadoop > 3.1|https://hadoop.apache.org/docs/r3.1.0/]. > To use Drill under Hadoop3.0 distribution we need this upgrade. Also the > newer version includes new features, which can be useful for Drill. > This upgrade is also needed to leverage the newest version of Zookeeper > libraries and Hive 3.1 version. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6996) Unauthorized command execution error received for Mongo
[
https://issues.apache.org/jira/browse/DRILL-6996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kris Dalpiaz updated DRILL-6996:
Description:
We recently upgraded from 1.13 to 1.15 for the support of Decimal128 between
Drill and some of our Mongo dbs. We are now receiving a MongoCommandException
error when trying to query a Mongo db. No auth settings have been changed for
the previously working login, and the same login is able to execute the command
through Robo 3T.
Running in embedded mode.
Storage:
{ "type": "mongo", "connection":
"mongodb://[user]:[pass]@[host1]:[port1],[host2]:[port2],[host3]:[port3]/[db]?authMechanism=SCRAM-SHA-1=true=[set]",
"enabled": true }
Query:
SELECT _id FROM [storage].[db].[collection];
Error:
Error: SYSTEM ERROR: MongoCommandException: Command failed with error 13
(Unauthorized): 'not authorized on [db] to execute command { count:
"[collection]", query: {} }' on server [host1]:[port1]. The full response is {
"ok" : 0.0, "errmsg" : "not authorized on [db] to execute command { count:
\"[collection]\", query: {} }", "code" : 13, "codeName" : "Unauthorized" }
Successful Robo command:
db.runCommand({
count: "[collection]",
query: {}
})
was:We recently upgraded from 1.13 to 1.15 for the support of Decimal128
between Drill and some of our Mongo dbs. We are now receiving a
MongoCommandException error when trying to query a Mongo db. No auth settings
have been changed for the previously working login, and the same login is able
to execute the command through Robo 3T.
> Unauthorized command execution error received for Mongo
> ---
>
> Key: DRILL-6996
> URL: https://issues.apache.org/jira/browse/DRILL-6996
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - MongoDB
>Affects Versions: 1.15.0
>Reporter: Kris Dalpiaz
>Priority: Major
>
> We recently upgraded from 1.13 to 1.15 for the support of Decimal128 between
> Drill and some of our Mongo dbs. We are now receiving a MongoCommandException
> error when trying to query a Mongo db. No auth settings have been changed for
> the previously working login, and the same login is able to execute the
> command through Robo 3T.
> Running in embedded mode.
> Storage:
> { "type": "mongo", "connection":
> "mongodb://[user]:[pass]@[host1]:[port1],[host2]:[port2],[host3]:[port3]/[db]?authMechanism=SCRAM-SHA-1=true=[set]",
> "enabled": true }
> Query:
> SELECT _id FROM [storage].[db].[collection];
> Error:
> Error: SYSTEM ERROR: MongoCommandException: Command failed with error 13
> (Unauthorized): 'not authorized on [db] to execute command { count:
> "[collection]", query: {} }' on server [host1]:[port1]. The full response is
> { "ok" : 0.0, "errmsg" : "not authorized on [db] to execute command { count:
> \"[collection]\", query: {} }", "code" : 13, "codeName" : "Unauthorized" }
> Successful Robo command:
> db.runCommand({
> count: "[collection]",
> query: {}
> })
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6996) Unauthorized command execution error received for Mongo
[
https://issues.apache.org/jira/browse/DRILL-6996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kris Dalpiaz updated DRILL-6996:
Environment: (was: Running in embedded mode.
Storage:
{
"type": "mongo",
"connection":
"mongodb://[user]:[pass]@[host1]:[port1],[host2]:[port2],[host3]:[port3]/[db]?authMechanism=SCRAM-SHA-1=true=[set]",
"enabled": true
}
Query:
SELECT _id FROM [storage].[db].[collection];
Error:
Error: SYSTEM ERROR: MongoCommandException: Command failed with error 13
(Unauthorized): 'not authorized on [db] to execute command \{ count:
"[collection]", query: {} }' on server [host1]:[port1]. The full response is {
"ok" : 0.0, "errmsg" : "not authorized on [db] to execute command \{ count:
\"[collection]\", query: {} }", "code" : 13, "codeName" : "Unauthorized" }
Successful Robo command:
db.runCommand({
count: "[collection]",
query: {}
}))
> Unauthorized command execution error received for Mongo
> ---
>
> Key: DRILL-6996
> URL: https://issues.apache.org/jira/browse/DRILL-6996
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - MongoDB
>Affects Versions: 1.15.0
>Reporter: Kris Dalpiaz
>Priority: Major
>
> We recently upgraded from 1.13 to 1.15 for the support of Decimal128 between
> Drill and some of our Mongo dbs. We are now receiving a MongoCommandException
> error when trying to query a Mongo db. No auth settings have been changed for
> the previously working login, and the same login is able to execute the
> command through Robo 3T.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6890) AppVeyor CI for Drill builds
[ https://issues.apache.org/jira/browse/DRILL-6890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vitalii Diravka updated DRILL-6890: --- Fix Version/s: (was: 1.16.0) Future > AppVeyor CI for Drill builds > > > Key: DRILL-6890 > URL: https://issues.apache.org/jira/browse/DRILL-6890 > Project: Apache Drill > Issue Type: Test > Components: Tools, Build Test >Affects Versions: 1.14.0 >Reporter: Vitalii Diravka >Assignee: Vitalii Diravka >Priority: Major > Fix For: Future > > > Current Drill Tavis build fails because of Travis build time out. > It has two main stages: build without tests and build with tests. To free up > some time we can drop build without tests and put it to AppVeyor CI. > Basically this CI targeted for builds on Windows, but currently Ubuntu image > is also supported there. > Apache Calcite uses this CI for build with tests on Windows. > AppVeyor CI can be enabled for Drill for maven build without tests on Ubuntu > and Windows. > The branch with appveyor.yml config file: > > [https://github.com/vdiravka/drill/commit/61dadaf5b849aa68fbb69d51b7bd4ef12b8c9e34] > This is the result of build with license headers checking (9 min 38 sec for > Ubuntu and 12 min 19 sec for Windows): > > [https://ci.appveyor.com/project/vdiravka/drill/builds/20883239|https://ci.appveyor.com/project/vdiravka/drill/builds/20886068] > Once it is fine the ticket for INFRA to enable AppVeyor should be created and > Drill badges should be updated. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6962) Function coalesce returns an Error when none of the columns in coalesce exist in a parquet file
[
https://issues.apache.org/jira/browse/DRILL-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi updated DRILL-6962:
---
Labels: ready-to-commit (was: )
> Function coalesce returns an Error when none of the columns in coalesce exist
> in a parquet file
> ---
>
> Key: DRILL-6962
> URL: https://issues.apache.org/jira/browse/DRILL-6962
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.13.0
>Reporter: Bohdan Kazydub
>Assignee: Bohdan Kazydub
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.16.0
>
>
> As Drill is schema-free, COALESCE function is expected to return a result and
> not error out even if none of the columns being referred to exists in files
> being queried.
> Here is an example for 2 columns, `unk_col` and `unk_col2`, which do not
> exist in the parquet files
> {code:java}
> select coalesce(unk_col, unk_col2) from dfs.`/tmp/parquetfiles`;
> java.lang.IndexOutOfBoundsException: index (0) must be less than size (0)
> at
> org.apache.drill.shaded.guava.com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:1196)
> Fragment 1:0
> [Error Id: 7b9193fb-289b-4fbf-a52a-2b93b01f0cd0 on dkvm2c:31010]
> (state=,code=0)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6995) Count() function is not working with columns alias
[
https://issues.apache.org/jira/browse/DRILL-6995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16749913#comment-16749913
]
Arina Ielchiieva commented on DRILL-6995:
-
Actually issue is in the Calcite -
https://issues.apache.org/jira/browse/CALCITE-2799.
> Count() function is not working with columns alias
> --
>
> Key: DRILL-6995
> URL: https://issues.apache.org/jira/browse/DRILL-6995
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.15.0
>Reporter: Denys Ordynskiy
>Priority: Major
> Fix For: 1.17.0
>
>
> The query works correctly if I use column name in the count() function:
> {code:java}
> select department_id c from cp.`employee.json` group by c having
> count(department_id) > 1
> {code}
> |c|
> |1|
> |2|
> |3|
> |11|
> |5|
> |4|
> |19|
> |14|
> |15|
> |16|
> But when I set column alias "c" in the count() function, Drill returns *an
> empty result set*:
> {code:java}
> select department_id c from cp.`employee.json` group by c having count(c) > 1
> {code}
> |c|
> | |
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6995) Count() function is not working with columns alias
[
https://issues.apache.org/jira/browse/DRILL-6995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva updated DRILL-6995:
Fix Version/s: 1.17.0
> Count() function is not working with columns alias
> --
>
> Key: DRILL-6995
> URL: https://issues.apache.org/jira/browse/DRILL-6995
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.15.0
>Reporter: Denys Ordynskiy
>Priority: Major
> Fix For: 1.17.0
>
>
> The query works correctly if I use column name in the count() function:
> {code:java}
> select department_id c from cp.`employee.json` group by c having
> count(department_id) > 1
> {code}
> |c|
> |1|
> |2|
> |3|
> |11|
> |5|
> |4|
> |19|
> |14|
> |15|
> |16|
> But when I set column alias "c" in the count() function, Drill returns *an
> empty result set*:
> {code:java}
> select department_id c from cp.`employee.json` group by c having count(c) > 1
> {code}
> |c|
> | |
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6995) Count() function is not working with columns alias
[
https://issues.apache.org/jira/browse/DRILL-6995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16749909#comment-16749909
]
Vitalii Diravka commented on DRILL-6995:
I think in the last case Drill wrongly pushes Project to the Scan operator, see
wrong _columns_ in the plan:
{noformat}
00-05 HashAgg(group=[{0}], agg#0=[COUNT($1)]) : rowType =
RecordType(ANY department_id, BIGINT $f1): rowcount = 46.3, cumulative cost =
...
00-06Scan(table=[[cp, employee.json]], groupscan=[EasyGroupScan
[selectionRoot=classpath:/employee.json, numFiles=1, columns=[`department_id`,
`c`], files=[classpath:/employee.json]]]) : rowType = RecordType(ANY
department_id, ANY c): rowcount = 463.0, cumulative cost = ...
{noformat}
> Count() function is not working with columns alias
> --
>
> Key: DRILL-6995
> URL: https://issues.apache.org/jira/browse/DRILL-6995
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.15.0
>Reporter: Denys Ordynskiy
>Priority: Major
>
> The query works correctly if I use column name in the count() function:
> {code:java}
> select department_id c from cp.`employee.json` group by c having
> count(department_id) > 1
> {code}
> |c|
> |1|
> |2|
> |3|
> |11|
> |5|
> |4|
> |19|
> |14|
> |15|
> |16|
> But when I set column alias "c" in the count() function, Drill returns *an
> empty result set*:
> {code:java}
> select department_id c from cp.`employee.json` group by c having count(c) > 1
> {code}
> |c|
> | |
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6995) Count() function is not working with columns alias
Denys Ordynskiy created DRILL-6995:
--
Summary: Count() function is not working with columns alias
Key: DRILL-6995
URL: https://issues.apache.org/jira/browse/DRILL-6995
Project: Apache Drill
Issue Type: Bug
Affects Versions: 1.15.0
Reporter: Denys Ordynskiy
The query works correctly if I use column name in the count() function:
{code:java}
select department_id c from cp.`employee.json` group by c having
count(department_id) > 1
{code}
|c|
|1|
|2|
|3|
|11|
|5|
|4|
|19|
|14|
|15|
|16|
But when I set column alias "c" in the count() function, Drill returns *an
empty result set*:
{code:java}
select department_id c from cp.`employee.json` group by c having count(c) > 1
{code}
|c|
| |
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6533) (type|sqlType|drillType|mode)Of functions fail when used with constants
[
https://issues.apache.org/jira/browse/DRILL-6533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva updated DRILL-6533:
Labels: ready-to-commit (was: )
> (type|sqlType|drillType|mode)Of functions fail when used with constants
> ---
>
> Key: DRILL-6533
> URL: https://issues.apache.org/jira/browse/DRILL-6533
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.14.0
>Reporter: Arina Ielchiieva
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.16.0
>
>
> Affected functions: typeOf, sqlTypeOf, drillTypeOf, modeOf
> Example:
> {noformat}
> 0: jdbc:drill:drillbit=localhost> select typeOf(1) from (values(1));
> Error: SYSTEM ERROR: IllegalArgumentException: Can not set
> org.apache.drill.exec.vector.complex.reader.FieldReader field
> org.apache.drill.exec.expr.fn.impl.UnionFunctions$GetType.input to
> org.apache.drill.exec.expr.holders.IntHolder
> [Error Id: 709cf2f7-7396-4034-8c5a-e94d78504297 on cv1:31010]
> (org.apache.drill.exec.work.foreman.ForemanException) Unexpected exception
> during fragment initialization: Error while applying rule
> ReduceExpressionsRule(Project), args
> [rel#3945:LogicalProject.NONE.ANY([]).[](input=rel#3944:Subset#0.NONE.ANY([]).[0],EXPR$0=TYPEOF(1))]
> org.apache.drill.exec.work.foreman.Foreman.run():294
> java.util.concurrent.ThreadPoolExecutor.runWorker():1149
> java.util.concurrent.ThreadPoolExecutor$Worker.run():624
> java.lang.Thread.run():748
> Caused By (java.lang.RuntimeException) Error while applying rule
> ReduceExpressionsRule(Project), args
> [rel#3945:LogicalProject.NONE.ANY([]).[](input=rel#3944:Subset#0.NONE.ANY([]).[0],EXPR$0=TYPEOF(1))]
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch():236
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp():652
> org.apache.calcite.tools.Programs$RuleSetProgram.run():368
>
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform():426
>
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform():366
>
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToRawDrel():255
>
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel():315
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan():180
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan():145
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan():83
> org.apache.drill.exec.work.foreman.Foreman.runSQL():567
> org.apache.drill.exec.work.foreman.Foreman.run():266
> java.util.concurrent.ThreadPoolExecutor.runWorker():1149
> java.util.concurrent.ThreadPoolExecutor$Worker.run():624
> java.lang.Thread.run():748
> Caused By (java.lang.RuntimeException) Error in evaluating function of
> typeof
>
> org.apache.drill.exec.expr.fn.interpreter.InterpreterEvaluator$EvalVisitor.visitFunctionHolderExpression():347
>
> org.apache.drill.exec.expr.fn.interpreter.InterpreterEvaluator$EvalVisitor.visitFunctionHolderExpression():194
> org.apache.drill.common.expression.FunctionHolderExpression.accept():53
>
> org.apache.drill.exec.expr.fn.interpreter.InterpreterEvaluator.evaluateConstantExpr():69
> org.apache.drill.exec.planner.logical.DrillConstExecutor.reduce():150
>
> org.apache.calcite.rel.rules.ReduceExpressionsRule.reduceExpressionsInternal():620
> org.apache.calcite.rel.rules.ReduceExpressionsRule.reduceExpressions():540
>
> org.apache.calcite.rel.rules.ReduceExpressionsRule$ProjectReduceExpressionsRule.onMatch():288
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch():212
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp():652
> org.apache.calcite.tools.Programs$RuleSetProgram.run():368
>
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform():426
>
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform():366
>
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToRawDrel():255
>
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel():315
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan():180
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan():145
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan():83
> org.apache.drill.exec.work.foreman.Foreman.runSQL():567
> org.apache.drill.exec.work.foreman.Foreman.run():266
> java.util.concurrent.ThreadPoolExecutor.runWorker():1149
> java.util.concurrent.ThreadPoolExecutor$Worker.run():624
> java.lang.Thread.run():748
> Caused By (java.lang.IllegalArgumentException) Can not set
>
[jira] [Updated] (DRILL-6910) A filtering column remains in scan when filter pruning happens
[
https://issues.apache.org/jira/browse/DRILL-6910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva updated DRILL-6910:
Labels: ready-to-commit (was: )
> A filtering column remains in scan when filter pruning happens
> --
>
> Key: DRILL-6910
> URL: https://issues.apache.org/jira/browse/DRILL-6910
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.14.0
>Reporter: Anton Gozhiy
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.16.0
>
>
> *Data:*
> {code:sql}
> create table dfs.tmp.`nation` as select * from cp.`tpch/nation.parquet` where
> n_regionkey < 10;
> {code}
> *Query:*
> {code:sql}
> explain plan for select n_nationkey from dfs.tmp.`nation` where n_regionkey <
> 10;
> {code}
> *Expected result:*
> The filtering column (n_regionkey) should not be present in scan operator.
> *Actual result:*
> It remains in scan in spite of filter pruning.
> {noformat}
> 00-00Screen : rowType = RecordType(ANY n_nationkey): rowcount = 25.0,
> cumulative cost = {52.5 rows, 77.5 cpu, 50.0 io, 0.0 network, 0.0 memory}, id
> = 112988
> 00-01 Project(n_nationkey=[$1]) : rowType = RecordType(ANY n_nationkey):
> rowcount = 25.0, cumulative cost = {50.0 rows, 75.0 cpu, 50.0 io, 0.0
> network, 0.0 memory}, id = 112987
> 00-02Scan(table=[[dfs, tmp, nation]], groupscan=[ParquetGroupScan
> [entries=[ReadEntryWithPath [path=maprfs:///tmp/nation]],
> selectionRoot=maprfs:/tmp/nation, numFiles=1, numRowGroups=1,
> usedMetadataFile=false, columns=[`n_regionkey`, `n_nationkey`]]]) : rowType =
> RecordType(ANY n_regionkey, ANY n_nationkey): rowcount = 25.0, cumulative
> cost = {25.0 rows, 50.0 cpu, 50.0 io, 0.0 network, 0.0 memory}, id = 112986
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6978) typeOf drillTypeOf sqlTypeOf not work with generated tables
[
https://issues.apache.org/jira/browse/DRILL-6978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva updated DRILL-6978:
Fix Version/s: 1.16.0
> typeOf drillTypeOf sqlTypeOf not work with generated tables
> ---
>
> Key: DRILL-6978
> URL: https://issues.apache.org/jira/browse/DRILL-6978
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.15.0
>Reporter: benj
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.16.0
>
>
>
> *TypeOf functions works when request on files but doesn't work on "generated"
> data
> This works :
> {code:java}
> SELECT typeof(md5), drillTypeOf(md5), sqlTypeOf(md5) FROM
> dfs.tmp.`mytable.csv` LIMIT 2;
> => (OK)
> +--+--++
> | EXPR$0 | EXPR$1 | EXPR$2 |
> +--+--++
> | VARCHAR | VARCHAR | CHARACTER VARYING |
> | VARCHAR | VARCHAR | CHARACTER VARYING |
> +--+--++{code}
> But not :
>
>
> {code:java}
> SELECT typeOf(a) FROM (SELECT CAST (5 as int) AS a) x;
> => (NOK)
> Error: SYSTEM ERROR: IllegalArgumentException: Can not set
> org.apache.drill.exec.vector.complex.reader.FieldReader field
> org.apache.drill.exec.expr.fn.impl.UnionFunctions$GetType.input to
> org.apache.drill.exec.expr.holders.IntHolder
> {code}
> And in a surprising way the next query works :
> {code:java}
> SELECT md5, typeof(t), drillTypeOf(t), sqlTypeOf(t) FROM ((SELECT 'foo' AS t
> ) union (SELECT 'far' AS t)) x;
> => (OK)
> +---+--+--++
> | md5 | EXPR$1 | EXPR$2 | EXPR$3 |
> +---+--+--++
> | foo | VARCHAR | VARCHAR | CHARACTER VARYING |
> | bar | VARCHAR | VARCHAR | CHARACTER VARYING |
> +---+--+--++{code}
>
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6533) (type|sqlType|drillType|mode)Of functions fail when used with constants
[
https://issues.apache.org/jira/browse/DRILL-6533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva updated DRILL-6533:
Reviewer: Arina Ielchiieva
> (type|sqlType|drillType|mode)Of functions fail when used with constants
> ---
>
> Key: DRILL-6533
> URL: https://issues.apache.org/jira/browse/DRILL-6533
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.14.0
>Reporter: Arina Ielchiieva
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.16.0
>
>
> Affected functions: typeOf, sqlTypeOf, drillTypeOf, modeOf
> Example:
> {noformat}
> 0: jdbc:drill:drillbit=localhost> select typeOf(1) from (values(1));
> Error: SYSTEM ERROR: IllegalArgumentException: Can not set
> org.apache.drill.exec.vector.complex.reader.FieldReader field
> org.apache.drill.exec.expr.fn.impl.UnionFunctions$GetType.input to
> org.apache.drill.exec.expr.holders.IntHolder
> [Error Id: 709cf2f7-7396-4034-8c5a-e94d78504297 on cv1:31010]
> (org.apache.drill.exec.work.foreman.ForemanException) Unexpected exception
> during fragment initialization: Error while applying rule
> ReduceExpressionsRule(Project), args
> [rel#3945:LogicalProject.NONE.ANY([]).[](input=rel#3944:Subset#0.NONE.ANY([]).[0],EXPR$0=TYPEOF(1))]
> org.apache.drill.exec.work.foreman.Foreman.run():294
> java.util.concurrent.ThreadPoolExecutor.runWorker():1149
> java.util.concurrent.ThreadPoolExecutor$Worker.run():624
> java.lang.Thread.run():748
> Caused By (java.lang.RuntimeException) Error while applying rule
> ReduceExpressionsRule(Project), args
> [rel#3945:LogicalProject.NONE.ANY([]).[](input=rel#3944:Subset#0.NONE.ANY([]).[0],EXPR$0=TYPEOF(1))]
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch():236
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp():652
> org.apache.calcite.tools.Programs$RuleSetProgram.run():368
>
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform():426
>
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform():366
>
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToRawDrel():255
>
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel():315
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan():180
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan():145
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan():83
> org.apache.drill.exec.work.foreman.Foreman.runSQL():567
> org.apache.drill.exec.work.foreman.Foreman.run():266
> java.util.concurrent.ThreadPoolExecutor.runWorker():1149
> java.util.concurrent.ThreadPoolExecutor$Worker.run():624
> java.lang.Thread.run():748
> Caused By (java.lang.RuntimeException) Error in evaluating function of
> typeof
>
> org.apache.drill.exec.expr.fn.interpreter.InterpreterEvaluator$EvalVisitor.visitFunctionHolderExpression():347
>
> org.apache.drill.exec.expr.fn.interpreter.InterpreterEvaluator$EvalVisitor.visitFunctionHolderExpression():194
> org.apache.drill.common.expression.FunctionHolderExpression.accept():53
>
> org.apache.drill.exec.expr.fn.interpreter.InterpreterEvaluator.evaluateConstantExpr():69
> org.apache.drill.exec.planner.logical.DrillConstExecutor.reduce():150
>
> org.apache.calcite.rel.rules.ReduceExpressionsRule.reduceExpressionsInternal():620
> org.apache.calcite.rel.rules.ReduceExpressionsRule.reduceExpressions():540
>
> org.apache.calcite.rel.rules.ReduceExpressionsRule$ProjectReduceExpressionsRule.onMatch():288
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch():212
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp():652
> org.apache.calcite.tools.Programs$RuleSetProgram.run():368
>
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform():426
>
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform():366
>
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToRawDrel():255
>
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel():315
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan():180
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan():145
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan():83
> org.apache.drill.exec.work.foreman.Foreman.runSQL():567
> org.apache.drill.exec.work.foreman.Foreman.run():266
> java.util.concurrent.ThreadPoolExecutor.runWorker():1149
> java.util.concurrent.ThreadPoolExecutor$Worker.run():624
> java.lang.Thread.run():748
> Caused By (java.lang.IllegalArgumentException) Can not set
>
[jira] [Assigned] (DRILL-6978) typeOf drillTypeOf sqlTypeOf not work with generated tables
[
https://issues.apache.org/jira/browse/DRILL-6978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva reassigned DRILL-6978:
---
Assignee: Volodymyr Vysotskyi
> typeOf drillTypeOf sqlTypeOf not work with generated tables
> ---
>
> Key: DRILL-6978
> URL: https://issues.apache.org/jira/browse/DRILL-6978
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.15.0
>Reporter: benj
>Assignee: Volodymyr Vysotskyi
>Priority: Major
>
>
> *TypeOf functions works when request on files but doesn't work on "generated"
> data
> This works :
> {code:java}
> SELECT typeof(md5), drillTypeOf(md5), sqlTypeOf(md5) FROM
> dfs.tmp.`mytable.csv` LIMIT 2;
> => (OK)
> +--+--++
> | EXPR$0 | EXPR$1 | EXPR$2 |
> +--+--++
> | VARCHAR | VARCHAR | CHARACTER VARYING |
> | VARCHAR | VARCHAR | CHARACTER VARYING |
> +--+--++{code}
> But not :
>
>
> {code:java}
> SELECT typeOf(a) FROM (SELECT CAST (5 as int) AS a) x;
> => (NOK)
> Error: SYSTEM ERROR: IllegalArgumentException: Can not set
> org.apache.drill.exec.vector.complex.reader.FieldReader field
> org.apache.drill.exec.expr.fn.impl.UnionFunctions$GetType.input to
> org.apache.drill.exec.expr.holders.IntHolder
> {code}
> And in a surprising way the next query works :
> {code:java}
> SELECT md5, typeof(t), drillTypeOf(t), sqlTypeOf(t) FROM ((SELECT 'foo' AS t
> ) union (SELECT 'far' AS t)) x;
> => (OK)
> +---+--+--++
> | md5 | EXPR$1 | EXPR$2 | EXPR$3 |
> +---+--+--++
> | foo | VARCHAR | VARCHAR | CHARACTER VARYING |
> | bar | VARCHAR | VARCHAR | CHARACTER VARYING |
> +---+--+--++{code}
>
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6910) A filtering column remains in scan when filter pruning happens
[
https://issues.apache.org/jira/browse/DRILL-6910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva updated DRILL-6910:
Reviewer: Arina Ielchiieva
> A filtering column remains in scan when filter pruning happens
> --
>
> Key: DRILL-6910
> URL: https://issues.apache.org/jira/browse/DRILL-6910
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.14.0
>Reporter: Anton Gozhiy
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.16.0
>
>
> *Data:*
> {code:sql}
> create table dfs.tmp.`nation` as select * from cp.`tpch/nation.parquet` where
> n_regionkey < 10;
> {code}
> *Query:*
> {code:sql}
> explain plan for select n_nationkey from dfs.tmp.`nation` where n_regionkey <
> 10;
> {code}
> *Expected result:*
> The filtering column (n_regionkey) should not be present in scan operator.
> *Actual result:*
> It remains in scan in spite of filter pruning.
> {noformat}
> 00-00Screen : rowType = RecordType(ANY n_nationkey): rowcount = 25.0,
> cumulative cost = {52.5 rows, 77.5 cpu, 50.0 io, 0.0 network, 0.0 memory}, id
> = 112988
> 00-01 Project(n_nationkey=[$1]) : rowType = RecordType(ANY n_nationkey):
> rowcount = 25.0, cumulative cost = {50.0 rows, 75.0 cpu, 50.0 io, 0.0
> network, 0.0 memory}, id = 112987
> 00-02Scan(table=[[dfs, tmp, nation]], groupscan=[ParquetGroupScan
> [entries=[ReadEntryWithPath [path=maprfs:///tmp/nation]],
> selectionRoot=maprfs:/tmp/nation, numFiles=1, numRowGroups=1,
> usedMetadataFile=false, columns=[`n_regionkey`, `n_nationkey`]]]) : rowType =
> RecordType(ANY n_regionkey, ANY n_nationkey): rowcount = 25.0, cumulative
> cost = {25.0 rows, 50.0 cpu, 50.0 io, 0.0 network, 0.0 memory}, id = 112986
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (DRILL-6797) Split function index extraction fails when result has no rows
[
https://issues.apache.org/jira/browse/DRILL-6797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anton Gozhiy closed DRILL-6797.
---
Tested with Drill version 1.16.0-SNAPSHOT (commit
172dc7cb4c3323e9650db2bf7fe1eab76c2fbbe1).
Cases verified:
- Straight case with v. 1.14.0 (Reproduced).
- Straight case with v. 1.16.0-SNAPSHOT (Not reproduced).
- Filter combinations.
- Limit 0.
- Subqueries (Reported a bug: DRILL-6976)
> Split function index extraction fails when result has no rows
> -
>
> Key: DRILL-6797
> URL: https://issues.apache.org/jira/browse/DRILL-6797
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.12.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.15.0
>
>
> *Query:*
> select split(n_name, ' ') [1] from cp.`tpch/nation.parquet` where n_nationkey
> = -1 group by n_name order by n_name limit 10
> *Error:*
> {noformat}
> Error: SYSTEM ERROR: CompileException: Line 23, Column 35: No applicable
> constructor/method found for actual parameters "int,
> org.apache.drill.exec.vector.UntypedNullHolder"; candidates are: "public
> abstract void
> org.apache.drill.exec.vector.complex.reader.BaseReader.read(int,
> org.apache.drill.exec.expr.holders.UnionHolder)", "public abstract void
> org.apache.drill.exec.vector.complex.reader.BaseReader.read(org.apache.drill.exec.expr.holders.UnionHolder)",
> "public abstract void
> org.apache.drill.exec.vector.complex.reader.BaseReader.read(int,
> org.apache.drill.exec.expr.holders.UnionHolder)", "public abstract void
> org.apache.drill.exec.vector.complex.reader.BaseReader.read(org.apache.drill.exec.expr.holders.UnionHolder)",
> "public abstract void
> org.apache.drill.exec.vector.complex.reader.TinyIntReader.read(org.apache.drill.exec.expr.holders.TinyIntHolder)",
> "public abstract void
> org.apache.drill.exec.vector.complex.reader.TinyIntReader.read(org.apache.drill.exec.expr.holders.NullableTinyIntHolder)",
> "public abstract void
> org.apache.drill.exec.vector.complex.reader.BaseReader.read(int,
> org.apache.drill.exec.expr.holders.UnionHolder)", "public abstract void
> org.apache.drill.exec.vector.complex.reader.BaseReader.read(org.apache.drill.exec.expr.holders.UnionHolder)",
> "public abstract void
> org.apache.drill.exec.vector.complex.reader.UInt1Reader.read(org.apache.drill.exec.expr.holders.UInt1Holder)",
> "public abstract void
> org.apache.drill.exec.vector.complex.reader.UInt1Reader.read(org.apache.drill.exec.expr.holders.NullableUInt1Holder)",
> "public abstract void
> org.apache.drill.exec.vector.complex.reader.BaseReader.read(int,
> org.apache.drill.exec.expr.holders.UnionHolder)", "public abstract void
> org.apache.drill.exec.vector.complex.reader.BaseReader.read(org.apache.drill.exec.expr.holders.UnionHolder)",
> "public abstract void
> org.apache.drill.exec.vector.complex.reader.UInt2Reader.read(org.apache.drill.exec.expr.holders.UInt2Holder)",
> "public abstract void
> org.apache.drill.exec.vector.complex.reader.UInt2Reader.read(org.apache.drill.exec.expr.holders.NullableUInt2Holder)",
> "public abstract void
> org.apache.drill.exec.vector.complex.reader.BaseReader.read(int,
> org.apache.drill.exec.expr.holders.UnionHolder)", "public abstract void
> org.apache.drill.exec.vector.complex.reader.BaseReader.read(org.apache.drill.exec.expr.holders.UnionHolder)",
> "public abstract void
> org.apache.drill.exec.vector.complex.reader.SmallIntReader.read(org.apache.drill.exec.expr.holders.SmallIntHolder)",
> "public abstract void
> org.apache.drill.exec.vector.complex.reader.SmallIntReader.read(org.apache.drill.exec.expr.holders.NullableSmallIntHolder)",
> "public abstract void
> org.apache.drill.exec.vector.complex.reader.BaseReader.read(int,
> org.apache.drill.exec.expr.holders.UnionHolder)", "public abstract void
> org.apache.drill.exec.vector.complex.reader.BaseReader.read(org.apache.drill.exec.expr.holders.UnionHolder)",
> "public abstract void
> org.apache.drill.exec.vector.complex.reader.IntReader.read(org.apache.drill.exec.expr.holders.IntHolder)",
> "public abstract void
> org.apache.drill.exec.vector.complex.reader.IntReader.read(org.apache.drill.exec.expr.holders.NullableIntHolder)",
> "public abstract void
> org.apache.drill.exec.vector.complex.reader.BaseReader.read(int,
> org.apache.drill.exec.expr.holders.UnionHolder)", "public abstract void
> org.apache.drill.exec.vector.complex.reader.BaseReader.read(org.apache.drill.exec.expr.holders.UnionHolder)",
> "public abstract void
> org.apache.drill.exec.vector.complex.reader.UInt4Reader.read(org.apache.drill.exec.expr.holders.UInt4Holder)",
> "public abstract void
>
