[jira] [Commented] (DRILL-5910) Logging exception when custom AuthenticatorFactory not found
[ https://issues.apache.org/jira/browse/DRILL-5910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16226540#comment-16226540 ] Volodymyr Tkach commented on DRILL-5910: Merge into master with commit id b0b7df10da14ab94cd657f0c1e3a0c3ff242bedd > Logging exception when custom AuthenticatorFactory not found > - > > Key: DRILL-5910 > URL: https://issues.apache.org/jira/browse/DRILL-5910 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.11.0 >Reporter: Volodymyr Tkach >Assignee: Volodymyr Tkach >Priority: Minor > Labels: ready-to-commit > Fix For: 1.12.0 > > > We need to log the exception when any of custom AuthenticatorFactory fails to > be instansiated in ClientAuthenticatorProvider constructor. We are doing this > to allow drill to use other available AuthenticatorFactory > Steps to repoduce: > 1) Configure plain authentication > 2) Add > -Ddrill.customAuthFactories=org.apache.drill.exec.rpc.security.maprsasl.MapRSaslFactory > to SQLLINE_JAVA_OPTS or another class that is not present in classpath. > 3) Run sqlline and connect to drillbit -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5895) Fix unit tests for mongo storage plugin
[ https://issues.apache.org/jira/browse/DRILL-5895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16226544#comment-16226544 ] Volodymyr Tkach commented on DRILL-5895: Merged into master with commit id c8d01b3141b0b755d4311340c3e5823214332c70 > Fix unit tests for mongo storage plugin > --- > > Key: DRILL-5895 > URL: https://issues.apache.org/jira/browse/DRILL-5895 > Project: Apache Drill > Issue Type: Bug > Components: Storage - MongoDB >Affects Versions: 1.11.0 >Reporter: Volodymyr Tkach >Assignee: Volodymyr Tkach > Labels: ready-to-commit > Fix For: 1.12.0 > > > Mongo tests finish with following exception intermittently. It happens > because [timeout > value|https://github.com/flapdoodle-oss/de.flapdoodle.embed.process/blob/1.7/src/main/java/de/flapdoodle/embed/process/runtime/ProcessControl.java#L132] > from de.flapdoodle.embed.process library is too low for the mongod process > to be stopped gracefully. > I have created > [issue|https://github.com/flapdoodle-oss/de.flapdoodle.embed.process/issues/64] > to suggest making timeout configurable. For now as temporary solution we > will log mongod exception instead of throwing it. > {noformat} > [mongod output] Exception in thread "Thread-25" > java.lang.IllegalStateException: Couldn't kill mongod process! > > Something bad happend. We couldn't kill mongod process, and tried a lot. > If you want this problem solved you can help us if you open a new issue. > Follow this link: > https://github.com/flapdoodle-oss/de.flapdoodle.embed.mongo/issues > Thank you:) > > at > de.flapdoodle.embed.process.runtime.ProcessControl.waitForProcessGotKilled(ProcessControl.java:192) > at > de.flapdoodle.embed.process.runtime.ProcessControl.stop(ProcessControl.java:76) > at > de.flapdoodle.embed.process.runtime.AbstractProcess.stopProcess(AbstractProcess.java:189) > at > de.flapdoodle.embed.mongo.AbstractMongoProcess.stopInternal(AbstractMongoProcess.java:117) > at > de.flapdoodle.embed.process.runtime.AbstractProcess.stop(AbstractProcess.java:170) > at > de.flapdoodle.embed.process.runtime.Executable.stop(Executable.java:73) > at > de.flapdoodle.embed.process.runtime.Executable$JobKiller.run(Executable.java:90) > at java.lang.Thread.run(Thread.java:745) > Exception in thread "Thread-13" java.lang.IllegalStateException: Couldn't > kill mongod process! > > Something bad happend. We couldn't kill mongod process, and tried a lot. > If you want this problem solved you can help us if you open a new issue. > Follow this link: > https://github.com/flapdoodle-oss/de.flapdoodle.embed.mongo/issues > Thank you:) > > at > de.flapdoodle.embed.process.runtime.ProcessControl.waitForProcessGotKilled(ProcessControl.java:192) > at > de.flapdoodle.embed.process.runtime.ProcessControl.stop(ProcessControl.java:76) > at > de.flapdoodle.embed.process.runtime.AbstractProcess.stopProcess(AbstractProcess.java:189) > at > de.flapdoodle.embed.mongo.AbstractMongoProcess.stopInternal(AbstractMongoProcess.java:117) > at > de.flapdoodle.embed.process.runtime.AbstractProcess.stop(AbstractProcess.java:170) > at > de.flapdoodle.embed.process.runtime.Executable.stop(Executable.java:73) > at > de.flapdoodle.embed.process.runtime.Executable$JobKiller.run(Executable.java:90) > at java.lang.Thread.run(Thread.java:745) > Results : > Tests in error: > MongoTestSuit.tearDownCluster:260 » IllegalState Couldn't kill mongod > process!... > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (DRILL-5822) The query with "SELECT *" with "ORDER BY" clause and `planner.slice_target`=1 doesn't preserve column order
[ https://issues.apache.org/jira/browse/DRILL-5822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vitalii Diravka updated DRILL-5822: --- Description: Columns ordering doesn't preserve for the star query with sorting when this is planned into multiple fragments. Repro steps: 1) {code}alter session set `planner.slice_target`=1;{code} 2) ORDER BY clause in the query. Scenarios: {code} 0: jdbc:drill:zk=local> alter session reset `planner.slice_target`; +---++ | ok |summary | +---++ | true | planner.slice_target updated. | +---++ 1 row selected (0.082 seconds) 0: jdbc:drill:zk=local> select * from cp.`tpch/nation.parquet` order by n_name limit 1; +--+--+--+--+ | n_nationkey | n_name | n_regionkey | n_comment | +--+--+--+--+ | 0| ALGERIA | 0| haggle. carefully final deposits detect slyly agai | +--+--+--+--+ 1 row selected (0.141 seconds) 0: jdbc:drill:zk=local> alter session set `planner.slice_target`=1; +---++ | ok |summary | +---++ | true | planner.slice_target updated. | +---++ 1 row selected (0.091 seconds) 0: jdbc:drill:zk=local> select * from cp.`tpch/nation.parquet` order by n_name limit 1; +--+--+--+--+ | n_comment | n_name | n_nationkey | n_regionkey | +--+--+--+--+ | haggle. carefully final deposits detect slyly agai | ALGERIA | 0 | 0| +--+--+--+--+ 1 row selected (0.201 seconds) {code} was: Repro steps: 1) {code}alter session set `planner.slice_target`=1;{code} 2) ORDER BY clause in the query. Scenarios: {code} 0: jdbc:drill:zk=local> alter session reset `planner.slice_target`; +---++ | ok |summary | +---++ | true | planner.slice_target updated. | +---++ 1 row selected (0.082 seconds) 0: jdbc:drill:zk=local> select * from cp.`tpch/nation.parquet` order by n_name limit 1; +--+--+--+--+ | n_nationkey | n_name | n_regionkey | n_comment | +--+--+--+--+ | 0| ALGERIA | 0| haggle. carefully final deposits detect slyly agai | +--+--+--+--+ 1 row selected (0.141 seconds) 0: jdbc:drill:zk=local> alter session set `planner.slice_target`=1; +---++ | ok |summary | +---++ | true | planner.slice_target updated. | +---++ 1 row selected (0.091 seconds) 0: jdbc:drill:zk=local> select * from cp.`tpch/nation.parquet` order by n_name limit 1; +--+--+--+--+ | n_comment | n_name | n_nationkey | n_regionkey | +--+--+--+--+ | haggle. carefully final deposits detect slyly agai | ALGERIA | 0 | 0| +--+--+--+--+ 1 row selected (0.201 seconds) {code} > The query with "SELECT *" with "ORDER BY" clause and `planner.slice_target`=1 > doesn't preserve column order > --- > > Key: DRILL-5822 > URL: https://issues.apache.org/jira/browse/DRILL-5822 > Project: Apache Drill > Issue Type: Bug > Components: Storage - JSON >Affects Versions: 1.11.0 >Reporter: Prasad Nagaraj Subramanya >Assignee: Vitalii Diravka > Fix For: 1.12.0 > > > Columns ordering doesn't preserve for the star query with sorting when this > is planned into multiple fr
[jira] [Commented] (DRILL-847) Merging Receiver requires to canonicalize schemas of input batches
[ https://issues.apache.org/jira/browse/DRILL-847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16226669#comment-16226669 ] ASF GitHub Bot commented on DRILL-847: -- GitHub user vdiravka opened a pull request: https://github.com/apache/drill/pull/1017 The query with "SELECT *" with "ORDER BY" clause and `planner.slice_t… …arget`=1 doesn't preserve column order Issue: Columns ordering doesn't preserve for the star query with sorting when this is planned into multiple fragments. Solution: The commit for DRILL-847 is oudated. There is no need to canonicalize the batch or container since RecordBatchLoader swallows the "schema change" for now if two batches have different column ordering. You can merge this pull request into a Git repository by running: $ git pull https://github.com/vdiravka/drill DRILL-5822 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/1017.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1017 commit 922505c35ed18f5cf76a5ded77a734827c15eb59 Author: Vitalii Diravka Date: 2017-10-26T18:07:33Z The query with "SELECT *" with "ORDER BY" clause and `planner.slice_target`=1 doesn't preserve column order - The commit for DRILL-847 is oudated. There is no need to canonicalize the batch or container since RecordBatchLoader swallows the "schema change" for now if two batches have different column ordering. > Merging Receiver requires to canonicalize schemas of input batches > --- > > Key: DRILL-847 > URL: https://issues.apache.org/jira/browse/DRILL-847 > Project: Apache Drill > Issue Type: Bug >Reporter: Jinfeng Ni >Assignee: Jinfeng Ni > Fix For: 0.4.0 > > Attachments: DRILL-847.1.patch > > > The Merge Receivers assumes all the incoming batches have the same schemas. > If the incoming batches happen to have different schemas, the run-time > generated code for merge receiver would not be able to handle the different > schemas, and could throw ClassCast Exception. > To fix it, we need canonicalize the schemas of incoming batches for merge > receiver. If the schemas are still different after this step, Merge Receiver > would throw exception. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (DRILL-5822) The query with "SELECT *" with "ORDER BY" clause and `planner.slice_target`=1 doesn't preserve column order
[ https://issues.apache.org/jira/browse/DRILL-5822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vitalii Diravka updated DRILL-5822: --- Labels: read (was: ) > The query with "SELECT *" with "ORDER BY" clause and `planner.slice_target`=1 > doesn't preserve column order > --- > > Key: DRILL-5822 > URL: https://issues.apache.org/jira/browse/DRILL-5822 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.11.0 >Reporter: Prasad Nagaraj Subramanya >Assignee: Vitalii Diravka > Labels: read > Fix For: 1.12.0 > > > Columns ordering doesn't preserve for the star query with sorting when this > is planned into multiple fragments. > Repro steps: > 1) {code}alter session set `planner.slice_target`=1;{code} > 2) ORDER BY clause in the query. > Scenarios: > {code} > 0: jdbc:drill:zk=local> alter session reset `planner.slice_target`; > +---++ > | ok |summary | > +---++ > | true | planner.slice_target updated. | > +---++ > 1 row selected (0.082 seconds) > 0: jdbc:drill:zk=local> select * from cp.`tpch/nation.parquet` order by > n_name limit 1; > +--+--+--+--+ > | n_nationkey | n_name | n_regionkey | n_comment > | > +--+--+--+--+ > | 0| ALGERIA | 0| haggle. carefully final deposits > detect slyly agai | > +--+--+--+--+ > 1 row selected (0.141 seconds) > 0: jdbc:drill:zk=local> alter session set `planner.slice_target`=1; > +---++ > | ok |summary | > +---++ > | true | planner.slice_target updated. | > +---++ > 1 row selected (0.091 seconds) > 0: jdbc:drill:zk=local> select * from cp.`tpch/nation.parquet` order by > n_name limit 1; > +--+--+--+--+ > | n_comment | n_name | > n_nationkey | n_regionkey | > +--+--+--+--+ > | haggle. carefully final deposits detect slyly agai | ALGERIA | 0 >| 0| > +--+--+--+--+ > 1 row selected (0.201 seconds) > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (DRILL-5822) The query with "SELECT *" with "ORDER BY" clause and `planner.slice_target`=1 doesn't preserve column order
[ https://issues.apache.org/jira/browse/DRILL-5822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vitalii Diravka updated DRILL-5822: --- Component/s: (was: Storage - JSON) > The query with "SELECT *" with "ORDER BY" clause and `planner.slice_target`=1 > doesn't preserve column order > --- > > Key: DRILL-5822 > URL: https://issues.apache.org/jira/browse/DRILL-5822 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.11.0 >Reporter: Prasad Nagaraj Subramanya >Assignee: Vitalii Diravka > Labels: read > Fix For: 1.12.0 > > > Columns ordering doesn't preserve for the star query with sorting when this > is planned into multiple fragments. > Repro steps: > 1) {code}alter session set `planner.slice_target`=1;{code} > 2) ORDER BY clause in the query. > Scenarios: > {code} > 0: jdbc:drill:zk=local> alter session reset `planner.slice_target`; > +---++ > | ok |summary | > +---++ > | true | planner.slice_target updated. | > +---++ > 1 row selected (0.082 seconds) > 0: jdbc:drill:zk=local> select * from cp.`tpch/nation.parquet` order by > n_name limit 1; > +--+--+--+--+ > | n_nationkey | n_name | n_regionkey | n_comment > | > +--+--+--+--+ > | 0| ALGERIA | 0| haggle. carefully final deposits > detect slyly agai | > +--+--+--+--+ > 1 row selected (0.141 seconds) > 0: jdbc:drill:zk=local> alter session set `planner.slice_target`=1; > +---++ > | ok |summary | > +---++ > | true | planner.slice_target updated. | > +---++ > 1 row selected (0.091 seconds) > 0: jdbc:drill:zk=local> select * from cp.`tpch/nation.parquet` order by > n_name limit 1; > +--+--+--+--+ > | n_comment | n_name | > n_nationkey | n_regionkey | > +--+--+--+--+ > | haggle. carefully final deposits detect slyly agai | ALGERIA | 0 >| 0| > +--+--+--+--+ > 1 row selected (0.201 seconds) > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (DRILL-5822) The query with "SELECT *" with "ORDER BY" clause and `planner.slice_target`=1 doesn't preserve column order
[ https://issues.apache.org/jira/browse/DRILL-5822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vitalii Diravka updated DRILL-5822: --- Labels: (was: read) > The query with "SELECT *" with "ORDER BY" clause and `planner.slice_target`=1 > doesn't preserve column order > --- > > Key: DRILL-5822 > URL: https://issues.apache.org/jira/browse/DRILL-5822 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.11.0 >Reporter: Prasad Nagaraj Subramanya >Assignee: Vitalii Diravka > Fix For: 1.12.0 > > > Columns ordering doesn't preserve for the star query with sorting when this > is planned into multiple fragments. > Repro steps: > 1) {code}alter session set `planner.slice_target`=1;{code} > 2) ORDER BY clause in the query. > Scenarios: > {code} > 0: jdbc:drill:zk=local> alter session reset `planner.slice_target`; > +---++ > | ok |summary | > +---++ > | true | planner.slice_target updated. | > +---++ > 1 row selected (0.082 seconds) > 0: jdbc:drill:zk=local> select * from cp.`tpch/nation.parquet` order by > n_name limit 1; > +--+--+--+--+ > | n_nationkey | n_name | n_regionkey | n_comment > | > +--+--+--+--+ > | 0| ALGERIA | 0| haggle. carefully final deposits > detect slyly agai | > +--+--+--+--+ > 1 row selected (0.141 seconds) > 0: jdbc:drill:zk=local> alter session set `planner.slice_target`=1; > +---++ > | ok |summary | > +---++ > | true | planner.slice_target updated. | > +---++ > 1 row selected (0.091 seconds) > 0: jdbc:drill:zk=local> select * from cp.`tpch/nation.parquet` order by > n_name limit 1; > +--+--+--+--+ > | n_comment | n_name | > n_nationkey | n_regionkey | > +--+--+--+--+ > | haggle. carefully final deposits detect slyly agai | ALGERIA | 0 >| 0| > +--+--+--+--+ > 1 row selected (0.201 seconds) > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5822) The query with "SELECT *" with "ORDER BY" clause and `planner.slice_target`=1 doesn't preserve column order
[ https://issues.apache.org/jira/browse/DRILL-5822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16226679#comment-16226679 ] Vitalii Diravka commented on DRILL-5822: This is an old topic which was discussed in DRILL-1499 and DRILL-3101. For now there is no need to canonicalize the batch or container since RecordBatchLoader swallows the "schema change" if two batches have different column ordering. That's why DRILL-847 is oudated. > The query with "SELECT *" with "ORDER BY" clause and `planner.slice_target`=1 > doesn't preserve column order > --- > > Key: DRILL-5822 > URL: https://issues.apache.org/jira/browse/DRILL-5822 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.11.0 >Reporter: Prasad Nagaraj Subramanya >Assignee: Vitalii Diravka > Fix For: 1.12.0 > > > Columns ordering doesn't preserve for the star query with sorting when this > is planned into multiple fragments. > Repro steps: > 1) {code}alter session set `planner.slice_target`=1;{code} > 2) ORDER BY clause in the query. > Scenarios: > {code} > 0: jdbc:drill:zk=local> alter session reset `planner.slice_target`; > +---++ > | ok |summary | > +---++ > | true | planner.slice_target updated. | > +---++ > 1 row selected (0.082 seconds) > 0: jdbc:drill:zk=local> select * from cp.`tpch/nation.parquet` order by > n_name limit 1; > +--+--+--+--+ > | n_nationkey | n_name | n_regionkey | n_comment > | > +--+--+--+--+ > | 0| ALGERIA | 0| haggle. carefully final deposits > detect slyly agai | > +--+--+--+--+ > 1 row selected (0.141 seconds) > 0: jdbc:drill:zk=local> alter session set `planner.slice_target`=1; > +---++ > | ok |summary | > +---++ > | true | planner.slice_target updated. | > +---++ > 1 row selected (0.091 seconds) > 0: jdbc:drill:zk=local> select * from cp.`tpch/nation.parquet` order by > n_name limit 1; > +--+--+--+--+ > | n_comment | n_name | > n_nationkey | n_regionkey | > +--+--+--+--+ > | haggle. carefully final deposits detect slyly agai | ALGERIA | 0 >| 0| > +--+--+--+--+ > 1 row selected (0.201 seconds) > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5717) change some date time unit cases with specific timezone or Local
[ https://issues.apache.org/jira/browse/DRILL-5717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16226693#comment-16226693 ] ASF GitHub Bot commented on DRILL-5717: --- Github user weijietong commented on the issue: https://github.com/apache/drill/pull/904 @vvysotskyi thanks for your help! have done, please review. > change some date time unit cases with specific timezone or Local > > > Key: DRILL-5717 > URL: https://issues.apache.org/jira/browse/DRILL-5717 > Project: Apache Drill > Issue Type: Bug > Components: Tools, Build & Test >Affects Versions: 1.9.0, 1.11.0 >Reporter: weijie.tong > > Some date time test cases like JodaDateValidatorTest is not Local > independent .This will cause other Local's users's test phase to fail. We > should let these test cases to be Local env independent. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (DRILL-5822) The query with "SELECT *" with "ORDER BY" clause and `planner.slice_target`=1 doesn't preserve column order
[ https://issues.apache.org/jira/browse/DRILL-5822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16226679#comment-16226679 ] Vitalii Diravka edited comment on DRILL-5822 at 10/31/17 1:37 PM: -- This is an old topic which was discussed in DRILL-1499 and DRILL-3101. For now there is no need to canonicalize the batch or container since RecordBatchLoader swallows the "schema change" if two batches have different column ordering. That's why DRILL-847 is oudated. PR for this ticket - https://github.com/apache/drill/pull/1017 was (Author: vitalii): This is an old topic which was discussed in DRILL-1499 and DRILL-3101. For now there is no need to canonicalize the batch or container since RecordBatchLoader swallows the "schema change" if two batches have different column ordering. That's why DRILL-847 is oudated. > The query with "SELECT *" with "ORDER BY" clause and `planner.slice_target`=1 > doesn't preserve column order > --- > > Key: DRILL-5822 > URL: https://issues.apache.org/jira/browse/DRILL-5822 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.11.0 >Reporter: Prasad Nagaraj Subramanya >Assignee: Vitalii Diravka > Fix For: 1.12.0 > > > Columns ordering doesn't preserve for the star query with sorting when this > is planned into multiple fragments. > Repro steps: > 1) {code}alter session set `planner.slice_target`=1;{code} > 2) ORDER BY clause in the query. > Scenarios: > {code} > 0: jdbc:drill:zk=local> alter session reset `planner.slice_target`; > +---++ > | ok |summary | > +---++ > | true | planner.slice_target updated. | > +---++ > 1 row selected (0.082 seconds) > 0: jdbc:drill:zk=local> select * from cp.`tpch/nation.parquet` order by > n_name limit 1; > +--+--+--+--+ > | n_nationkey | n_name | n_regionkey | n_comment > | > +--+--+--+--+ > | 0| ALGERIA | 0| haggle. carefully final deposits > detect slyly agai | > +--+--+--+--+ > 1 row selected (0.141 seconds) > 0: jdbc:drill:zk=local> alter session set `planner.slice_target`=1; > +---++ > | ok |summary | > +---++ > | true | planner.slice_target updated. | > +---++ > 1 row selected (0.091 seconds) > 0: jdbc:drill:zk=local> select * from cp.`tpch/nation.parquet` order by > n_name limit 1; > +--+--+--+--+ > | n_comment | n_name | > n_nationkey | n_regionkey | > +--+--+--+--+ > | haggle. carefully final deposits detect slyly agai | ALGERIA | 0 >| 0| > +--+--+--+--+ > 1 row selected (0.201 seconds) > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5717) change some date time unit cases with specific timezone or Local
[ https://issues.apache.org/jira/browse/DRILL-5717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16226854#comment-16226854 ] ASF GitHub Bot commented on DRILL-5717: --- Github user vvysotskyi commented on a diff in the pull request: https://github.com/apache/drill/pull/904#discussion_r148000393 --- Diff: exec/java-exec/src/test/java/org/apache/drill/exec/fn/impl/testing/TestDateConversions.java --- @@ -104,6 +110,13 @@ public void testPostgresDate() throws Exception { @Test public void testJodaTime() throws Exception { +new MockUp() { --- End diff -- Since the same block of code is used in many places, we should move it to the separate method (and add Javadoc). > change some date time unit cases with specific timezone or Local > > > Key: DRILL-5717 > URL: https://issues.apache.org/jira/browse/DRILL-5717 > Project: Apache Drill > Issue Type: Bug > Components: Tools, Build & Test >Affects Versions: 1.9.0, 1.11.0 >Reporter: weijie.tong > > Some date time test cases like JodaDateValidatorTest is not Local > independent .This will cause other Local's users's test phase to fail. We > should let these test cases to be Local env independent. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5717) change some date time unit cases with specific timezone or Local
[ https://issues.apache.org/jira/browse/DRILL-5717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16226853#comment-16226853 ] ASF GitHub Bot commented on DRILL-5717: --- Github user vvysotskyi commented on a diff in the pull request: https://github.com/apache/drill/pull/904#discussion_r148002267 --- Diff: logical/src/test/java/org/apache/drill/common/expression/fn/JodaDateValidatorTest.java --- @@ -119,13 +118,14 @@ public void testDateDayOfYearYearFormat() { @Test public void testTimeHoursMinutesSecondsFormat() { +Locale.setDefault(Locale.US); --- End diff -- Not sure that this change is necessary. > change some date time unit cases with specific timezone or Local > > > Key: DRILL-5717 > URL: https://issues.apache.org/jira/browse/DRILL-5717 > Project: Apache Drill > Issue Type: Bug > Components: Tools, Build & Test >Affects Versions: 1.9.0, 1.11.0 >Reporter: weijie.tong > > Some date time test cases like JodaDateValidatorTest is not Local > independent .This will cause other Local's users's test phase to fail. We > should let these test cases to be Local env independent. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5717) change some date time unit cases with specific timezone or Local
[ https://issues.apache.org/jira/browse/DRILL-5717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16226852#comment-16226852 ] ASF GitHub Bot commented on DRILL-5717: --- Github user vvysotskyi commented on a diff in the pull request: https://github.com/apache/drill/pull/904#discussion_r147996953 --- Diff: exec/java-exec/src/test/java/org/apache/drill/exec/fn/impl/TestNewDateFunctions.java --- @@ -26,14 +26,14 @@ import org.junit.Test; import org.junit.experimental.categories.Category; -import java.sql.Date; @Category({UnlikelyTest.class, SqlFunctionTest.class}) public class TestNewDateFunctions extends BaseTestQuery { DateTime date; DateTimeFormatter formatter; long unixTimeStamp = -1; + --- End diff -- Please revert the addition of empty line there. > change some date time unit cases with specific timezone or Local > > > Key: DRILL-5717 > URL: https://issues.apache.org/jira/browse/DRILL-5717 > Project: Apache Drill > Issue Type: Bug > Components: Tools, Build & Test >Affects Versions: 1.9.0, 1.11.0 >Reporter: weijie.tong > > Some date time test cases like JodaDateValidatorTest is not Local > independent .This will cause other Local's users's test phase to fail. We > should let these test cases to be Local env independent. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5717) change some date time unit cases with specific timezone or Local
[ https://issues.apache.org/jira/browse/DRILL-5717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16226855#comment-16226855 ] ASF GitHub Bot commented on DRILL-5717: --- Github user vvysotskyi commented on a diff in the pull request: https://github.com/apache/drill/pull/904#discussion_r147997514 --- Diff: exec/java-exec/src/test/java/org/apache/drill/exec/fn/impl/TestNewDateFunctions.java --- @@ -26,14 +26,14 @@ import org.junit.Test; import org.junit.experimental.categories.Category; -import java.sql.Date; --- End diff -- When you removed unused import, two empty lines stayed. Could you please also remove one of them? > change some date time unit cases with specific timezone or Local > > > Key: DRILL-5717 > URL: https://issues.apache.org/jira/browse/DRILL-5717 > Project: Apache Drill > Issue Type: Bug > Components: Tools, Build & Test >Affects Versions: 1.9.0, 1.11.0 >Reporter: weijie.tong > > Some date time test cases like JodaDateValidatorTest is not Local > independent .This will cause other Local's users's test phase to fail. We > should let these test cases to be Local env independent. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5717) change some date time unit cases with specific timezone or Local
[ https://issues.apache.org/jira/browse/DRILL-5717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16226856#comment-16226856 ] ASF GitHub Bot commented on DRILL-5717: --- Github user vvysotskyi commented on a diff in the pull request: https://github.com/apache/drill/pull/904#discussion_r148000718 --- Diff: exec/java-exec/src/test/java/org/apache/drill/exec/fn/impl/testing/TestDateConversions.java --- @@ -117,10 +130,18 @@ public void testJodaTime() throws Exception { .baselineValues(true, true) .baselineValues(false, true) .go(); + --- End diff -- Please revert the addition of empty line. > change some date time unit cases with specific timezone or Local > > > Key: DRILL-5717 > URL: https://issues.apache.org/jira/browse/DRILL-5717 > Project: Apache Drill > Issue Type: Bug > Components: Tools, Build & Test >Affects Versions: 1.9.0, 1.11.0 >Reporter: weijie.tong > > Some date time test cases like JodaDateValidatorTest is not Local > independent .This will cause other Local's users's test phase to fail. We > should let these test cases to be Local env independent. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (DRILL-5916) Drill document window function example on LAST_VALUE is incorrect
Raymond Wong created DRILL-5916: --- Summary: Drill document window function example on LAST_VALUE is incorrect Key: DRILL-5916 URL: https://issues.apache.org/jira/browse/DRILL-5916 Project: Apache Drill Issue Type: Bug Components: Documentation Affects Versions: 1.11.0 Reporter: Raymond Wong Priority: Minor The top and bottom review count example query result is showing incorrect values for the LAST_VALUE column. ([https://drill.apache.org/docs/analyzing-data-using-window-functions/] ) The LAST_VALUE column should have the same value as the review count of each row because the default Window Frame is RANGE BETWEEN UNBOUNDED PRECEDING AND *CURRENT ROW*. Query result using 2017 yelp data set. {quote} SELECT name, city, review_count, FIRST_VALUE(review_count) OVER(PARTITION BY city ORDER BY review_count DESC) AS top_review_count, LAST_VALUE(review_count) OVER(PARTITION BY city ORDER BY review_count DESC) AS bottom_review_count FROM dfs.yelp.`yelp_academic_dataset_business.json` LIMIT 30 ||name ||city ||review_count ||top_review_count||bottom_review_count || |Lululemon Athletica ||5 |5|5 | |Aberdour Castle |Aberdour|4 |4|4 | |Cupz N' Crepes|Ahwatukee |236 |236 |236 | |My Wine Cellar|Ahwatukee |158 |236 |158 | |Florencia Pizza Bistro|Ahwatukee |129 |236 |129 | |Barro's Pizza |Ahwatukee |62 |236 |62 | |Kathy's Alterations |Ahwatukee |30 |236 |30 | |Hertz Rent A Car |Ahwatukee |26 |236 |26 | |Active Kids Pediatrics|Ahwatukee |18 |236 |18 | |Dental by Design |Ahwatukee |18 |236 |18 | |Desert Dog Pet Care |Ahwatukee |10 |236 |10 | |McDonald's|Ahwatukee |7 |236 |7 | |U-Haul|Ahwatukee |6 |236 |6 | |Sprinkler Detective |Ahwatukee |5 |236 |5 | |Hi-Health |Ahwatukee |4 |236 |4 | |Healthy and Clean Living Environments |Ahwatukee |4 |236 |4 | |Designs By Christa|Ahwatukee |4 |236 |4 | {quote} Changing the LAST_VAULE's Window Frame to RANGE BETWEEN UNBOUNDED PRECEDING AND *UNBOUNDED FOLLOWING*. {quote} SELECT name, city, review_count, FIRST_VALUE(review_count) OVER(PARTITION BY city ORDER BY review_count DESC) AS top_review_count, LAST_VALUE(review_count) OVER(PARTITION BY city ORDER BY review_count DESC RANGE BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING) AS bottom_review_count FROM dfs.yelp.`yelp_academic_dataset_business.json` LIMIT 30 ; ||name ||city ||review_count ||top_review_count ||bottom_review_count || |Lululemon Athletica ||5 |5|5 | |Aberdour Castle |Aberdour|4 |4|4 | |Cupz N' Crepes|Ahwatukee |236 |236 |4 | |My Wine Cellar|Ahwatukee |158 |236 |4 | |Florencia Pizza Bistro|Ahwatukee |129 |236 |4 | |Barro's Pizza |Ahwatukee |62 |236 |4 | |Kathy's Alterations |Ahwatukee |30 |236 |4 | |Hertz Rent A Car |Ahwatukee |26 |236 |4
[jira] [Commented] (DRILL-3993) Rebase Drill on Calcite master branch
[ https://issues.apache.org/jira/browse/DRILL-3993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16226880#comment-16226880 ] Roman Kulyk commented on DRILL-3993: Still working on the java-exec test module. There are 18 tests left. You can see the list of errors below: {code} TestFunctionsWithTypeExpoQueries.testEqualBetweenIntervalAndTimestampDiff:403->BaseTestQuery.testRunAndReturn:360 » Rpc TestUnionDistinct.testDiffDataTypesAndModes:288->BaseTestQuery.testRunAndReturn:360 » Rpc TestExampleQueries.testDRILL_3004:1036->BaseTestQuery.testRunAndReturn:360 » Rpc TestExampleQueries.testFilterInSubqueryAndOutside » UserRemote DATA_READ ERROR... TestExampleQueries.testDateImplicitCasting:1205->BaseTestQuery.testRunAndReturn:360 » Rpc TestCaseNullableTypes.testCaseNullableTypesVarchar:45->BaseTestQuery.testRunAndReturn:360 » Rpc TestInfoSchema.describeTableWithSchemaAndColQualifier:271->BaseTestQuery.testRunAndReturn:360 » Rpc TestInfoSchema.describeTableWithColQualifier:258->BaseTestQuery.testRunAndReturn:360 » Rpc TestSqlBracketlessSyntax.checkComplexExpressionParsing:54 » NoClassDefFound co... TestNestedLoopJoin.testNLJWithEmptyBatch:229->BaseTestQuery.testRunAndReturn:360 » Rpc TestFlatten.testFlattenOnEmptyArrayAndNestedMap:600->BaseTestQuery.testRunAndReturn:360 » Rpc TestParquetWriter.testWriteDecimal:591 » After matching 0 records, did not fi... TestCastEmptyStrings.testCastInputTypeNonNullableVarCharToDecimal » UserRemote TestCastEmptyStrings.testCastInputTypeNonNullableVarCharToNumeric » UserRemote TestDateTruncFunctions.dateTruncOnIntervalDay:301->BaseTestQuery.testRunAndReturn:360 » Rpc TestFunctionsQuery.testDecimalRoundUp:417 » After matching 0 records, did not... TestFunctionsQuery.testRandom:922 » After matching 0 records, did not find ex... TestUnionAll.testDiffDataTypesAndModes:272->BaseTestQuery.testRunAndReturn:360 » Rpc {code} > Rebase Drill on Calcite master branch > - > > Key: DRILL-3993 > URL: https://issues.apache.org/jira/browse/DRILL-3993 > Project: Apache Drill > Issue Type: Bug > Components: Query Planning & Optimization >Affects Versions: 1.2.0 >Reporter: Sudheesh Katkam >Assignee: Roman Kulyk > > Calcite keeps moving, and now we need to catch up to Calcite 1.5, and ensure > there are no regressions. > Also, how do we resolve this 'catching up' issue in the long term? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5822) The query with "SELECT *" with "ORDER BY" clause and `planner.slice_target`=1 doesn't preserve column order
[ https://issues.apache.org/jira/browse/DRILL-5822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16226962#comment-16226962 ] ASF GitHub Bot commented on DRILL-5822: --- Github user vdiravka commented on the issue: https://github.com/apache/drill/pull/1017 @paul-rogers Could you please review this PR? You can find a short description here or more detailed - in the jira ticket [DRILL-5822](https://issues.apache.org/jira/browse/DRILL-5822). > The query with "SELECT *" with "ORDER BY" clause and `planner.slice_target`=1 > doesn't preserve column order > --- > > Key: DRILL-5822 > URL: https://issues.apache.org/jira/browse/DRILL-5822 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.11.0 >Reporter: Prasad Nagaraj Subramanya >Assignee: Vitalii Diravka > Fix For: 1.12.0 > > > Columns ordering doesn't preserve for the star query with sorting when this > is planned into multiple fragments. > Repro steps: > 1) {code}alter session set `planner.slice_target`=1;{code} > 2) ORDER BY clause in the query. > Scenarios: > {code} > 0: jdbc:drill:zk=local> alter session reset `planner.slice_target`; > +---++ > | ok |summary | > +---++ > | true | planner.slice_target updated. | > +---++ > 1 row selected (0.082 seconds) > 0: jdbc:drill:zk=local> select * from cp.`tpch/nation.parquet` order by > n_name limit 1; > +--+--+--+--+ > | n_nationkey | n_name | n_regionkey | n_comment > | > +--+--+--+--+ > | 0| ALGERIA | 0| haggle. carefully final deposits > detect slyly agai | > +--+--+--+--+ > 1 row selected (0.141 seconds) > 0: jdbc:drill:zk=local> alter session set `planner.slice_target`=1; > +---++ > | ok |summary | > +---++ > | true | planner.slice_target updated. | > +---++ > 1 row selected (0.091 seconds) > 0: jdbc:drill:zk=local> select * from cp.`tpch/nation.parquet` order by > n_name limit 1; > +--+--+--+--+ > | n_comment | n_name | > n_nationkey | n_regionkey | > +--+--+--+--+ > | haggle. carefully final deposits detect slyly agai | ALGERIA | 0 >| 0| > +--+--+--+--+ > 1 row selected (0.201 seconds) > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (DRILL-5917) Ban json.org library in Drill
Arina Ielchiieva created DRILL-5917: --- Summary: Ban json.org library in Drill Key: DRILL-5917 URL: https://issues.apache.org/jira/browse/DRILL-5917 Project: Apache Drill Issue Type: Task Affects Versions: 1.11.0 Reporter: Arina Ielchiieva Fix For: 1.12.0 Apache Drill has dependencies on json.org lib indirectly from two libraries: com.mapr.hadoop:maprfs:jar:5.2.1-mapr com.mapr.fs:mapr-hbase:jar:5.2.1-mapr {noformat} [INFO] org.apache.drill.contrib:drill-format-mapr:jar:1.12.0-SNAPSHOT [INFO] +- com.mapr.hadoop:maprfs:jar:5.2.1-mapr:compile [INFO] | \- org.json:json:jar:20080701:compile [INFO] \- com.mapr.fs:mapr-hbase:jar:5.2.1-mapr:compile [INFO]\- (org.json:json:jar:20080701:compile - omitted for duplicate) {noformat} Need to make sure we won't have any dependencies from these libs to json.org lib and ban this lib in main pom.xml file. Issue is critical since Apache release won't happen until we make sure json.org lib is not used (https://www.apache.org/legal/resolved.html). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5917) Ban json.org library in Drill
[ https://issues.apache.org/jira/browse/DRILL-5917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16226997#comment-16226997 ] Arina Ielchiieva commented on DRILL-5917: - FYI [~priteshm], [~amansinha100], [~vrozov] > Ban json.org library in Drill > - > > Key: DRILL-5917 > URL: https://issues.apache.org/jira/browse/DRILL-5917 > Project: Apache Drill > Issue Type: Task >Affects Versions: 1.11.0 >Reporter: Arina Ielchiieva > Fix For: 1.12.0 > > > Apache Drill has dependencies on json.org lib indirectly from two libraries: > com.mapr.hadoop:maprfs:jar:5.2.1-mapr > com.mapr.fs:mapr-hbase:jar:5.2.1-mapr > {noformat} > [INFO] org.apache.drill.contrib:drill-format-mapr:jar:1.12.0-SNAPSHOT > [INFO] +- com.mapr.hadoop:maprfs:jar:5.2.1-mapr:compile > [INFO] | \- org.json:json:jar:20080701:compile > [INFO] \- com.mapr.fs:mapr-hbase:jar:5.2.1-mapr:compile > [INFO]\- (org.json:json:jar:20080701:compile - omitted for duplicate) > {noformat} > Need to make sure we won't have any dependencies from these libs to json.org > lib and ban this lib in main pom.xml file. > Issue is critical since Apache release won't happen until we make sure > json.org lib is not used (https://www.apache.org/legal/resolved.html). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5834) Add Networking Functions
[ https://issues.apache.org/jira/browse/DRILL-5834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16227020#comment-16227020 ] ASF GitHub Bot commented on DRILL-5834: --- Github user arina-ielchiieva commented on the issue: https://github.com/apache/drill/pull/971 @cgivre under squash commits @paul-rogers meant that you should end up with one commit (now you have 13). You might consider using rebase and then do force push in your branch to overwrite this 13 commits. Also please make sure your commit message has proper naming: DRILL-XXX: . In your case, `DRILL-5834: Add networking functions`. > Add Networking Functions > > > Key: DRILL-5834 > URL: https://issues.apache.org/jira/browse/DRILL-5834 > Project: Apache Drill > Issue Type: Improvement > Components: Functions - Drill >Affects Versions: 1.11.0 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Minor > Labels: doc-impacting > Fix For: 1.12.0 > > > On the heels of the PCAP plugin, this is a collection of functions that would > facilitate network analysis using Drill. > The functions include: > inet_aton(): Converts an IPv4 address into an integer. > inet_ntoa( ): Converts an integer IP into dotted decimal notation > in_network( , ): Returns true if the IP address is in the given > CIDR block > address_count( ): Returns the number of IPs in a given CIDR block > broadcast_address( ): Returns the broadcast address for a given CIDR > block > netmask( ): Returns the netmask for a given CIDR block. > low_address(): Returns the first address in a given CIDR block. > high_address(): Returns the last address in a given CIDR block. > url_encode( ): Returns a URL encoded string. > url_decode( ): Decodes a URL encoded string. > is_valid_IP(): Returns true if the IP is a valid IP address > is_private_ip(): Returns true if the IP is a private IPv4 address > is_valid_IPv4(): Returns true if the IP is a valid IPv4 address > is_valid_IPv6(): Returns true if the IP is a valid IPv6 address -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5878) TableNotFound exception is being reported for a wrong storage plugin.
[ https://issues.apache.org/jira/browse/DRILL-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16227026#comment-16227026 ] ASF GitHub Bot commented on DRILL-5878: --- Github user arina-ielchiieva commented on the issue: https://github.com/apache/drill/pull/996 @HanumathRao thanks for making the changes. +1, LGTM. > TableNotFound exception is being reported for a wrong storage plugin. > - > > Key: DRILL-5878 > URL: https://issues.apache.org/jira/browse/DRILL-5878 > Project: Apache Drill > Issue Type: Bug > Components: SQL Parser >Affects Versions: 1.11.0 >Reporter: Hanumath Rao Maduri >Assignee: Hanumath Rao Maduri >Priority: Minor > Fix For: 1.12.0 > > > Drill is reporting TableNotFound exception for a wrong storage plugin. > Consider the following query where employee.json is queried using cp plugin. > {code} > 0: jdbc:drill:zk=local> select * from cp.`employee.json` limit 10; > +--++-++--+-+---++-++--++---+-+-++ > | employee_id | full_name | first_name | last_name | position_id > | position_title | store_id | department_id | birth_date | > hire_date| salary | supervisor_id | education_level | > marital_status | gender | management_role | > +--++-++--+-+---++-++--++---+-+-++ > | 1| Sheri Nowmer | Sheri | Nowmer | 1 > | President | 0 | 1 | 1961-08-26 | > 1994-12-01 00:00:00.0 | 8.0 | 0 | Graduate Degree | S >| F | Senior Management | > | 2| Derrick Whelply| Derrick | Whelply| 2 > | VP Country Manager | 0 | 1 | 1915-07-03 | > 1994-12-01 00:00:00.0 | 4.0 | 1 | Graduate Degree | M >| M | Senior Management | > | 4| Michael Spence | Michael | Spence | 2 > | VP Country Manager | 0 | 1 | 1969-06-20 | > 1998-01-01 00:00:00.0 | 4.0 | 1 | Graduate Degree | S >| M | Senior Management | > | 5| Maya Gutierrez | Maya| Gutierrez | 2 > | VP Country Manager | 0 | 1 | 1951-05-10 | > 1998-01-01 00:00:00.0 | 35000.0 | 1 | Bachelors Degree | M >| F | Senior Management | > | 6| Roberta Damstra| Roberta | Damstra| 3 > | VP Information Systems | 0 | 2 | 1942-10-08 | > 1994-12-01 00:00:00.0 | 25000.0 | 1 | Bachelors Degree | M >| F | Senior Management | > | 7| Rebecca Kanagaki | Rebecca | Kanagaki | 4 > | VP Human Resources | 0 | 3 | 1949-03-27 | > 1994-12-01 00:00:00.0 | 15000.0 | 1 | Bachelors Degree | M >| F | Senior Management | > | 8| Kim Brunner| Kim | Brunner| 11 > | Store Manager | 9 | 11 | 1922-08-10 | > 1998-01-01 00:00:00.0 | 1.0 | 5 | Bachelors Degree | S >| F | Store Management | > | 9| Brenda Blumberg| Brenda | Blumberg | 11 > | Store Manager | 21| 11 | 1979-06-23 | > 1998-01-01 00:00:00.0 | 17000.0 | 5 | Graduate Degree | M >| F | Store Management | > | 10 | Darren Stanz | Darren | Stanz | 5 > | VP Finance | 0 | 5 | 1949-08-26 | > 1994-12-01 00:00:00.0 | 5.0 | 1 | Partial College | M >| M | Senior Management | > | 11 | Jonathan Murraiin | Jonathan| Murraiin | 11 > | Store Manager | 1 | 11 | 1967-06-20 | > 1998-01-01 00:00:00.0 | 15000.0 | 5 | Graduate Degree | S >| M | Store Management | > +--++-++--+-+---++-+---
[jira] [Assigned] (DRILL-5878) TableNotFound exception is being reported for a wrong storage plugin.
[ https://issues.apache.org/jira/browse/DRILL-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva reassigned DRILL-5878: --- Assignee: Hanumath Rao Maduri (was: Arina Ielchiieva) > TableNotFound exception is being reported for a wrong storage plugin. > - > > Key: DRILL-5878 > URL: https://issues.apache.org/jira/browse/DRILL-5878 > Project: Apache Drill > Issue Type: Bug > Components: SQL Parser >Affects Versions: 1.11.0 >Reporter: Hanumath Rao Maduri >Assignee: Hanumath Rao Maduri >Priority: Minor > Labels: ready-to-commit > Fix For: 1.12.0 > > > Drill is reporting TableNotFound exception for a wrong storage plugin. > Consider the following query where employee.json is queried using cp plugin. > {code} > 0: jdbc:drill:zk=local> select * from cp.`employee.json` limit 10; > +--++-++--+-+---++-++--++---+-+-++ > | employee_id | full_name | first_name | last_name | position_id > | position_title | store_id | department_id | birth_date | > hire_date| salary | supervisor_id | education_level | > marital_status | gender | management_role | > +--++-++--+-+---++-++--++---+-+-++ > | 1| Sheri Nowmer | Sheri | Nowmer | 1 > | President | 0 | 1 | 1961-08-26 | > 1994-12-01 00:00:00.0 | 8.0 | 0 | Graduate Degree | S >| F | Senior Management | > | 2| Derrick Whelply| Derrick | Whelply| 2 > | VP Country Manager | 0 | 1 | 1915-07-03 | > 1994-12-01 00:00:00.0 | 4.0 | 1 | Graduate Degree | M >| M | Senior Management | > | 4| Michael Spence | Michael | Spence | 2 > | VP Country Manager | 0 | 1 | 1969-06-20 | > 1998-01-01 00:00:00.0 | 4.0 | 1 | Graduate Degree | S >| M | Senior Management | > | 5| Maya Gutierrez | Maya| Gutierrez | 2 > | VP Country Manager | 0 | 1 | 1951-05-10 | > 1998-01-01 00:00:00.0 | 35000.0 | 1 | Bachelors Degree | M >| F | Senior Management | > | 6| Roberta Damstra| Roberta | Damstra| 3 > | VP Information Systems | 0 | 2 | 1942-10-08 | > 1994-12-01 00:00:00.0 | 25000.0 | 1 | Bachelors Degree | M >| F | Senior Management | > | 7| Rebecca Kanagaki | Rebecca | Kanagaki | 4 > | VP Human Resources | 0 | 3 | 1949-03-27 | > 1994-12-01 00:00:00.0 | 15000.0 | 1 | Bachelors Degree | M >| F | Senior Management | > | 8| Kim Brunner| Kim | Brunner| 11 > | Store Manager | 9 | 11 | 1922-08-10 | > 1998-01-01 00:00:00.0 | 1.0 | 5 | Bachelors Degree | S >| F | Store Management | > | 9| Brenda Blumberg| Brenda | Blumberg | 11 > | Store Manager | 21| 11 | 1979-06-23 | > 1998-01-01 00:00:00.0 | 17000.0 | 5 | Graduate Degree | M >| F | Store Management | > | 10 | Darren Stanz | Darren | Stanz | 5 > | VP Finance | 0 | 5 | 1949-08-26 | > 1994-12-01 00:00:00.0 | 5.0 | 1 | Partial College | M >| M | Senior Management | > | 11 | Jonathan Murraiin | Jonathan| Murraiin | 11 > | Store Manager | 1 | 11 | 1967-06-20 | > 1998-01-01 00:00:00.0 | 15000.0 | 5 | Graduate Degree | S >| M | Store Management | > +--++-++--+-+---++-++--++---+-+-++ > {code} >
[jira] [Assigned] (DRILL-5878) TableNotFound exception is being reported for a wrong storage plugin.
[ https://issues.apache.org/jira/browse/DRILL-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva reassigned DRILL-5878: --- Assignee: Arina Ielchiieva (was: Hanumath Rao Maduri) > TableNotFound exception is being reported for a wrong storage plugin. > - > > Key: DRILL-5878 > URL: https://issues.apache.org/jira/browse/DRILL-5878 > Project: Apache Drill > Issue Type: Bug > Components: SQL Parser >Affects Versions: 1.11.0 >Reporter: Hanumath Rao Maduri >Assignee: Arina Ielchiieva >Priority: Minor > Labels: ready-to-commit > Fix For: 1.12.0 > > > Drill is reporting TableNotFound exception for a wrong storage plugin. > Consider the following query where employee.json is queried using cp plugin. > {code} > 0: jdbc:drill:zk=local> select * from cp.`employee.json` limit 10; > +--++-++--+-+---++-++--++---+-+-++ > | employee_id | full_name | first_name | last_name | position_id > | position_title | store_id | department_id | birth_date | > hire_date| salary | supervisor_id | education_level | > marital_status | gender | management_role | > +--++-++--+-+---++-++--++---+-+-++ > | 1| Sheri Nowmer | Sheri | Nowmer | 1 > | President | 0 | 1 | 1961-08-26 | > 1994-12-01 00:00:00.0 | 8.0 | 0 | Graduate Degree | S >| F | Senior Management | > | 2| Derrick Whelply| Derrick | Whelply| 2 > | VP Country Manager | 0 | 1 | 1915-07-03 | > 1994-12-01 00:00:00.0 | 4.0 | 1 | Graduate Degree | M >| M | Senior Management | > | 4| Michael Spence | Michael | Spence | 2 > | VP Country Manager | 0 | 1 | 1969-06-20 | > 1998-01-01 00:00:00.0 | 4.0 | 1 | Graduate Degree | S >| M | Senior Management | > | 5| Maya Gutierrez | Maya| Gutierrez | 2 > | VP Country Manager | 0 | 1 | 1951-05-10 | > 1998-01-01 00:00:00.0 | 35000.0 | 1 | Bachelors Degree | M >| F | Senior Management | > | 6| Roberta Damstra| Roberta | Damstra| 3 > | VP Information Systems | 0 | 2 | 1942-10-08 | > 1994-12-01 00:00:00.0 | 25000.0 | 1 | Bachelors Degree | M >| F | Senior Management | > | 7| Rebecca Kanagaki | Rebecca | Kanagaki | 4 > | VP Human Resources | 0 | 3 | 1949-03-27 | > 1994-12-01 00:00:00.0 | 15000.0 | 1 | Bachelors Degree | M >| F | Senior Management | > | 8| Kim Brunner| Kim | Brunner| 11 > | Store Manager | 9 | 11 | 1922-08-10 | > 1998-01-01 00:00:00.0 | 1.0 | 5 | Bachelors Degree | S >| F | Store Management | > | 9| Brenda Blumberg| Brenda | Blumberg | 11 > | Store Manager | 21| 11 | 1979-06-23 | > 1998-01-01 00:00:00.0 | 17000.0 | 5 | Graduate Degree | M >| F | Store Management | > | 10 | Darren Stanz | Darren | Stanz | 5 > | VP Finance | 0 | 5 | 1949-08-26 | > 1994-12-01 00:00:00.0 | 5.0 | 1 | Partial College | M >| M | Senior Management | > | 11 | Jonathan Murraiin | Jonathan| Murraiin | 11 > | Store Manager | 1 | 11 | 1967-06-20 | > 1998-01-01 00:00:00.0 | 15000.0 | 5 | Graduate Degree | S >| M | Store Management | > +--++-++--+-+---++-++--++---+-+-++ > {code} > How
[jira] [Updated] (DRILL-5878) TableNotFound exception is being reported for a wrong storage plugin.
[ https://issues.apache.org/jira/browse/DRILL-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva updated DRILL-5878: Labels: ready-to-commit (was: ) > TableNotFound exception is being reported for a wrong storage plugin. > - > > Key: DRILL-5878 > URL: https://issues.apache.org/jira/browse/DRILL-5878 > Project: Apache Drill > Issue Type: Bug > Components: SQL Parser >Affects Versions: 1.11.0 >Reporter: Hanumath Rao Maduri >Assignee: Arina Ielchiieva >Priority: Minor > Labels: ready-to-commit > Fix For: 1.12.0 > > > Drill is reporting TableNotFound exception for a wrong storage plugin. > Consider the following query where employee.json is queried using cp plugin. > {code} > 0: jdbc:drill:zk=local> select * from cp.`employee.json` limit 10; > +--++-++--+-+---++-++--++---+-+-++ > | employee_id | full_name | first_name | last_name | position_id > | position_title | store_id | department_id | birth_date | > hire_date| salary | supervisor_id | education_level | > marital_status | gender | management_role | > +--++-++--+-+---++-++--++---+-+-++ > | 1| Sheri Nowmer | Sheri | Nowmer | 1 > | President | 0 | 1 | 1961-08-26 | > 1994-12-01 00:00:00.0 | 8.0 | 0 | Graduate Degree | S >| F | Senior Management | > | 2| Derrick Whelply| Derrick | Whelply| 2 > | VP Country Manager | 0 | 1 | 1915-07-03 | > 1994-12-01 00:00:00.0 | 4.0 | 1 | Graduate Degree | M >| M | Senior Management | > | 4| Michael Spence | Michael | Spence | 2 > | VP Country Manager | 0 | 1 | 1969-06-20 | > 1998-01-01 00:00:00.0 | 4.0 | 1 | Graduate Degree | S >| M | Senior Management | > | 5| Maya Gutierrez | Maya| Gutierrez | 2 > | VP Country Manager | 0 | 1 | 1951-05-10 | > 1998-01-01 00:00:00.0 | 35000.0 | 1 | Bachelors Degree | M >| F | Senior Management | > | 6| Roberta Damstra| Roberta | Damstra| 3 > | VP Information Systems | 0 | 2 | 1942-10-08 | > 1994-12-01 00:00:00.0 | 25000.0 | 1 | Bachelors Degree | M >| F | Senior Management | > | 7| Rebecca Kanagaki | Rebecca | Kanagaki | 4 > | VP Human Resources | 0 | 3 | 1949-03-27 | > 1994-12-01 00:00:00.0 | 15000.0 | 1 | Bachelors Degree | M >| F | Senior Management | > | 8| Kim Brunner| Kim | Brunner| 11 > | Store Manager | 9 | 11 | 1922-08-10 | > 1998-01-01 00:00:00.0 | 1.0 | 5 | Bachelors Degree | S >| F | Store Management | > | 9| Brenda Blumberg| Brenda | Blumberg | 11 > | Store Manager | 21| 11 | 1979-06-23 | > 1998-01-01 00:00:00.0 | 17000.0 | 5 | Graduate Degree | M >| F | Store Management | > | 10 | Darren Stanz | Darren | Stanz | 5 > | VP Finance | 0 | 5 | 1949-08-26 | > 1994-12-01 00:00:00.0 | 5.0 | 1 | Partial College | M >| M | Senior Management | > | 11 | Jonathan Murraiin | Jonathan| Murraiin | 11 > | Store Manager | 1 | 11 | 1967-06-20 | > 1998-01-01 00:00:00.0 | 15000.0 | 5 | Graduate Degree | S >| M | Store Management | > +--++-++--+-+---++-++--++---+-+-++ > {code} > However if cp1 is used instead o
[jira] [Updated] (DRILL-5878) TableNotFound exception is being reported for a wrong storage plugin.
[ https://issues.apache.org/jira/browse/DRILL-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva updated DRILL-5878: Reviewer: Arina Ielchiieva > TableNotFound exception is being reported for a wrong storage plugin. > - > > Key: DRILL-5878 > URL: https://issues.apache.org/jira/browse/DRILL-5878 > Project: Apache Drill > Issue Type: Bug > Components: SQL Parser >Affects Versions: 1.11.0 >Reporter: Hanumath Rao Maduri >Assignee: Hanumath Rao Maduri >Priority: Minor > Labels: ready-to-commit > Fix For: 1.12.0 > > > Drill is reporting TableNotFound exception for a wrong storage plugin. > Consider the following query where employee.json is queried using cp plugin. > {code} > 0: jdbc:drill:zk=local> select * from cp.`employee.json` limit 10; > +--++-++--+-+---++-++--++---+-+-++ > | employee_id | full_name | first_name | last_name | position_id > | position_title | store_id | department_id | birth_date | > hire_date| salary | supervisor_id | education_level | > marital_status | gender | management_role | > +--++-++--+-+---++-++--++---+-+-++ > | 1| Sheri Nowmer | Sheri | Nowmer | 1 > | President | 0 | 1 | 1961-08-26 | > 1994-12-01 00:00:00.0 | 8.0 | 0 | Graduate Degree | S >| F | Senior Management | > | 2| Derrick Whelply| Derrick | Whelply| 2 > | VP Country Manager | 0 | 1 | 1915-07-03 | > 1994-12-01 00:00:00.0 | 4.0 | 1 | Graduate Degree | M >| M | Senior Management | > | 4| Michael Spence | Michael | Spence | 2 > | VP Country Manager | 0 | 1 | 1969-06-20 | > 1998-01-01 00:00:00.0 | 4.0 | 1 | Graduate Degree | S >| M | Senior Management | > | 5| Maya Gutierrez | Maya| Gutierrez | 2 > | VP Country Manager | 0 | 1 | 1951-05-10 | > 1998-01-01 00:00:00.0 | 35000.0 | 1 | Bachelors Degree | M >| F | Senior Management | > | 6| Roberta Damstra| Roberta | Damstra| 3 > | VP Information Systems | 0 | 2 | 1942-10-08 | > 1994-12-01 00:00:00.0 | 25000.0 | 1 | Bachelors Degree | M >| F | Senior Management | > | 7| Rebecca Kanagaki | Rebecca | Kanagaki | 4 > | VP Human Resources | 0 | 3 | 1949-03-27 | > 1994-12-01 00:00:00.0 | 15000.0 | 1 | Bachelors Degree | M >| F | Senior Management | > | 8| Kim Brunner| Kim | Brunner| 11 > | Store Manager | 9 | 11 | 1922-08-10 | > 1998-01-01 00:00:00.0 | 1.0 | 5 | Bachelors Degree | S >| F | Store Management | > | 9| Brenda Blumberg| Brenda | Blumberg | 11 > | Store Manager | 21| 11 | 1979-06-23 | > 1998-01-01 00:00:00.0 | 17000.0 | 5 | Graduate Degree | M >| F | Store Management | > | 10 | Darren Stanz | Darren | Stanz | 5 > | VP Finance | 0 | 5 | 1949-08-26 | > 1994-12-01 00:00:00.0 | 5.0 | 1 | Partial College | M >| M | Senior Management | > | 11 | Jonathan Murraiin | Jonathan| Murraiin | 11 > | Store Manager | 1 | 11 | 1967-06-20 | > 1998-01-01 00:00:00.0 | 15000.0 | 5 | Graduate Degree | S >| M | Store Management | > +--++-++--+-+---++-++--++---+-+-++ > {code} > However if cp1 is used instead of c
[jira] [Commented] (DRILL-5822) The query with "SELECT *" with "ORDER BY" clause and `planner.slice_target`=1 doesn't preserve column order
[ https://issues.apache.org/jira/browse/DRILL-5822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16227074#comment-16227074 ] Paul Rogers commented on DRILL-5822: The general rule for the SQL project clause is the following: * If the list is explicit, `SELECT b, c, a` then columns are returned in that order, even if the table defines them in the order (a, b, c). * If the lis is implicit using a wildcard, `SELECT *`, then the column order is that defined by the table. In our example above, the order would be `a, b, c`. Since Drill is distributed and schema-on-read, we run into the issue that two tables might have the same columns, but defined in different orders. For example, `{"a": 10, "b": 20, "c": 30}` and `{"c": 40, "b": 50, "c": 60}`. In this case, there is no "correct" order. Instead, Drill must: 1. Recognize that the above scenario can occur. 2. Define each merging operator to follow some reconciliation rule. Here a "merging" operator is anything that can see batches from two distinct scans. That is, almost all operators, but at least the receivers. A good reconciliation rule is that the first schema wins, and all other batches are projected into that first schema. In our example, `a, b, c` and `c, b, a` are both projected into `a, b, c`. The PMC has asked that we not discuss design issues in PR reviews. So, can you perhaps please explain here the approach that this PR takes to solve the problem? Do we agree on the description above? Or, did this PR take a different approach? > The query with "SELECT *" with "ORDER BY" clause and `planner.slice_target`=1 > doesn't preserve column order > --- > > Key: DRILL-5822 > URL: https://issues.apache.org/jira/browse/DRILL-5822 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.11.0 >Reporter: Prasad Nagaraj Subramanya >Assignee: Vitalii Diravka > Fix For: 1.12.0 > > > Columns ordering doesn't preserve for the star query with sorting when this > is planned into multiple fragments. > Repro steps: > 1) {code}alter session set `planner.slice_target`=1;{code} > 2) ORDER BY clause in the query. > Scenarios: > {code} > 0: jdbc:drill:zk=local> alter session reset `planner.slice_target`; > +---++ > | ok |summary | > +---++ > | true | planner.slice_target updated. | > +---++ > 1 row selected (0.082 seconds) > 0: jdbc:drill:zk=local> select * from cp.`tpch/nation.parquet` order by > n_name limit 1; > +--+--+--+--+ > | n_nationkey | n_name | n_regionkey | n_comment > | > +--+--+--+--+ > | 0| ALGERIA | 0| haggle. carefully final deposits > detect slyly agai | > +--+--+--+--+ > 1 row selected (0.141 seconds) > 0: jdbc:drill:zk=local> alter session set `planner.slice_target`=1; > +---++ > | ok |summary | > +---++ > | true | planner.slice_target updated. | > +---++ > 1 row selected (0.091 seconds) > 0: jdbc:drill:zk=local> select * from cp.`tpch/nation.parquet` order by > n_name limit 1; > +--+--+--+--+ > | n_comment | n_name | > n_nationkey | n_regionkey | > +--+--+--+--+ > | haggle. carefully final deposits detect slyly agai | ALGERIA | 0 >| 0| > +--+--+--+--+ > 1 row selected (0.201 seconds) > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (DRILL-5822) The query with "SELECT *" with "ORDER BY" clause and `planner.slice_target`=1 doesn't preserve column order
[ https://issues.apache.org/jira/browse/DRILL-5822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16227074#comment-16227074 ] Paul Rogers edited comment on DRILL-5822 at 10/31/17 4:41 PM: -- The general rule for the SQL project clause is the following: * If the list is explicit, {{SELECT b, c, a}} then columns are returned in that order, even if the table defines them in the order (a, b, c). * If the lis is implicit using a wildcard, {{SELECT *}}, then the column order is that defined by the table. In our example above, the order would be {{a, b, c}}. Since Drill is distributed and schema-on-read, we run into the issue that two tables might have the same columns, but defined in different orders. For example: {noformat} Table 1: {"a": 10, "b": 20, "c": 30} Table 2: {"c": 40, "b": 50, "a": 60} {noformat} In this case, there is no "correct" order. Instead, Drill must: 1. Recognize that the above scenario can occur. 2. Define each merging operator to follow some reconciliation rule. Here a "merging" operator is anything that can see batches from two distinct scans. That is, almost all operators, but at least the receivers. A good reconciliation rule is that the first schema wins, and all other batches are projected into that first schema. In our example, {{a, b, c}} and {{c, b, a}} are both projected into {{a, b, c}}. The PMC has asked that we not discuss design issues in PR reviews on Github. So, can you perhaps please explain here the approach that this PR takes to solve the problem? Do we agree on the description above? Or, did this PR take a different approach? was (Author: paul.rogers): The general rule for the SQL project clause is the following: * If the list is explicit, `SELECT b, c, a` then columns are returned in that order, even if the table defines them in the order (a, b, c). * If the lis is implicit using a wildcard, `SELECT *`, then the column order is that defined by the table. In our example above, the order would be `a, b, c`. Since Drill is distributed and schema-on-read, we run into the issue that two tables might have the same columns, but defined in different orders. For example, `{"a": 10, "b": 20, "c": 30}` and `{"c": 40, "b": 50, "c": 60}`. In this case, there is no "correct" order. Instead, Drill must: 1. Recognize that the above scenario can occur. 2. Define each merging operator to follow some reconciliation rule. Here a "merging" operator is anything that can see batches from two distinct scans. That is, almost all operators, but at least the receivers. A good reconciliation rule is that the first schema wins, and all other batches are projected into that first schema. In our example, `a, b, c` and `c, b, a` are both projected into `a, b, c`. The PMC has asked that we not discuss design issues in PR reviews. So, can you perhaps please explain here the approach that this PR takes to solve the problem? Do we agree on the description above? Or, did this PR take a different approach? > The query with "SELECT *" with "ORDER BY" clause and `planner.slice_target`=1 > doesn't preserve column order > --- > > Key: DRILL-5822 > URL: https://issues.apache.org/jira/browse/DRILL-5822 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.11.0 >Reporter: Prasad Nagaraj Subramanya >Assignee: Vitalii Diravka > Fix For: 1.12.0 > > > Columns ordering doesn't preserve for the star query with sorting when this > is planned into multiple fragments. > Repro steps: > 1) {code}alter session set `planner.slice_target`=1;{code} > 2) ORDER BY clause in the query. > Scenarios: > {code} > 0: jdbc:drill:zk=local> alter session reset `planner.slice_target`; > +---++ > | ok |summary | > +---++ > | true | planner.slice_target updated. | > +---++ > 1 row selected (0.082 seconds) > 0: jdbc:drill:zk=local> select * from cp.`tpch/nation.parquet` order by > n_name limit 1; > +--+--+--+--+ > | n_nationkey | n_name | n_regionkey | n_comment > | > +--+--+--+--+ > | 0| ALGERIA | 0| haggle. carefully final deposits > detect slyly agai | > +--+--+--+--+ > 1 row selected (0.141 seconds) > 0: jdbc:drill:zk=local> alter session set `planner.slice_target`=1; > +---++ > | ok |
[jira] [Assigned] (DRILL-5875) user mixed up problem in apache drill
[ https://issues.apache.org/jira/browse/DRILL-5875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pritesh Maker reassigned DRILL-5875: Assignee: Arina Ielchiieva > user mixed up problem in apache drill > - > > Key: DRILL-5875 > URL: https://issues.apache.org/jira/browse/DRILL-5875 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.8.0, 1.9.0, 1.10.0, 1.11.0 >Reporter: flyfantasy >Assignee: Arina Ielchiieva > > Hi guys. > we have used drill for quite a long time. We used apache-drill-1.8.0 at the > beginning and recently we upgraded to apache-drill-1.11.0. Drill is great and > now drill already have 40+ user in our company. It accelerate olap queries > quite a lot. But as the number of drill user is getting bigger and bigger, a > problem we called user mix-up is getting more and more serious. > Let me explain the problem. We are using drill with user impersonation. > Different user have different privileges. As we have many drill user, so is > it quite common two or more people are using drill at the same time. A user > we called u1 posted a query to table t1 located in hdfs which he has > privilege through drill and may get an error which tells him that he has no > privilege to the table as he was u2 ( another user). " And u2 may get a > similar error with his query. The only thing u1 can do in this situation is > to exit drill and reconnect to drill through a new session. > This problem occurs quite frequently. It occur in apache-drill-1.8.0 and > also in apache-drill-1.11.0. User get confused and maybe frustrated while > data security is under threaten. > PS: we are running drill on a 8 nodes cluster which will connect to a 100 > nodes hadoop cluster. Hadoop version is 2.6.3. Drill version is 1.11.0. Below > is drill-override.conf > drill.exec: { > cluster-id: "olap-drill", > zk.connect: > "zk01:2181/olap_drill,zk02:2181/olap_drill,zk03:2181/olap_drill", >security.user.auth: {enabled: true, > packages += "org.apache.drill.exec.rpc.user.security", > impl: "pam" > pam_profiles: ["login", "sudo"] >}, >impersonation: { > enabled: true, > max_chained_user_hops: 3 >} > } > Thanks for your attention. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (DRILL-5166) Select with options returns NPE
[ https://issues.apache.org/jira/browse/DRILL-5166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva updated DRILL-5166: Fix Version/s: Future > Select with options returns NPE > --- > > Key: DRILL-5166 > URL: https://issues.apache.org/jira/browse/DRILL-5166 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.9.0 >Reporter: Arina Ielchiieva >Assignee: Arina Ielchiieva > Fix For: Future > > > When querying two csv files: > First file (2 records): > {noformat} > key_header, value_header > key_1,value_1 > {noformat} > Second file (50 records): > {noformat} > key_header, value_header > key_1,value_1 > ... > key_49,value_49 > {noformat} > Select with options returns NPE: > {noformat} > select * from table(dfs.root.`/home/arina/files/ver/*.csv`(type => > 'text',extractHeader => true, fieldDelimiter => ',')) limit 10; > {noformat} > Querying without options works file: > {noformat} > select * from dfs.root.`/home/arina/files/ver/*.csv` limit 10; > {noformat} > Error: > {noformat} > Caused by: org.apache.drill.common.exceptions.UserRemoteException: SYSTEM > ERROR: NullPointerException > Fragment 1:0 > [Error Id: b789f5f8-f090-4097-b7ff-9f4efd3d01e8 on localhost:31013] > (com.fasterxml.jackson.databind.JsonMappingException) Instantiation of > [simple type, class org.apache.drill.exec.store.dfs.easy.EasySubScan] value > failed (java.lang.NullPointerException): null > at [Source: { > "pop" : "single-sender", > "@id" : 0, > "receiver-major-fragment" : 0, > "receiver-minor-fragment" : 0, > "child" : { > "pop" : "selection-vector-remover", > "@id" : 1, > "child" : { > "pop" : "limit", > "@id" : 2, > "child" : { > "pop" : "fs-sub-scan", > "@id" : 3, > "userName" : "arina", > "files" : [ { > "start" : 0, > "length" : 11777804, > "path" : "file:/home/arina/files/ver/key_value_50.csv" > } ], > "storage" : { > "type" : "file", > "enabled" : true, > "connection" : "file:///", > "config" : null, > "workspaces" : { > "root" : { > "location" : "/", > "writable" : false, > "defaultInputFormat" : null > }, > "tmp" : { > "location" : "/tmp", > "writable" : false, > "defaultInputFormat" : null > } > }, > "formats" : { > "psv" : { > "type" : "text", > "extensions" : [ "tbl" ], > "delimiter" : "|" > }, > "csv" : { > "type" : "text", > "extensions" : [ "csv" ], > "delimiter" : "," > }, > "tsv" : { > "type" : "text", > "extensions" : [ "tsv" ], > "delimiter" : "\t" > }, > "httpd" : { > "type" : "httpd", > "logFormat" : "%h %t \"%r\" %>s %b \"%{Referer}i\"", > "timestampFormat" : null > }, > "parquet" : { > "type" : "parquet" > }, > "json" : { > "type" : "json", > "extensions" : [ "json" ] > }, > "avro" : { > "type" : "avro" > }, > "sequencefile" : { > "type" : "sequencefile", > "extensions" : [ "seq" ] > }, > "csvh" : { > "type" : "text", > "extensions" : [ "csvh" ], > "extractHeader" : true, > "delimiter" : "," > } > } > }, > "format" : { > "type" : "named", > "name" : "text" > }, > "columns" : [ "`*`" ], > "selectionRoot" : "file:/home/arina/files/ver", > "initialAllocation" : 100, > "maxAllocation" : 100, > "cost" : 0.0 > }, > "first" : 0, > "last" : 10, > "initialAllocation" : 100, > "maxAllocation" : 100, > "cost" : 10.0 > }, > "initialAllocation" : 100, > "maxAllocation" : 100, > "cost" : 10.0 > }, > "destination" : "Cglsb2NhbGhvc3QQpfIBGKbyASCn8gEyDzEuMTAuMC1TTkFQU0hPVA==", > "initialAllocation" : 100, > "maxAllocation" : 100, > "cost" : 10.0 > }; line: 90, column: 7] (through reference chain: > org.apache.drill.exec.physical.config.SingleSender["child"]->org.apache.drill.exec.physical.config.SelectionVectorRemover["child"]->org.apache.drill.exec.physical.config.Limit["child"]) > com.fasterxml.jackson.databind.JsonMappingException.from():223 > > com.f
[jira] [Updated] (DRILL-5166) Select with options returns NPE
[ https://issues.apache.org/jira/browse/DRILL-5166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva updated DRILL-5166: Fix Version/s: (was: Future) > Select with options returns NPE > --- > > Key: DRILL-5166 > URL: https://issues.apache.org/jira/browse/DRILL-5166 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.9.0 >Reporter: Arina Ielchiieva >Assignee: Arina Ielchiieva > Fix For: Future > > > When querying two csv files: > First file (2 records): > {noformat} > key_header, value_header > key_1,value_1 > {noformat} > Second file (50 records): > {noformat} > key_header, value_header > key_1,value_1 > ... > key_49,value_49 > {noformat} > Select with options returns NPE: > {noformat} > select * from table(dfs.root.`/home/arina/files/ver/*.csv`(type => > 'text',extractHeader => true, fieldDelimiter => ',')) limit 10; > {noformat} > Querying without options works file: > {noformat} > select * from dfs.root.`/home/arina/files/ver/*.csv` limit 10; > {noformat} > Error: > {noformat} > Caused by: org.apache.drill.common.exceptions.UserRemoteException: SYSTEM > ERROR: NullPointerException > Fragment 1:0 > [Error Id: b789f5f8-f090-4097-b7ff-9f4efd3d01e8 on localhost:31013] > (com.fasterxml.jackson.databind.JsonMappingException) Instantiation of > [simple type, class org.apache.drill.exec.store.dfs.easy.EasySubScan] value > failed (java.lang.NullPointerException): null > at [Source: { > "pop" : "single-sender", > "@id" : 0, > "receiver-major-fragment" : 0, > "receiver-minor-fragment" : 0, > "child" : { > "pop" : "selection-vector-remover", > "@id" : 1, > "child" : { > "pop" : "limit", > "@id" : 2, > "child" : { > "pop" : "fs-sub-scan", > "@id" : 3, > "userName" : "arina", > "files" : [ { > "start" : 0, > "length" : 11777804, > "path" : "file:/home/arina/files/ver/key_value_50.csv" > } ], > "storage" : { > "type" : "file", > "enabled" : true, > "connection" : "file:///", > "config" : null, > "workspaces" : { > "root" : { > "location" : "/", > "writable" : false, > "defaultInputFormat" : null > }, > "tmp" : { > "location" : "/tmp", > "writable" : false, > "defaultInputFormat" : null > } > }, > "formats" : { > "psv" : { > "type" : "text", > "extensions" : [ "tbl" ], > "delimiter" : "|" > }, > "csv" : { > "type" : "text", > "extensions" : [ "csv" ], > "delimiter" : "," > }, > "tsv" : { > "type" : "text", > "extensions" : [ "tsv" ], > "delimiter" : "\t" > }, > "httpd" : { > "type" : "httpd", > "logFormat" : "%h %t \"%r\" %>s %b \"%{Referer}i\"", > "timestampFormat" : null > }, > "parquet" : { > "type" : "parquet" > }, > "json" : { > "type" : "json", > "extensions" : [ "json" ] > }, > "avro" : { > "type" : "avro" > }, > "sequencefile" : { > "type" : "sequencefile", > "extensions" : [ "seq" ] > }, > "csvh" : { > "type" : "text", > "extensions" : [ "csvh" ], > "extractHeader" : true, > "delimiter" : "," > } > } > }, > "format" : { > "type" : "named", > "name" : "text" > }, > "columns" : [ "`*`" ], > "selectionRoot" : "file:/home/arina/files/ver", > "initialAllocation" : 100, > "maxAllocation" : 100, > "cost" : 0.0 > }, > "first" : 0, > "last" : 10, > "initialAllocation" : 100, > "maxAllocation" : 100, > "cost" : 10.0 > }, > "initialAllocation" : 100, > "maxAllocation" : 100, > "cost" : 10.0 > }, > "destination" : "Cglsb2NhbGhvc3QQpfIBGKbyASCn8gEyDzEuMTAuMC1TTkFQU0hPVA==", > "initialAllocation" : 100, > "maxAllocation" : 100, > "cost" : 10.0 > }; line: 90, column: 7] (through reference chain: > org.apache.drill.exec.physical.config.SingleSender["child"]->org.apache.drill.exec.physical.config.SelectionVectorRemover["child"]->org.apache.drill.exec.physical.config.Limit["child"]) > com.fasterxml.jackson.databind.JsonMappingException.from():223 >
[jira] [Commented] (DRILL-5822) The query with "SELECT *" with "ORDER BY" clause and `planner.slice_target`=1 doesn't preserve column order
[ https://issues.apache.org/jira/browse/DRILL-5822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16227349#comment-16227349 ] Vitalii Diravka commented on DRILL-5822: [~Paul.Rogers] Here is the issue of unnecessary sorting of columns for query with the following conditions: using wildcard in the query and ORDER BY clause, and when this is planned into multiple fragments ("alter session set `planner.slice_target`=1;"). The issue is connected to adding canonicalizing the schemas of input batches for Merging Receiver in DRILL-847. But this approach is outdated since for now in the process of loading batches in the RecordBatchLoader the new batch with same columns (SchemaPaths) but other ordering of them is perceived with same schema as the previous batch has: [All fields from the last batch is a hashMap structure|https://github.com/apache/drill/blob/fe79a633a3da8b4f6db50454fde64c30c73233bb/exec/java-exec/src/main/java/org/apache/drill/exec/record/RecordBatchLoader.java#L90] and [when new batch appears the columns are just removed from the old one by the key|https://github.com/apache/drill/blob/fe79a633a3da8b4f6db50454fde64c30c73233bb/exec/java-exec/src/main/java/org/apache/drill/exec/record/RecordBatchLoader.java#L102]. So the schemaChange flag still equals to false. And then [the schema will built|https://github.com/apache/drill/blob/fe79a633a3da8b4f6db50454fde64c30c73233bb/exec/java-exec/src/main/java/org/apache/drill/exec/record/RecordBatchLoader.java#L138]. Here is only the issue that RecordBatchLoader permutes column order for the above case. And it was described in the jira ticket created by you DRILL-5828 and can be fixed there. So my changes fix the current issue but not fully cover the requirements from your comment. Will It be reasonably if that changes will be done in context of DRILL-5828? > The query with "SELECT *" with "ORDER BY" clause and `planner.slice_target`=1 > doesn't preserve column order > --- > > Key: DRILL-5822 > URL: https://issues.apache.org/jira/browse/DRILL-5822 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.11.0 >Reporter: Prasad Nagaraj Subramanya >Assignee: Vitalii Diravka > Fix For: 1.12.0 > > > Columns ordering doesn't preserve for the star query with sorting when this > is planned into multiple fragments. > Repro steps: > 1) {code}alter session set `planner.slice_target`=1;{code} > 2) ORDER BY clause in the query. > Scenarios: > {code} > 0: jdbc:drill:zk=local> alter session reset `planner.slice_target`; > +---++ > | ok |summary | > +---++ > | true | planner.slice_target updated. | > +---++ > 1 row selected (0.082 seconds) > 0: jdbc:drill:zk=local> select * from cp.`tpch/nation.parquet` order by > n_name limit 1; > +--+--+--+--+ > | n_nationkey | n_name | n_regionkey | n_comment > | > +--+--+--+--+ > | 0| ALGERIA | 0| haggle. carefully final deposits > detect slyly agai | > +--+--+--+--+ > 1 row selected (0.141 seconds) > 0: jdbc:drill:zk=local> alter session set `planner.slice_target`=1; > +---++ > | ok |summary | > +---++ > | true | planner.slice_target updated. | > +---++ > 1 row selected (0.091 seconds) > 0: jdbc:drill:zk=local> select * from cp.`tpch/nation.parquet` order by > n_name limit 1; > +--+--+--+--+ > | n_comment | n_name | > n_nationkey | n_regionkey | > +--+--+--+--+ > | haggle. carefully final deposits detect slyly agai | ALGERIA | 0 >| 0| > +--+--+--+--+ > 1 row selected (0.201 seconds) > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (DRILL-5918) Allow manual configuration when using Runtime.getRuntime().availableProcessors()
Elijah Zupancic created DRILL-5918: -- Summary: Allow manual configuration when using Runtime.getRuntime().availableProcessors() Key: DRILL-5918 URL: https://issues.apache.org/jira/browse/DRILL-5918 Project: Apache Drill Issue Type: Improvement Affects Versions: 1.11.0 Reporter: Elijah Zupancic Priority: Critical Currently Drill auto-configures the number of threads in various thread pools based on the processor count: https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/server/BootStrapContext.java#L110 https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/server/options/TypeValidators.java#L269 https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/rm/AbstractResourceManager.java#L55 https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/rm/DefaultResourceManager.java#L99 In a number of situations, this configuration is incorrect. In particular, the settings aren't correct when [running in a bare-metal container](https://docs.google.com/document/d/1WGPGiJtbJZPZBMSD9YHAAz-O4AlAET9qOnS_GeriZtE/edit?usp=sharing) because you don't necessarily have access to all of the compute shares or memory of the underlying host. Ideally, the number of CPUs to use could be specified as a system parameter to Drill. This would allow users to configure Drill to their optimal settings. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5863) Sortable table incorrectly sorts minor fragments and time elements lexically instead of sorting by implicit value
[ https://issues.apache.org/jira/browse/DRILL-5863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16227527#comment-16227527 ] ASF GitHub Bot commented on DRILL-5863: --- Github user kkhatua commented on a diff in the pull request: https://github.com/apache/drill/pull/987#discussion_r148123715 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/profile/OperatorWrapper.java --- @@ -76,12 +77,14 @@ public String getId() { public String getContent() { TableBuilder builder = new TableBuilder(OPERATOR_COLUMNS, OPERATOR_COLUMNS_TOOLTIP, true); +Map attributeMap = new HashMap(); //Reusing for different fragments for (ImmutablePair, String> ip : opsAndHosts) { int minor = ip.getLeft().getRight(); OperatorProfile op = ip.getLeft().getLeft(); + attributeMap.put("data-order", String.valueOf(minor)); //Overwrite values from previous fragments String path = new OperatorPathBuilder().setMajor(major).setMinor(minor).setOperator(op).build(); - builder.appendCell(path); + builder.appendCell(path, attributeMap); --- End diff -- The appendCell() method has been overloaded with to allow for a map, from which any set of attributes can be injected. https://github.com/kkhatua/drill/blob/783ca992f3bbb52ae606a8ca10745d6dd2ed0d4f/exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/profile/TableBuilder.java#L95 This allows for any future libraries that might need to inject their own set of attributes to be incorporated without the need for any significant modification to the TableBuilder. So, each appendCell is provided its own unique map that is consumed during the construction of the `` element > Sortable table incorrectly sorts minor fragments and time elements lexically > instead of sorting by implicit value > - > > Key: DRILL-5863 > URL: https://issues.apache.org/jira/browse/DRILL-5863 > Project: Apache Drill > Issue Type: Bug > Components: Web Server >Affects Versions: 1.12.0 >Reporter: Kunal Khatua >Assignee: Kunal Khatua >Priority: Minor > Fix For: 1.12.0 > > > The fix for this is to use dataTable library's {{data-order}} attribute for > the data elements that need to sort by an implicit value. > ||Old order of Minor Fragment||New order of Minor Fragment|| > |...|...| > |01-09-01 | 01-09-01| > |01-10-01 | 01-10-01| > |01-100-01 | 01-11-01| > |01-101-01 | 01-12-01| > |... | ... | > ||Old order of Duration||New order of Duration||| > |...|...| > |1m15s | 55.03s| > |55s | 1m15s| > |...|...| -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5863) Sortable table incorrectly sorts minor fragments and time elements lexically instead of sorting by implicit value
[ https://issues.apache.org/jira/browse/DRILL-5863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16227565#comment-16227565 ] ASF GitHub Bot commented on DRILL-5863: --- Github user kkhatua commented on a diff in the pull request: https://github.com/apache/drill/pull/987#discussion_r148126237 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/profile/OperatorWrapper.java --- @@ -76,12 +77,14 @@ public String getId() { public String getContent() { TableBuilder builder = new TableBuilder(OPERATOR_COLUMNS, OPERATOR_COLUMNS_TOOLTIP, true); +Map attributeMap = new HashMap(); //Reusing for different fragments for (ImmutablePair, String> ip : opsAndHosts) { int minor = ip.getLeft().getRight(); OperatorProfile op = ip.getLeft().getLeft(); + attributeMap.put("data-order", String.valueOf(minor)); //Overwrite values from previous fragments --- End diff -- You are correct that the values must be zero padded. However, the padding by default is only up till achieving a width of two digits. The sequencing didn't get messed up as long as the number of minor fragments for each major fragment was less than 100. For greater than 100, the sequencing broke the way you described it. So, until this fix, you'll get the minor fragment ordered as: 01-01-XX . 01-10-XX, 01-100-XX, instead of 01-01-XX . 01-10-XX, 01-11-XX, 01-10-XX, 01-11-XX, 01-100-XX, 01-101-XX What we are doing is injecting the element with an attribute that carries the actual numeric value. The dataTable library will discover and sort the elements of a column by the value of this attribute instead of the value that the element encapsulates. > Sortable table incorrectly sorts minor fragments and time elements lexically > instead of sorting by implicit value > - > > Key: DRILL-5863 > URL: https://issues.apache.org/jira/browse/DRILL-5863 > Project: Apache Drill > Issue Type: Bug > Components: Web Server >Affects Versions: 1.12.0 >Reporter: Kunal Khatua >Assignee: Kunal Khatua >Priority: Minor > Fix For: 1.12.0 > > > The fix for this is to use dataTable library's {{data-order}} attribute for > the data elements that need to sort by an implicit value. > ||Old order of Minor Fragment||New order of Minor Fragment|| > |...|...| > |01-09-01 | 01-09-01| > |01-10-01 | 01-10-01| > |01-100-01 | 01-11-01| > |01-101-01 | 01-12-01| > |... | ... | > ||Old order of Duration||New order of Duration||| > |...|...| > |1m15s | 55.03s| > |55s | 1m15s| > |...|...| -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5468) TPCH Q18 regressed ~3x due to execution plan changes
[ https://issues.apache.org/jira/browse/DRILL-5468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16227570#comment-16227570 ] Dechang Gu commented on DRILL-5468: --- I tried this setting on SF100, Q18 did not show the 3x regression, performance is on par with previous normal runs. > TPCH Q18 regressed ~3x due to execution plan changes > > > Key: DRILL-5468 > URL: https://issues.apache.org/jira/browse/DRILL-5468 > Project: Apache Drill > Issue Type: Bug > Components: Functions - Drill >Affects Versions: 1.11.0 > Environment: 10+1 node ucs-micro cluster RHEL6.4 >Reporter: Dechang Gu >Assignee: Jinfeng Ni > Attachments: Q18_profile_gitid_841ead4, Q18_profile_gitid_adbf363 > > > In a regular regression test on Drill master (commit id 841ead4) TPCH Q18 on > SF100 parquet dataset took ~81 secs, while the same query on 1.10.0 took only > ~27 secs. The query time on the commit adbf363 which is right before 841ead4 > is ~32 secs. > Profiles shows the plans for the query changed quite a bit (profiles will be > uploaded) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5783) Make code generation in the TopN operator more modular and test it
[ https://issues.apache.org/jira/browse/DRILL-5783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16227708#comment-16227708 ] ASF GitHub Bot commented on DRILL-5783: --- Github user ilooner commented on a diff in the pull request: https://github.com/apache/drill/pull/984#discussion_r148144235 --- Diff: exec/java-exec/src/test/java/org/apache/drill/test/BaseDirTestWatcher.java --- @@ -0,0 +1,184 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.drill.test; + +import com.google.common.base.Charsets; +import org.apache.commons.io.FileUtils; +import org.apache.drill.common.util.TestTools; +import org.apache.drill.exec.util.TestUtilities; +import org.junit.runner.Description; + +import java.io.File; +import java.io.IOException; + +public class BaseDirTestWatcher extends DirTestWatcher { + public enum DirType { +ROOT, +TEST_TMP; + } + + private File tmpDir; + private File storeDir; + private File dfsTestTmpParentDir; + private File dfsTestTmpDir; + private File rootDir; + + public BaseDirTestWatcher() { +super(); + } + + @Override + protected void starting(Description description) { +super.starting(description); + +rootDir = makeSubDir("root"); +tmpDir = new File(rootDir, "tmp"); +storeDir = new File(rootDir, "store"); +dfsTestTmpParentDir = new File(rootDir, "dfsTestTmp"); + +tmpDir.mkdirs(); +storeDir.mkdirs(); +dfsTestTmpParentDir.mkdirs(); + } + + public File getTmpDir() { +return tmpDir; + } + + public File getStoreDir() { +return storeDir; + } + + public File getDfsTestTmpParentDir() { +return dfsTestTmpParentDir; + } + + public File getDfsTestTmpDir() { +return dfsTestTmpDir; + } + + public String getDfsTestTmpDirPath() { +return dfsTestTmpDir.getAbsolutePath().replaceAll("/\\./", "/"); + } + + public File getRootDir() { +return rootDir; + } + + public String getRootDirPath() { +return rootDir.getAbsolutePath().replaceAll("/\\./", "/"); + } + + public void newDfsTestTmpDir() { +dfsTestTmpDir = TestUtilities.createTempDir(BaseTestQuery.dirTestWatcher.getDfsTestTmpParentDir()); + } + + private File getDir(DirType type) { +switch (type) { + case ROOT: +return rootDir; + case TEST_TMP: +return dfsTestTmpDir; + default: +throw new IllegalArgumentException(String.format("Unsupported type %s", type)); +} + } + + public File makeRootSubDir(String relPath) { +return makeSubDir(relPath, DirType.ROOT); + } + + public File makeTestTmpSubDir(String relPath) { +return makeSubDir(relPath, DirType.TEST_TMP); + } + + private File makeSubDir(String relPath, DirType type) { +File subDir = new File(getDir(type), relPath); +subDir.mkdirs(); +return subDir; + } + + /** + * This preserves the relative path of the directory in root + * @param relPath + * @return + */ + public File copyResourceToRoot(String relPath) { +return copyTo(relPath, relPath, TestTools.DataType.RESOURCE, DirType.ROOT); + } + + public File copyFileToRoot(String relPath) { +return copyTo(relPath, relPath, TestTools.DataType.PROJECT, DirType.ROOT); + } + + public File copyResourceToRoot(String relPath, String destPath) { +return copyTo(relPath, destPath, TestTools.DataType.RESOURCE, DirType.ROOT); + } + + public File copyResourceToTestTmp(String relPath, String destPath) { +return copyTo(relPath, destPath, TestTools.DataType.RESOURCE, DirType.TEST_TMP); + } + + private File copyTo(String relPath, String destPath, TestTools
[jira] [Commented] (DRILL-5783) Make code generation in the TopN operator more modular and test it
[ https://issues.apache.org/jira/browse/DRILL-5783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16227797#comment-16227797 ] ASF GitHub Bot commented on DRILL-5783: --- Github user ilooner commented on a diff in the pull request: https://github.com/apache/drill/pull/984#discussion_r148157140 --- Diff: exec/java-exec/src/test/java/org/apache/drill/test/BatchUtils.java --- @@ -0,0 +1,280 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.drill.test; + +import com.google.common.collect.Lists; +import com.google.common.collect.Maps; +import org.apache.drill.exec.record.VectorContainer; +import org.apache.drill.exec.record.VectorWrapper; +import org.apache.drill.exec.record.selection.SelectionVector4; +import org.apache.drill.exec.vector.ValueVector; +import org.junit.Assert; + +import java.io.UnsupportedEncodingException; +import java.util.Arrays; +import java.util.Collections; +import java.util.Comparator; +import java.util.List; +import java.util.Map; + +public class BatchUtils { + static final org.slf4j.Logger logger = org.slf4j.LoggerFactory.getLogger(BatchUtils.class); + + public static Map> containerToObjects(VectorContainer vectorContainer) { +Map> rows = Maps.newHashMap(); +int numCols = vectorContainer.getNumberOfColumns(); +int recordCount = vectorContainer.getRecordCount(); + +for (int columnIndex = 0; columnIndex < numCols; columnIndex++) { + String columnName = vectorContainer.getSchema().getColumn(columnIndex).getName(); + List data = Lists.newArrayList(); + + ValueVector.Accessor valueVectorAccessor = vectorContainer +.getValueVector(columnIndex) +.getValueVector() +.getAccessor(); + + for (int recordIndex = 0; recordIndex < recordCount; recordIndex++) { +data.add(valueVectorAccessor.getObject(recordIndex)); + } + + rows.put(columnName, data); +} + +return rows; + } + + public static Map> hyperBatchAndSelectorToObjects(VectorContainer vectorContainer, SelectionVector4 selectionVector4) { --- End diff -- Done > Make code generation in the TopN operator more modular and test it > -- > > Key: DRILL-5783 > URL: https://issues.apache.org/jira/browse/DRILL-5783 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5783) Make code generation in the TopN operator more modular and test it
[ https://issues.apache.org/jira/browse/DRILL-5783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16227798#comment-16227798 ] ASF GitHub Bot commented on DRILL-5783: --- Github user ilooner commented on a diff in the pull request: https://github.com/apache/drill/pull/984#discussion_r148157170 --- Diff: exec/java-exec/src/test/java/org/apache/drill/test/BatchUtils.java --- @@ -0,0 +1,280 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.drill.test; + +import com.google.common.collect.Lists; +import com.google.common.collect.Maps; +import org.apache.drill.exec.record.VectorContainer; +import org.apache.drill.exec.record.VectorWrapper; +import org.apache.drill.exec.record.selection.SelectionVector4; +import org.apache.drill.exec.vector.ValueVector; +import org.junit.Assert; + +import java.io.UnsupportedEncodingException; +import java.util.Arrays; +import java.util.Collections; +import java.util.Comparator; +import java.util.List; +import java.util.Map; + +public class BatchUtils { + static final org.slf4j.Logger logger = org.slf4j.LoggerFactory.getLogger(BatchUtils.class); + + public static Map> containerToObjects(VectorContainer vectorContainer) { +Map> rows = Maps.newHashMap(); +int numCols = vectorContainer.getNumberOfColumns(); +int recordCount = vectorContainer.getRecordCount(); + +for (int columnIndex = 0; columnIndex < numCols; columnIndex++) { + String columnName = vectorContainer.getSchema().getColumn(columnIndex).getName(); + List data = Lists.newArrayList(); + + ValueVector.Accessor valueVectorAccessor = vectorContainer +.getValueVector(columnIndex) +.getValueVector() +.getAccessor(); + + for (int recordIndex = 0; recordIndex < recordCount; recordIndex++) { +data.add(valueVectorAccessor.getObject(recordIndex)); + } + + rows.put(columnName, data); +} + +return rows; + } + + public static Map> hyperBatchAndSelectorToObjects(VectorContainer vectorContainer, SelectionVector4 selectionVector4) { +Map> rows = Maps.newHashMap(); +int numCols = vectorContainer.getNumberOfColumns(); +int numIndices = selectionVector4.getCount(); + +for (int columnIndex = 0; columnIndex < numCols; columnIndex++) { + String columnName = vectorContainer.getSchema().getColumn(columnIndex).getName(); + List data = Lists.newArrayList(); + + VectorWrapper vectorWrapper = vectorContainer.getValueVector(columnIndex); + + for (int indexIndex = 0; indexIndex < numIndices; indexIndex++) { +int sv4Index = selectionVector4.get(indexIndex); +int batchIndex = SelectionVector4.getBatchIndex(sv4Index); +int recordIndex = SelectionVector4.getRecordIndex(sv4Index); + +ValueVector valueVector = vectorWrapper.getValueVectors()[batchIndex]; +Object columnValue = valueVector.getAccessor().getObject(recordIndex); +data.add(columnValue); + } + + rows.put(columnName, data); +} + +return rows; + } + + public static String toString(Map> table) { +if (table.isEmpty()) { + return "[ empty table ]"; +} + +List columnNames = Lists.newArrayList(table.keySet()); +Collections.sort(columnNames); +int numRecords = table.get(columnNames.get(0)).size(); + +StringBuilder sb = new StringBuilder(); + +{ + sb.append("[ "); + String separator = ""; + + for (String columnName : columnNames) { +sb.append(separator); +separator = ", "; +sb.append(columnName); + } + + sb.append(" ]\n"); +} + +for (int recordIndex = 0; recordIndex < numRecords; recordIndex++)
[jira] [Commented] (DRILL-5783) Make code generation in the TopN operator more modular and test it
[ https://issues.apache.org/jira/browse/DRILL-5783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16227807#comment-16227807 ] ASF GitHub Bot commented on DRILL-5783: --- Github user ilooner commented on a diff in the pull request: https://github.com/apache/drill/pull/984#discussion_r148157850 --- Diff: exec/java-exec/src/test/java/org/apache/drill/test/BatchUtils.java --- @@ -0,0 +1,280 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.drill.test; + +import com.google.common.collect.Lists; +import com.google.common.collect.Maps; +import org.apache.drill.exec.record.VectorContainer; +import org.apache.drill.exec.record.VectorWrapper; +import org.apache.drill.exec.record.selection.SelectionVector4; +import org.apache.drill.exec.vector.ValueVector; +import org.junit.Assert; + +import java.io.UnsupportedEncodingException; +import java.util.Arrays; +import java.util.Collections; +import java.util.Comparator; +import java.util.List; +import java.util.Map; + +public class BatchUtils { + static final org.slf4j.Logger logger = org.slf4j.LoggerFactory.getLogger(BatchUtils.class); + + public static Map> containerToObjects(VectorContainer vectorContainer) { +Map> rows = Maps.newHashMap(); +int numCols = vectorContainer.getNumberOfColumns(); +int recordCount = vectorContainer.getRecordCount(); + +for (int columnIndex = 0; columnIndex < numCols; columnIndex++) { + String columnName = vectorContainer.getSchema().getColumn(columnIndex).getName(); + List data = Lists.newArrayList(); + + ValueVector.Accessor valueVectorAccessor = vectorContainer +.getValueVector(columnIndex) +.getValueVector() +.getAccessor(); + + for (int recordIndex = 0; recordIndex < recordCount; recordIndex++) { +data.add(valueVectorAccessor.getObject(recordIndex)); + } + + rows.put(columnName, data); +} + +return rows; + } + + public static Map> hyperBatchAndSelectorToObjects(VectorContainer vectorContainer, SelectionVector4 selectionVector4) { +Map> rows = Maps.newHashMap(); +int numCols = vectorContainer.getNumberOfColumns(); +int numIndices = selectionVector4.getCount(); + +for (int columnIndex = 0; columnIndex < numCols; columnIndex++) { + String columnName = vectorContainer.getSchema().getColumn(columnIndex).getName(); + List data = Lists.newArrayList(); + + VectorWrapper vectorWrapper = vectorContainer.getValueVector(columnIndex); + + for (int indexIndex = 0; indexIndex < numIndices; indexIndex++) { +int sv4Index = selectionVector4.get(indexIndex); +int batchIndex = SelectionVector4.getBatchIndex(sv4Index); +int recordIndex = SelectionVector4.getRecordIndex(sv4Index); + +ValueVector valueVector = vectorWrapper.getValueVectors()[batchIndex]; +Object columnValue = valueVector.getAccessor().getObject(recordIndex); +data.add(columnValue); + } + + rows.put(columnName, data); +} + +return rows; + } + + public static String toString(Map> table) { +if (table.isEmpty()) { + return "[ empty table ]"; +} + +List columnNames = Lists.newArrayList(table.keySet()); +Collections.sort(columnNames); +int numRecords = table.get(columnNames.get(0)).size(); + +StringBuilder sb = new StringBuilder(); + +{ + sb.append("[ "); + String separator = ""; + + for (String columnName : columnNames) { +sb.append(separator); +separator = ", "; +sb.append(columnName); + } + + sb.append(" ]\n"); +} + +for (int recordIndex = 0; recordIndex < numRecords; recordIndex++)
[jira] [Commented] (DRILL-5842) Refactor and simplify the fragment, operator contexts for testing
[ https://issues.apache.org/jira/browse/DRILL-5842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16227812#comment-16227812 ] ASF GitHub Bot commented on DRILL-5842: --- Github user sohami commented on a diff in the pull request: https://github.com/apache/drill/pull/978#discussion_r148150382 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/ops/BaseOperatorContext.java --- @@ -0,0 +1,201 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.drill.exec.ops; + +import java.io.IOException; +import java.util.concurrent.ExecutorService; + +import org.apache.drill.common.exceptions.UserException; +import org.apache.drill.exec.memory.BufferAllocator; +import org.apache.drill.exec.physical.base.PhysicalOperator; +import org.apache.drill.exec.store.dfs.DrillFileSystem; +import org.apache.drill.exec.testing.ControlsInjector; +import org.apache.drill.exec.testing.ExecutionControls; +import org.apache.hadoop.conf.Configuration; + +import com.google.common.base.Preconditions; + +import io.netty.buffer.DrillBuf; + +/** + * Implementation of {@link OperatorExecContext} that provides services --- End diff -- `OperatorContext` instead of _OperatorExecContext_ > Refactor and simplify the fragment, operator contexts for testing > - > > Key: DRILL-5842 > URL: https://issues.apache.org/jira/browse/DRILL-5842 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.12.0 >Reporter: Paul Rogers >Assignee: Paul Rogers > Fix For: 1.12.0 > > > Drill's execution engine has a "fragment context" that provides state for a > fragment as a whole, and an "operator context" which provides state for a > single operator. Historically, these have both been concrete classes that > make generous references to the Drillbit context, and hence need a full Drill > server in order to operate. > Drill has historically made extensive use of system-level testing: build the > entire server and fire queries at it to test each component. Over time, we > are augmenting that approach with unit tests: the ability to test each > operator (or parts of an operator) in isolation. > Since each operator requires access to both the operator and fragment > context, the fact that the contexts depend on the overall server creates a > large barrier to unit testing. An earlier checkin started down the path of > defining the contexts as interfaces that can have different run-time and > test-time implementations to enable testing. > This ticket asks to refactor those interfaces: simplifying the operator > context and introducing an interface for the fragment context. New code will > use these new interfaces, while older code continues to use the concrete > implementations. Over time, as operators are enhanced, they can be modified > to allow unit-level testing. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5842) Refactor and simplify the fragment, operator contexts for testing
[ https://issues.apache.org/jira/browse/DRILL-5842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16227813#comment-16227813 ] ASF GitHub Bot commented on DRILL-5842: --- Github user sohami commented on a diff in the pull request: https://github.com/apache/drill/pull/978#discussion_r148153093 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/ops/OperatorContextImpl.java --- @@ -98,37 +70,25 @@ public boolean isClosed() { @Override public void close() { if (closed) { - logger.debug("Attempted to close Operator context for {}, but context is already closed", popConfig != null ? popConfig.getClass().getName() : null); + logger.debug("Attempted to close Operator context for {}, but context is already closed", popConfig != null ? getName() : null); return; } -logger.debug("Closing context for {}", popConfig != null ? popConfig.getClass().getName() : null); +logger.debug("Closing context for {}", popConfig != null ? getName() : null); closed = true; --- End diff -- `closed = true` should happen after the call to `super.close()` since base class close can throw exception. > Refactor and simplify the fragment, operator contexts for testing > - > > Key: DRILL-5842 > URL: https://issues.apache.org/jira/browse/DRILL-5842 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.12.0 >Reporter: Paul Rogers >Assignee: Paul Rogers > Fix For: 1.12.0 > > > Drill's execution engine has a "fragment context" that provides state for a > fragment as a whole, and an "operator context" which provides state for a > single operator. Historically, these have both been concrete classes that > make generous references to the Drillbit context, and hence need a full Drill > server in order to operate. > Drill has historically made extensive use of system-level testing: build the > entire server and fire queries at it to test each component. Over time, we > are augmenting that approach with unit tests: the ability to test each > operator (or parts of an operator) in isolation. > Since each operator requires access to both the operator and fragment > context, the fact that the contexts depend on the overall server creates a > large barrier to unit testing. An earlier checkin started down the path of > defining the contexts as interfaces that can have different run-time and > test-time implementations to enable testing. > This ticket asks to refactor those interfaces: simplifying the operator > context and introducing an interface for the fragment context. New code will > use these new interfaces, while older code continues to use the concrete > implementations. Over time, as operators are enhanced, they can be modified > to allow unit-level testing. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5842) Refactor and simplify the fragment, operator contexts for testing
[ https://issues.apache.org/jira/browse/DRILL-5842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16227814#comment-16227814 ] ASF GitHub Bot commented on DRILL-5842: --- Github user sohami commented on a diff in the pull request: https://github.com/apache/drill/pull/978#discussion_r148146597 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/ops/OperatorContextImpl.java --- @@ -17,33 +17,23 @@ */ package org.apache.drill.exec.ops; -import java.io.IOException; import java.security.PrivilegedExceptionAction; import java.util.concurrent.Callable; -import java.util.concurrent.ExecutorService; -import org.apache.drill.common.exceptions.DrillRuntimeException; import org.apache.drill.exec.exception.OutOfMemoryException; import org.apache.drill.exec.physical.base.PhysicalOperator; -import org.apache.drill.exec.store.dfs.DrillFileSystem; import org.apache.drill.exec.work.WorkManager; -import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.security.UserGroupInformation; -import com.google.common.base.Preconditions; import com.google.common.util.concurrent.ListenableFuture; import com.google.common.util.concurrent.ListeningExecutorService; import com.google.common.util.concurrent.MoreExecutors; -class OperatorContextImpl extends AbstractOperatorExecContext implements OperatorContext, AutoCloseable { +class OperatorContextImpl extends BaseOperatorContext implements OperatorContext, AutoCloseable { static final org.slf4j.Logger logger = org.slf4j.LoggerFactory.getLogger(OperatorContextImpl.class); --- End diff -- Why we need to have `OperatorContextImpl implements OperatorContext` when `BaseOperatorContex`t already implements that interface ? > Refactor and simplify the fragment, operator contexts for testing > - > > Key: DRILL-5842 > URL: https://issues.apache.org/jira/browse/DRILL-5842 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.12.0 >Reporter: Paul Rogers >Assignee: Paul Rogers > Fix For: 1.12.0 > > > Drill's execution engine has a "fragment context" that provides state for a > fragment as a whole, and an "operator context" which provides state for a > single operator. Historically, these have both been concrete classes that > make generous references to the Drillbit context, and hence need a full Drill > server in order to operate. > Drill has historically made extensive use of system-level testing: build the > entire server and fire queries at it to test each component. Over time, we > are augmenting that approach with unit tests: the ability to test each > operator (or parts of an operator) in isolation. > Since each operator requires access to both the operator and fragment > context, the fact that the contexts depend on the overall server creates a > large barrier to unit testing. An earlier checkin started down the path of > defining the contexts as interfaces that can have different run-time and > test-time implementations to enable testing. > This ticket asks to refactor those interfaces: simplifying the operator > context and introducing an interface for the fragment context. New code will > use these new interfaces, while older code continues to use the concrete > implementations. Over time, as operators are enhanced, they can be modified > to allow unit-level testing. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5842) Refactor and simplify the fragment, operator contexts for testing
[ https://issues.apache.org/jira/browse/DRILL-5842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16227815#comment-16227815 ] ASF GitHub Bot commented on DRILL-5842: --- Github user sohami commented on a diff in the pull request: https://github.com/apache/drill/pull/978#discussion_r148095543 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/ops/BaseOperatorContext.java --- @@ -0,0 +1,201 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.drill.exec.ops; + +import java.io.IOException; +import java.util.concurrent.ExecutorService; + +import org.apache.drill.common.exceptions.UserException; +import org.apache.drill.exec.memory.BufferAllocator; +import org.apache.drill.exec.physical.base.PhysicalOperator; +import org.apache.drill.exec.store.dfs.DrillFileSystem; +import org.apache.drill.exec.testing.ControlsInjector; +import org.apache.drill.exec.testing.ExecutionControls; +import org.apache.hadoop.conf.Configuration; + +import com.google.common.base.Preconditions; + +import io.netty.buffer.DrillBuf; + +/** + * Implementation of {@link OperatorExecContext} that provides services + * needed by most run-time operators. Excludes services that need the + * entire Drillbit. This class provides services common to the test-time + * version of the operator context and the full production-time context + * that includes network services. + */ + +public abstract class BaseOperatorContext implements OperatorContext { + static final org.slf4j.Logger logger = org.slf4j.LoggerFactory.getLogger(BaseOperatorContext.class); + + protected final FragmentContextInterface context; + protected final BufferAllocator allocator; + protected final PhysicalOperator popConfig; + protected final BufferManager manager; + protected OperatorStatReceiver statsWriter; + private DrillFileSystem fs; + private ControlsInjector injector; + + public BaseOperatorContext(FragmentContextInterface context, BufferAllocator allocator, + PhysicalOperator popConfig, + OperatorStatReceiver stats) { +this.context = context; +this.allocator = allocator; +this.popConfig = popConfig; +this.manager = new BufferManagerImpl(allocator); +statsWriter = stats; + } + + @Override + public FragmentContextInterface getFragmentContext() { +return context; + } + + @SuppressWarnings("unchecked") + @Override + public T getOperatorDefn() { +return (T) popConfig; + } + + public String getName() { +return popConfig.getClass().getName(); + } + + @Override + public DrillBuf replace(DrillBuf old, int newSize) { +return manager.replace(old, newSize); + } + + @Override + public DrillBuf getManagedBuffer() { +return manager.getManagedBuffer(); + } + + @Override + public DrillBuf getManagedBuffer(int size) { +return manager.getManagedBuffer(size); + } + + @Override + public ExecutionControls getExecutionControls() { +return context.getExecutionControls(); + } + + @Override + public BufferAllocator getAllocator() { +if (allocator == null) { + throw new UnsupportedOperationException("Operator context does not have an allocator"); +} +return allocator; + } + + // Allow an operator to use the thread pool + @Override + public ExecutorService getExecutor() { +return context.getDrillbitContext().getExecutor(); + } + + @Override + public ExecutorService getScanExecutor() { +return context.getDrillbitContext().getScanExecutor(); + } + + @Override + public ExecutorService getScanDecodeExecutor() { +return context.getDrillbitContext().getScanDecodeExecutor(); + } + + @Overrid
[jira] [Commented] (DRILL-5842) Refactor and simplify the fragment, operator contexts for testing
[ https://issues.apache.org/jira/browse/DRILL-5842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16233528#comment-16233528 ] ASF GitHub Bot commented on DRILL-5842: --- Github user sohami commented on the issue: https://github.com/apache/drill/pull/978 Looking into the PR and based on description it says below: _A result of these changes is that OperatorContext is now a one-stop-shop for services needed by operators. It now provides:_ - Access to the fragment context: getFragmentContext() - Access to the physical operator definition (AKA “physical operator”): getOperatorDefn(). _Because of these changes, we need no longer pass around the triple of fragment context, operator definition and operator context — something needed by the “managed” sort and upcoming PRs._ Based on my understanding it looks like existing interface and implementation `OperExecContextImpl` do provide access to `FragmentContext` and `PopConfig` (operator Defn). The instance of this context is created inside `ExternalSortBatch` which has access to FragmentContext and PopConfig already. Also `SortImpl` only takes in `OperExecContext`. I am little confused how the change avoided the need to pass around the triplet ? > Refactor and simplify the fragment, operator contexts for testing > - > > Key: DRILL-5842 > URL: https://issues.apache.org/jira/browse/DRILL-5842 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.12.0 >Reporter: Paul Rogers >Assignee: Paul Rogers >Priority: Major > Fix For: 1.12.0 > > > Drill's execution engine has a "fragment context" that provides state for a > fragment as a whole, and an "operator context" which provides state for a > single operator. Historically, these have both been concrete classes that > make generous references to the Drillbit context, and hence need a full Drill > server in order to operate. > Drill has historically made extensive use of system-level testing: build the > entire server and fire queries at it to test each component. Over time, we > are augmenting that approach with unit tests: the ability to test each > operator (or parts of an operator) in isolation. > Since each operator requires access to both the operator and fragment > context, the fact that the contexts depend on the overall server creates a > large barrier to unit testing. An earlier checkin started down the path of > defining the contexts as interfaces that can have different run-time and > test-time implementations to enable testing. > This ticket asks to refactor those interfaces: simplifying the operator > context and introducing an interface for the fragment context. New code will > use these new interfaces, while older code continues to use the concrete > implementations. Over time, as operators are enhanced, they can be modified > to allow unit-level testing. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-4091) Support more functions in gis contrib module
[ https://issues.apache.org/jira/browse/DRILL-4091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16233590#comment-16233590 ] ASF GitHub Bot commented on DRILL-4091: --- Github user cgivre commented on a diff in the pull request: https://github.com/apache/drill/pull/258#discussion_r148117858 --- Diff: contrib/gis/src/main/java/org/apache/drill/exec/expr/fn/impl/gis/STContains.java --- @@ -0,0 +1,62 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.drill.exec.expr.fn.impl.gis; + +import javax.inject.Inject; + +import org.apache.drill.exec.expr.DrillSimpleFunc; +import org.apache.drill.exec.expr.annotations.FunctionTemplate; +import org.apache.drill.exec.expr.annotations.Output; +import org.apache.drill.exec.expr.annotations.Param; +import org.apache.drill.exec.expr.holders.BitHolder; +import org.apache.drill.exec.expr.holders.VarBinaryHolder; + +import io.netty.buffer.DrillBuf; + --- End diff -- Please add a comment about what the function does. This can be cut/pasted from the PostGIS docs if need be. IE `/*Returns true if and only if no points of B lie in the exterior of A, and at least one point of the interior of B lies in the interior of A.*/` > Support more functions in gis contrib module > > > Key: DRILL-4091 > URL: https://issues.apache.org/jira/browse/DRILL-4091 > Project: Apache Drill > Issue Type: Improvement > Components: Functions - Drill >Reporter: Karol Potocki >Priority: Major > > Support for commonly used gis functions in gis contrib module: relate, > contains, crosses, intersects, touches, difference, disjoint, buffer, union > etc. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-4091) Support more functions in gis contrib module
[ https://issues.apache.org/jira/browse/DRILL-4091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16233592#comment-16233592 ] ASF GitHub Bot commented on DRILL-4091: --- Github user cgivre commented on a diff in the pull request: https://github.com/apache/drill/pull/258#discussion_r148114646 --- Diff: contrib/gis/src/main/java/com/esri/core/geometry/VertexGeomAccessor.java --- @@ -0,0 +1,25 @@ +/** --- End diff -- Please put the Apache license in a regular java comment instead of a Javadoc. Please update all files accordingly. > Support more functions in gis contrib module > > > Key: DRILL-4091 > URL: https://issues.apache.org/jira/browse/DRILL-4091 > Project: Apache Drill > Issue Type: Improvement > Components: Functions - Drill >Reporter: Karol Potocki >Priority: Major > > Support for commonly used gis functions in gis contrib module: relate, > contains, crosses, intersects, touches, difference, disjoint, buffer, union > etc. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-4091) Support more functions in gis contrib module
[ https://issues.apache.org/jira/browse/DRILL-4091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16233594#comment-16233594 ] ASF GitHub Bot commented on DRILL-4091: --- Github user cgivre commented on a diff in the pull request: https://github.com/apache/drill/pull/258#discussion_r148115797 --- Diff: contrib/gis/src/main/java/org/apache/drill/exec/expr/fn/impl/gis/STBuffer.java --- @@ -0,0 +1,65 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.drill.exec.expr.fn.impl.gis; + +import javax.inject.Inject; + +import org.apache.drill.exec.expr.DrillSimpleFunc; +import org.apache.drill.exec.expr.annotations.FunctionTemplate; +import org.apache.drill.exec.expr.annotations.Output; +import org.apache.drill.exec.expr.annotations.Param; +import org.apache.drill.exec.expr.holders.Float8Holder; +import org.apache.drill.exec.expr.holders.VarBinaryHolder; + +import io.netty.buffer.DrillBuf; + +@FunctionTemplate(name = "st_buffer", scope = FunctionTemplate.FunctionScope.SIMPLE, + nulls = FunctionTemplate.NullHandling.NULL_IF_NULL) +public class STBuffer implements DrillSimpleFunc { + @Param + VarBinaryHolder geom1Param; + + @Param(constant = true) --- End diff -- Are you setting a constant that is used in multiple function iterations? If so, I've not seen it done this way before. The way I've done this is by using a Java primitive in the `@Workspace` and setting the value in the `setup()` function. Is there some advantage to doing it this way? > Support more functions in gis contrib module > > > Key: DRILL-4091 > URL: https://issues.apache.org/jira/browse/DRILL-4091 > Project: Apache Drill > Issue Type: Improvement > Components: Functions - Drill >Reporter: Karol Potocki >Priority: Major > > Support for commonly used gis functions in gis contrib module: relate, > contains, crosses, intersects, touches, difference, disjoint, buffer, union > etc. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-4091) Support more functions in gis contrib module
[ https://issues.apache.org/jira/browse/DRILL-4091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16233593#comment-16233593 ] ASF GitHub Bot commented on DRILL-4091: --- Github user cgivre commented on a diff in the pull request: https://github.com/apache/drill/pull/258#discussion_r148172786 --- Diff: contrib/gis/src/main/java/org/apache/drill/exec/expr/fn/impl/gis/STUnionAggregate.java --- @@ -0,0 +1,114 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.drill.exec.expr.fn.impl.gis; + +import javax.inject.Inject; + +import org.apache.drill.exec.expr.DrillAggFunc; +import org.apache.drill.exec.expr.annotations.FunctionTemplate; +import org.apache.drill.exec.expr.annotations.Output; +import org.apache.drill.exec.expr.annotations.Param; +import org.apache.drill.exec.expr.annotations.Workspace; +import org.apache.drill.exec.expr.holders.BigIntHolder; +import org.apache.drill.exec.expr.holders.IntHolder; +import org.apache.drill.exec.expr.holders.NullableVarBinaryHolder; +import org.apache.drill.exec.expr.holders.ObjectHolder; +import org.apache.drill.exec.expr.holders.UInt1Holder; + +import com.esri.core.geometry.SpatialReference; + +import io.netty.buffer.DrillBuf; + +@FunctionTemplate(name = "st_unionaggregate", scope = FunctionTemplate.FunctionScope.POINT_AGGREGATE) +public class STUnionAggregate implements DrillAggFunc { + @Param NullableVarBinaryHolder in; + @Workspace ObjectHolder value; --- End diff -- I believe the ObjectHolder is deprecated. However, it would be worth having a discussion about "undeprecating" it as without it, it is really difficult if not impossible to write aggregate functions. > Support more functions in gis contrib module > > > Key: DRILL-4091 > URL: https://issues.apache.org/jira/browse/DRILL-4091 > Project: Apache Drill > Issue Type: Improvement > Components: Functions - Drill >Reporter: Karol Potocki >Priority: Major > > Support for commonly used gis functions in gis contrib module: relate, > contains, crosses, intersects, touches, difference, disjoint, buffer, union > etc. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-4091) Support more functions in gis contrib module
[ https://issues.apache.org/jira/browse/DRILL-4091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16233591#comment-16233591 ] ASF GitHub Bot commented on DRILL-4091: --- Github user cgivre commented on a diff in the pull request: https://github.com/apache/drill/pull/258#discussion_r148172580 --- Diff: contrib/gis/src/main/java/org/apache/drill/exec/expr/fn/impl/gis/STUnionAggregate.java --- @@ -0,0 +1,114 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.drill.exec.expr.fn.impl.gis; + +import javax.inject.Inject; + +import org.apache.drill.exec.expr.DrillAggFunc; +import org.apache.drill.exec.expr.annotations.FunctionTemplate; +import org.apache.drill.exec.expr.annotations.Output; +import org.apache.drill.exec.expr.annotations.Param; +import org.apache.drill.exec.expr.annotations.Workspace; +import org.apache.drill.exec.expr.holders.BigIntHolder; +import org.apache.drill.exec.expr.holders.IntHolder; +import org.apache.drill.exec.expr.holders.NullableVarBinaryHolder; +import org.apache.drill.exec.expr.holders.ObjectHolder; +import org.apache.drill.exec.expr.holders.UInt1Holder; + +import com.esri.core.geometry.SpatialReference; + +import io.netty.buffer.DrillBuf; + +@FunctionTemplate(name = "st_unionaggregate", scope = FunctionTemplate.FunctionScope.POINT_AGGREGATE) --- End diff -- Do you want to include NULL handling here? > Support more functions in gis contrib module > > > Key: DRILL-4091 > URL: https://issues.apache.org/jira/browse/DRILL-4091 > Project: Apache Drill > Issue Type: Improvement > Components: Functions - Drill >Reporter: Karol Potocki >Priority: Major > > Support for commonly used gis functions in gis contrib module: relate, > contains, crosses, intersects, touches, difference, disjoint, buffer, union > etc. -- This message was sent by Atlassian JIRA (v6.4.14#64029)