[jira] [Commented] (DRILL-5717) date time test cases is not Local independent
[ https://issues.apache.org/jira/browse/DRILL-5717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123568#comment-16123568 ] ASF GitHub Bot commented on DRILL-5717: --- Github user vvysotskyi commented on the issue: https://github.com/apache/drill/pull/904 Some tests from `TestDateFunctions`, `TestDateConversions` and `TestConstantFolding` classes also fails with some other locales (I checked it for `zh_TW` locale). I think they should also be fixed in the border of this PR. Are there any other tests that depend on the locale? > date time test cases is not Local independent > - > > Key: DRILL-5717 > URL: https://issues.apache.org/jira/browse/DRILL-5717 > Project: Apache Drill > Issue Type: Bug > Components: Tools, Build & Test >Affects Versions: 1.9.0, 1.11.0 >Reporter: weijie.tong > > Some date time test cases like JodaDateValidatorTest is not Local > independent .This will cause other Local's users's test phase to fail. We > should let these test cases to be Local env independent. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (DRILL-5712) Update the pom files with dependency exclusions for commons-codec
[ https://issues.apache.org/jira/browse/DRILL-5712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sindhuri Ramanarayan Rayavaram updated DRILL-5712: -- Component/s: Tools, Build & Test > Update the pom files with dependency exclusions for commons-codec > - > > Key: DRILL-5712 > URL: https://issues.apache.org/jira/browse/DRILL-5712 > Project: Apache Drill > Issue Type: Bug > Components: Tools, Build & Test >Reporter: Sindhuri Ramanarayan Rayavaram >Assignee: Sindhuri Ramanarayan Rayavaram > > In java-exec, we are adding a dependency for commons-codec of version 1.10. > Other dependencies like hadoop-common, parquet-column etc are trying to > download different versions for common codec. Exclusions should be added for > common-codec in these dependencies. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-4264) Dots in identifier are not escaped correctly
[ https://issues.apache.org/jira/browse/DRILL-4264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123836#comment-16123836 ] John Omernik commented on DRILL-4264: - Lots to process here... Let me add a simple thing though... "As a user I have a new data set, I have no idea what's in it, from Drill (that's the key) I should be able to select * from directory, and if it's a known format (JSON, Parquet, CSV etc) I should get results back. I know I can do select `field.one`, `field.two` from directory and get it, but say it's a parquet file created in Spark... there is no way for me explore that data in Drill ... I need select * > Dots in identifier are not escaped correctly > > > Key: DRILL-4264 > URL: https://issues.apache.org/jira/browse/DRILL-4264 > Project: Apache Drill > Issue Type: Improvement > Components: Execution - Codegen >Reporter: Alex >Assignee: Volodymyr Vysotskyi > Labels: doc-impacting > Fix For: 1.12.0 > > > If you have some json data like this... > {code:javascript} > { > "0.0.1":{ > "version":"0.0.1", > "date_created":"2014-03-15" > }, > "0.1.2":{ > "version":"0.1.2", > "date_created":"2014-05-21" > } > } > {code} > ... there is no way to select any of the rows since their identifiers contain > dots and when trying to select them, Drill throws the following error: > Error: SYSTEM ERROR: UnsupportedOperationException: Unhandled field reference > "0.0.1"; a field reference identifier must not have the form of a qualified > name > This must be fixed since there are many json data files containing dots in > some of the keys (e.g. when specifying version numbers etc) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5431) Support SSL
[ https://issues.apache.org/jira/browse/DRILL-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123633#comment-16123633 ] Parth Chandra commented on DRILL-5431: -- [~cshi] Please take a look at the design doc. > Support SSL > --- > > Key: DRILL-5431 > URL: https://issues.apache.org/jira/browse/DRILL-5431 > Project: Apache Drill > Issue Type: New Feature > Components: Client - Java, Client - ODBC >Reporter: Sudheesh Katkam >Assignee: Sudheesh Katkam > > Support SSL between Drillbit and JDBC/ODBC drivers. Drill already supports > HTTPS for web traffic. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5717) date time test cases is not Local independent
[ https://issues.apache.org/jira/browse/DRILL-5717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123020#comment-16123020 ] ASF GitHub Bot commented on DRILL-5717: --- GitHub user weijietong opened a pull request: https://github.com/apache/drill/pull/904 DRILL-5717: let some date time test cases be Local independent Some date time test cases is US Local .This will cause developers from other areas to fail. Fix is to specify the explicit Local to the test cases. You can merge this pull request into a Git repository by running: $ git pull https://github.com/weijietong/drill DRILL-5717 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/904.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #904 commit b44f780a948c4a0898e7cee042c0590f0713f780 Author: weijietongDate: 2017-06-08T08:03:46Z Merge pull request #1 from apache/master sync commit d045c757c80a759b435479cc89f33c749fc16ac2 Author: weijie.tong Date: 2017-08-11T08:01:36Z Merge branch 'master' of github.com:weijietong/drill commit 906eef175f6751663e58f36519b325f5a4e6dfea Author: weijie.tong Date: 2017-08-11T08:14:54Z let some test cases be Local independent > date time test cases is not Local independent > - > > Key: DRILL-5717 > URL: https://issues.apache.org/jira/browse/DRILL-5717 > Project: Apache Drill > Issue Type: Bug > Components: Tools, Build & Test >Affects Versions: 1.9.0, 1.11.0 >Reporter: weijie.tong > > Some date time test cases like JodaDateValidatorTest is not Local > independent .This will cause other Local's users's test phase to fail. We > should let these test cases to be Local env independent. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (DRILL-3867) Store relative paths in metadata file
[ https://issues.apache.org/jira/browse/DRILL-3867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vitalii Diravka updated DRILL-3867: --- Description: git.commit.id.abbrev=cf4f745 git.commit.time=29.09.2015 @ 23\:19\:52 UTC The below sequence of steps reproduces the issue 1. Create the cache file {code} 0: jdbc:drill:zk=10.10.103.60:5181> refresh table metadata dfs.`/drill/testdata/metadata_caching/lineitem`; +---+-+ | ok | summary | +---+-+ | true | Successfully updated metadata for table /drill/testdata/metadata_caching/lineitem. | +---+-+ 1 row selected (1.558 seconds) {code} 2. Move the directory {code} hadoop fs -mv /drill/testdata/metadata_caching/lineitem /drill/ {code} 3. Now run a query on top of it {code} 0: jdbc:drill:zk=10.10.103.60:5181> select * from dfs.`/drill/lineitem` limit 1; Error: SYSTEM ERROR: FileNotFoundException: Requested file maprfs:///drill/testdata/metadata_caching/lineitem/2006/1 does not exist. [Error Id: b456d912-57a0-4690-a44b-140d4964903e on pssc-66.qa.lab:31010] (state=,code=0) {code} This is obvious given the fact that we are storing absolute file paths in the cache file. *Summary description of the fix:* In Drill 1.11 and later, Drill stores the paths to the Parquet files as relative paths instead of absolute paths. You can move partitioned Parquet directories from one location in the distributed files system to another without issuing the REFRESH TABLE METADATA command to rebuild the Parquet metadata files; the metadata remains valid in the new location. Note Reverting back to a previous version of Drill from 1.11 is not recommended because Drill will incorrectly interpret the Parquet metadata files created by Drill 1.11. Should this occur, remove the Parquet metadata files and run the refresh table metadata command to rebuild the files in the older format. was: git.commit.id.abbrev=cf4f745 git.commit.time=29.09.2015 @ 23\:19\:52 UTC The below sequence of steps reproduces the issue 1. Create the cache file {code} 0: jdbc:drill:zk=10.10.103.60:5181> refresh table metadata dfs.`/drill/testdata/metadata_caching/lineitem`; +---+-+ | ok | summary | +---+-+ | true | Successfully updated metadata for table /drill/testdata/metadata_caching/lineitem. | +---+-+ 1 row selected (1.558 seconds) {code} 2. Move the directory {code} hadoop fs -mv /drill/testdata/metadata_caching/lineitem /drill/ {code} 3. Now run a query on top of it {code} 0: jdbc:drill:zk=10.10.103.60:5181> select * from dfs.`/drill/lineitem` limit 1; Error: SYSTEM ERROR: FileNotFoundException: Requested file maprfs:///drill/testdata/metadata_caching/lineitem/2006/1 does not exist. [Error Id: b456d912-57a0-4690-a44b-140d4964903e on pssc-66.qa.lab:31010] (state=,code=0) {code} This is obvious given the fact that we are storing absolute file paths in the cache file > Store relative paths in metadata file > - > > Key: DRILL-3867 > URL: https://issues.apache.org/jira/browse/DRILL-3867 > Project: Apache Drill > Issue Type: Bug > Components: Metadata >Affects Versions: 1.2.0 >Reporter: Rahul Challapalli >Assignee: Vitalii Diravka > Labels: doc-impacting, ready-to-commit > Fix For: 1.11.0 > > > git.commit.id.abbrev=cf4f745 > git.commit.time=29.09.2015 @ 23\:19\:52 UTC > The below sequence of steps reproduces the issue > 1. Create the cache file > {code} > 0: jdbc:drill:zk=10.10.103.60:5181> refresh table metadata > dfs.`/drill/testdata/metadata_caching/lineitem`; > +---+-+ > | ok | summary > | > +---+-+ > | true | Successfully updated metadata for table > /drill/testdata/metadata_caching/lineitem. | > +---+-+ > 1 row selected (1.558 seconds) > {code} > 2. Move the directory > {code} > hadoop fs -mv /drill/testdata/metadata_caching/lineitem /drill/ >
[jira] [Created] (DRILL-5717) date time test cases is not Local independent
weijie.tong created DRILL-5717: -- Summary: date time test cases is not Local independent Key: DRILL-5717 URL: https://issues.apache.org/jira/browse/DRILL-5717 Project: Apache Drill Issue Type: Bug Components: Tools, Build & Test Affects Versions: 1.11.0, 1.9.0 Reporter: weijie.tong Some date time test cases like JodaDateValidatorTest is not Local independent .This will cause other Local's users's test phase to fail. We should let these test cases to be Local env independent. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (DRILL-5717) date time test cases is not Local independent
[ https://issues.apache.org/jira/browse/DRILL-5717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] weijie.tong updated DRILL-5717: --- Reviewer: Arina Ielchiieva > date time test cases is not Local independent > - > > Key: DRILL-5717 > URL: https://issues.apache.org/jira/browse/DRILL-5717 > Project: Apache Drill > Issue Type: Bug > Components: Tools, Build & Test >Affects Versions: 1.9.0, 1.11.0 >Reporter: weijie.tong > > Some date time test cases like JodaDateValidatorTest is not Local > independent .This will cause other Local's users's test phase to fail. We > should let these test cases to be Local env independent. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-1162) 25 way join ended up with OOM
[ https://issues.apache.org/jira/browse/DRILL-1162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123300#comment-16123300 ] ASF GitHub Bot commented on DRILL-1162: --- GitHub user vvysotskyi opened a pull request: https://github.com/apache/drill/pull/905 DRILL-1162: Fix OOM for hash join operator when the right input has a much larger actual number of rows than the left one Please see the problem description in [DRILL-1162](https://issues.apache.org/jira/browse/DRILL-1162). You can merge this pull request into a Git repository by running: $ git pull https://github.com/vvysotskyi/drill DRILL-1162 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/905.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #905 commit de9a32997774acf34d2cb42f534decac5fd75cb5 Author: Volodymyr VysotskyiDate: 2017-08-09T21:21:48Z DRILL-1162: Fix OOM for hash join operator when the right input has a much larger actual number of rows than the left one > 25 way join ended up with OOM > - > > Key: DRILL-1162 > URL: https://issues.apache.org/jira/browse/DRILL-1162 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Flow, Query Planning & Optimization >Reporter: Rahul Challapalli >Assignee: Volodymyr Vysotskyi >Priority: Critical > Fix For: Future > > Attachments: error.log, oom_error.log > > > git.commit.id.abbrev=e5c2da0 > The below query results in 0 results being returned > {code:sql} > select count(*) from `lineitem1.parquet` a > inner join `part.parquet` j on a.l_partkey = j.p_partkey > inner join `orders.parquet` k on a.l_orderkey = k.o_orderkey > inner join `supplier.parquet` l on a.l_suppkey = l.s_suppkey > inner join `partsupp.parquet` m on j.p_partkey = m.ps_partkey and l.s_suppkey > = m.ps_suppkey > inner join `customer.parquet` n on k.o_custkey = n.c_custkey > inner join `lineitem2.parquet` b on a.l_orderkey = b.l_orderkey > inner join `lineitem2.parquet` c on a.l_partkey = c.l_partkey > inner join `lineitem2.parquet` d on a.l_suppkey = d.l_suppkey > inner join `lineitem2.parquet` e on a.l_extendedprice = e.l_extendedprice > inner join `lineitem2.parquet` f on a.l_comment = f.l_comment > inner join `lineitem2.parquet` g on a.l_shipdate = g.l_shipdate > inner join `lineitem2.parquet` h on a.l_commitdate = h.l_commitdate > inner join `lineitem2.parquet` i on a.l_receiptdate = i.l_receiptdate > inner join `lineitem2.parquet` o on a.l_receiptdate = o.l_receiptdate > inner join `lineitem2.parquet` p on a.l_receiptdate = p.l_receiptdate > inner join `lineitem2.parquet` q on a.l_receiptdate = q.l_receiptdate > inner join `lineitem2.parquet` r on a.l_receiptdate = r.l_receiptdate > inner join `lineitem2.parquet` s on a.l_receiptdate = s.l_receiptdate > inner join `lineitem2.parquet` t on a.l_receiptdate = t.l_receiptdate > inner join `lineitem2.parquet` u on a.l_receiptdate = u.l_receiptdate > inner join `lineitem2.parquet` v on a.l_receiptdate = v.l_receiptdate > inner join `lineitem2.parquet` w on a.l_receiptdate = w.l_receiptdate > inner join `lineitem2.parquet` x on a.l_receiptdate = x.l_receiptdate; > {code} > However when we remove the last 'inner join' and run the query it returns > '716372534'. Since the last inner join is similar to the one's before it, it > should match some records and return the data appropriately. > The logs indicated that it actually returned 0 results. Attached the log file. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5546) Schema change problems caused by empty batch
[ https://issues.apache.org/jira/browse/DRILL-5546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124096#comment-16124096 ] ASF GitHub Bot commented on DRILL-5546: --- GitHub user jinfengni opened a pull request: https://github.com/apache/drill/pull/906 DRILL-5546: Handle schema change exception failure caused by empty in… …put or empty batche. 1. Modify ScanBatch's logic when it iterates list of RecordReader. 1) Skip RecordReader if it returns 0 row && present same schema. A new schema (by calling Mutator.isNewSchema() ) means either a new top level field is added, or a field in a nested field is added, or an existing field type is changed. 2) Implicit columns are added and populated only when the input is not empty, i.e. the batch contains > 0 row or rowCount == 0 && new schema. 3) ScanBatch will return NONE directly (called as "fast NONE"), if all its RecordReaders haver empty input and thus are skipped, in stead of returing OK_NEW_SCHEMA first. 2. Modify IteratorValidatorBatchIterator to allow 1) fast NONE ( before seeing a OK_NEW_SCHEMA) 2) batch with empty list of columns. 2. Modify JsonRecordReader when it get 0 row. Do not insert a nullable-int column for 0 row input. Together with ScanBatch, Drill will skip empty json files. 3. Modify binary operators such as join, union to handle fast none for either one side or both sides. Abstract the logic in AbstractBinaryRecordBatch, except for MergeJoin as its implementation is quite different from others. 4. Fix and refactor union all operator. 1) Correct union operator hanndling 0 input rows. Previously, it will ignore inputs with 0 row and put nullable-int into output schema, which causes various of schema change issue in down-stream operator. The new behavior is to take schema with 0 into account in determining the output schema, in the same way with > 0 input rows. By doing that, we ensure Union operator will not behave like a schema-lossy operator. 2) Add a UnionInputIterator to simplify the logic to iterate the left/right inputs, removing significant chunk of duplicate codes in previous implementation. The new union all operator reduces the code size into half, comparing the old one. 5. Introduce UntypedNullVector to handle convertFromJson() function, when the input batch contains 0 row. Problem: The function convertFromJSon() is different from other regular functions in that it only knows the output schema after evaluation is performed. When input has 0 row, Drill essentially does not have a way to know the output type, and previously will assume Map type. That works under the assumption other operators like Union would ignore batch with 0 row, which is no longer the case in the current implementation. Solution: Use MinorType.NULL at the output type for convertFromJSON() when input contains 0 row. The new UntypedNullVector is used to represent a column with MinorType.NULL. 6. HBaseGroupScan convert star column into list of row_key and column family. HBaseRecordReader should reject column star since it expectes star has been converted somewhere else. In HBase a column family always has map type, and a non-rowkey column always has nullable varbinary type, this ensures that HBaseRecordReader across different HBase regions will have the same top level schema, even if the region is empty or prune all the rows due to filter pushdown optimization. In other words, we will not see different top level schema from different HBaseRecordReader for the same table. However, such change will not be able to handle hard schema change : c1 exists in cf1 in one region, but not in another region. Further work is required to handle hard schema change. 7. Modify scan cost estimation when the query involves * column. This is to remove the planning randomness since previously two different operators could have same cost. 8. Add a new flag 'outputProj' to Project operator, to indicate if Project is for the query's final output. Such Project is added by TopProjectVisitor, to handle fast NONE when all the inputs to the query are empty and are skipped. 1) column star is replaced with empty list 2) regular column reference is replaced with nullable-int column 3) An expression will go through ExpressionTreeMaterializer, and use the type of materialized expression as the output type 4) Return an OK_NEW_SCHEMA with the schema using the above logic, then return a NONE to down-stream operator. 9. Add unit test to test operators handling empty input. 10. Add unit test to test query when inputs are all empty. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jinfengni/incubator-drill
[jira] [Commented] (DRILL-5717) date time test cases is not Local independent
[ https://issues.apache.org/jira/browse/DRILL-5717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123865#comment-16123865 ] Paul Rogers commented on DRILL-5717: Drill dates, by design, a defined to be in the server's time zone, even when sent to the client, and even when being converted from data without a time zone. Given an input of 2017-08-11T23:00:00 Drill assumes that this is a date in the local time zone, say PST. But, it represents the time as if PST were UTC. That is, parse the date using Java Date and get a time relative to the epoch UTC but interpret it as relative to the epoch PST. Then, the date is sent to the client where the PST-as-UTC is interpreted as GMT-as (say) EST. That is, again, the time is a number relative to the epoch UTC (set as if that were PST), but reinterpreted as seconds since the epoch EST. The point is, the same number is sometimes PST, sometimes EST but (to Java) is UTC. (This whole area is terribly confusing, so the above may be slightly off; DRILL-5360, DRILL-5334 explored these issues in detail.) This can push the date into a different day. This is by design (though, it can be argued, not perhaps the best possible design). Tests must somehow work around this creative interpretation of dates. > date time test cases is not Local independent > - > > Key: DRILL-5717 > URL: https://issues.apache.org/jira/browse/DRILL-5717 > Project: Apache Drill > Issue Type: Bug > Components: Tools, Build & Test >Affects Versions: 1.9.0, 1.11.0 >Reporter: weijie.tong > > Some date time test cases like JodaDateValidatorTest is not Local > independent .This will cause other Local's users's test phase to fail. We > should let these test cases to be Local env independent. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5431) Support SSL
[ https://issues.apache.org/jira/browse/DRILL-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124141#comment-16124141 ] Laurent Goujon commented on DRILL-5431: --- Can we allow comments for the document? or shall we comment on the JIRA itself? > Support SSL > --- > > Key: DRILL-5431 > URL: https://issues.apache.org/jira/browse/DRILL-5431 > Project: Apache Drill > Issue Type: New Feature > Components: Client - Java, Client - ODBC >Reporter: Sudheesh Katkam >Assignee: Sudheesh Katkam > > Support SSL between Drillbit and JDBC/ODBC drivers. Drill already supports > HTTPS for web traffic. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5704) Improve error message on client side when queries fail with "Failed to create schema tree." when Impersonation is enabled and logins are anonymous
[ https://issues.apache.org/jira/browse/DRILL-5704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124006#comment-16124006 ] ASF GitHub Bot commented on DRILL-5704: --- Github user parthchandra commented on the issue: https://github.com/apache/drill/pull/895 +1 > Improve error message on client side when queries fail with "Failed to create > schema tree." when Impersonation is enabled and logins are anonymous > -- > > Key: DRILL-5704 > URL: https://issues.apache.org/jira/browse/DRILL-5704 > Project: Apache Drill > Issue Type: Improvement >Reporter: Sorabh Hamirwasia >Assignee: Sorabh Hamirwasia > Fix For: 1.12.0 > > > Reported by [~agirish] > When username is not specified then Drill set's the session user as anonymous > if impersonation is enabled. During query execution Drill tries to build > schema tree and as part of that it validates if the user has access to the > workspace or not by using FileClient Api liststatus which verifies the user > from the OS user. Since impersonation is only enabled here without > authentication and we don't specify any user in connection string, Drill will > use default user which is "anonymous" and pass that to check workspace > permission which will fail as node doesn't have any valid user with that name. > {code:java} > Caused by: java.io.IOException: Error getting user info for current user, > anonymous >.. >.. > at > org.apache.drill.exec.store.dfs.DrillFileSystem.listStatus(DrillFileSystem.java:523) > ~[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory.accessible(WorkspaceSchemaFactory.java:157) > ~[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.dfs.FileSystemSchemaFactory$FileSystemSchema.(FileSystemSchemaFactory.java:78) > ~[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.dfs.FileSystemSchemaFactory.registerSchemas(FileSystemSchemaFactory.java:65) > ~[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.dfs.FileSystemPlugin.registerSchemas(FileSystemPlugin.java:150) > ~[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.StoragePluginRegistryImpl$DrillSchemaFactory.registerSchemas(StoragePluginRegistryImpl.java:365) > ~[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema(SchemaTreeProvider.java:72) > [drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > ... 10 common frames omitted > {code} > # $DRILL_HOME/bin/sqlline -u "jdbc:drill:zk=localhost:5181" > sqlline> select * from sys.drillbits; > User Error Occurred > org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: Failed to > create schema tree. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (DRILL-5704) Improve error message on client side when queries fail with "Failed to create schema tree." when Impersonation is enabled and logins are anonymous
[ https://issues.apache.org/jira/browse/DRILL-5704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sorabh Hamirwasia updated DRILL-5704: - Labels: ready-to-commit (was: ) > Improve error message on client side when queries fail with "Failed to create > schema tree." when Impersonation is enabled and logins are anonymous > -- > > Key: DRILL-5704 > URL: https://issues.apache.org/jira/browse/DRILL-5704 > Project: Apache Drill > Issue Type: Improvement >Reporter: Sorabh Hamirwasia >Assignee: Sorabh Hamirwasia > Labels: ready-to-commit > Fix For: 1.12.0 > > > Reported by [~agirish] > When username is not specified then Drill set's the session user as anonymous > if impersonation is enabled. During query execution Drill tries to build > schema tree and as part of that it validates if the user has access to the > workspace or not by using FileClient Api liststatus which verifies the user > from the OS user. Since impersonation is only enabled here without > authentication and we don't specify any user in connection string, Drill will > use default user which is "anonymous" and pass that to check workspace > permission which will fail as node doesn't have any valid user with that name. > {code:java} > Caused by: java.io.IOException: Error getting user info for current user, > anonymous >.. >.. > at > org.apache.drill.exec.store.dfs.DrillFileSystem.listStatus(DrillFileSystem.java:523) > ~[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory.accessible(WorkspaceSchemaFactory.java:157) > ~[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.dfs.FileSystemSchemaFactory$FileSystemSchema.(FileSystemSchemaFactory.java:78) > ~[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.dfs.FileSystemSchemaFactory.registerSchemas(FileSystemSchemaFactory.java:65) > ~[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.dfs.FileSystemPlugin.registerSchemas(FileSystemPlugin.java:150) > ~[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.StoragePluginRegistryImpl$DrillSchemaFactory.registerSchemas(StoragePluginRegistryImpl.java:365) > ~[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema(SchemaTreeProvider.java:72) > [drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > ... 10 common frames omitted > {code} > # $DRILL_HOME/bin/sqlline -u "jdbc:drill:zk=localhost:5181" > sqlline> select * from sys.drillbits; > User Error Occurred > org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: Failed to > create schema tree. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5601) Rollup of External Sort memory management fixes
[ https://issues.apache.org/jira/browse/DRILL-5601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124217#comment-16124217 ] ASF GitHub Bot commented on DRILL-5601: --- Github user paul-rogers commented on the issue: https://github.com/apache/drill/pull/860 Rebased onto master. > Rollup of External Sort memory management fixes > --- > > Key: DRILL-5601 > URL: https://issues.apache.org/jira/browse/DRILL-5601 > Project: Apache Drill > Issue Type: Task >Affects Versions: 1.11.0 >Reporter: Paul Rogers >Assignee: Paul Rogers > Labels: ready-to-commit > Fix For: 1.12.0 > > > Rollup of a set of specific JIRA entries that all relate to the very > difficult problem of managing memory within Drill in order for the external > sort to stay within a memory budget. In general, the fixes relate to better > estimating memory used by the three ways that Drill allocates vector memory > (see DRILL-5522) and to predicting the size of vectors that the sort will > create, to avoid repeated realloc-copy cycles (see DRILL-5594). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (DRILL-5663) Drillbit fails to start when only keystore path is provided without keystore password.
[ https://issues.apache.org/jira/browse/DRILL-5663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sindhuri Ramanarayan Rayavaram updated DRILL-5663: -- Labels: ready-to-commit (was: ) > Drillbit fails to start when only keystore path is provided without keystore > password. > -- > > Key: DRILL-5663 > URL: https://issues.apache.org/jira/browse/DRILL-5663 > Project: Apache Drill > Issue Type: Bug >Reporter: Sorabh Hamirwasia >Assignee: Sindhuri Ramanarayan Rayavaram > Labels: ready-to-commit > > When we configure keystore path without keystore password inside > drill-override.conf for WebServer, then Drillbit fails to start. We should > explicitly check for either both being present or both being absent. If any > one of them is only present then throw startup exception for Drill. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-4735) Count(dir0) on parquet returns 0 result
[ https://issues.apache.org/jira/browse/DRILL-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124339#comment-16124339 ] ASF GitHub Bot commented on DRILL-4735: --- Github user jinfengni commented on the issue: https://github.com/apache/drill/pull/900 +1 LGTM. Thanks for the PR! > Count(dir0) on parquet returns 0 result > --- > > Key: DRILL-4735 > URL: https://issues.apache.org/jira/browse/DRILL-4735 > Project: Apache Drill > Issue Type: Bug > Components: Query Planning & Optimization, Storage - Parquet >Affects Versions: 1.0.0, 1.4.0, 1.6.0, 1.7.0 >Reporter: Krystal >Assignee: Arina Ielchiieva >Priority: Critical > > Selecting a count of dir0, dir1, etc against a parquet directory returns 0 > rows. > select count(dir0) from `min_max_dir`; > +-+ > | EXPR$0 | > +-+ > | 0 | > +-+ > select count(dir1) from `min_max_dir`; > +-+ > | EXPR$0 | > +-+ > | 0 | > +-+ > If I put both dir0 and dir1 in the same select, it returns expected result: > select count(dir0), count(dir1) from `min_max_dir`; > +-+-+ > | EXPR$0 | EXPR$1 | > +-+-+ > | 600 | 600 | > +-+-+ > Here is the physical plan for count(dir0) query: > {code} > 00-00Screen : rowType = RecordType(BIGINT EXPR$0): rowcount = 20.0, > cumulative cost = {22.0 rows, 22.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id > = 1346 > 00-01 Project(EXPR$0=[$0]) : rowType = RecordType(BIGINT EXPR$0): > rowcount = 20.0, cumulative cost = {20.0 rows, 20.0 cpu, 0.0 io, 0.0 network, > 0.0 memory}, id = 1345 > 00-02Project(EXPR$0=[$0]) : rowType = RecordType(BIGINT EXPR$0): > rowcount = 20.0, cumulative cost = {20.0 rows, 20.0 cpu, 0.0 io, 0.0 network, > 0.0 memory}, id = 1344 > 00-03 > Scan(groupscan=[org.apache.drill.exec.store.pojo.PojoRecordReader@3da85d3b[columns > = null, isStarQuery = false, isSkipQuery = false]]) : rowType = > RecordType(BIGINT count): rowcount = 20.0, cumulative cost = {20.0 rows, 20.0 > cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 1343 > {code} > Here is part of the explain plan for the count(dir0) and count(dir1) in the > same select: > {code} > 00-00Screen : rowType = RecordType(BIGINT EXPR$0, BIGINT EXPR$1): > rowcount = 60.0, cumulative cost = {1206.0 rows, 15606.0 cpu, 0.0 io, 0.0 > network, 0.0 memory}, id = 1623 > 00-01 Project(EXPR$0=[$0], EXPR$1=[$1]) : rowType = RecordType(BIGINT > EXPR$0, BIGINT EXPR$1): rowcount = 60.0, cumulative cost = {1200.0 rows, > 15600.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 1622 > 00-02StreamAgg(group=[{}], EXPR$0=[COUNT($0)], EXPR$1=[COUNT($1)]) : > rowType = RecordType(BIGINT EXPR$0, BIGINT EXPR$1): rowcount = 60.0, > cumulative cost = {1200.0 rows, 15600.0 cpu, 0.0 io, 0.0 network, 0.0 > memory}, id = 1621 > 00-03 Scan(groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath > [path=maprfs:/drill/testdata/min_max_dir/1999/Apr/voter20.parquet/0_0_0.parquet], > ReadEntryWithPath > [path=maprfs:/drill/testdata/min_max_dir/1999/MAR/voter15.parquet/0_0_0.parquet], > ReadEntryWithPath > [path=maprfs:/drill/testdata/min_max_dir/1985/jan/voter5.parquet/0_0_0.parquet], > ReadEntryWithPath > [path=maprfs:/drill/testdata/min_max_dir/1985/apr/voter60.parquet/0_0_0.parquet],..., > ReadEntryWithPath > [path=maprfs:/drill/testdata/min_max_dir/2014/jul/voter35.parquet/0_0_0.parquet]], > selectionRoot=maprfs:/drill/testdata/min_max_dir, numFiles=16, > usedMetadataFile=false, columns=[`dir0`, `dir1`]]]) : rowType = > RecordType(ANY dir0, ANY dir1): rowcount = 600.0, cumulative cost = {600.0 > rows, 1200.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 1620 > {code} > Notice that in the first case, > "org.apache.drill.exec.store.pojo.PojoRecordReader" is used. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (DRILL-4735) Count(dir0) on parquet returns 0 result
[ https://issues.apache.org/jira/browse/DRILL-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jinfeng Ni updated DRILL-4735: -- Labels: ready-to-commit (was: ) > Count(dir0) on parquet returns 0 result > --- > > Key: DRILL-4735 > URL: https://issues.apache.org/jira/browse/DRILL-4735 > Project: Apache Drill > Issue Type: Bug > Components: Query Planning & Optimization, Storage - Parquet >Affects Versions: 1.0.0, 1.4.0, 1.6.0, 1.7.0 >Reporter: Krystal >Assignee: Arina Ielchiieva >Priority: Critical > Labels: ready-to-commit > > Selecting a count of dir0, dir1, etc against a parquet directory returns 0 > rows. > select count(dir0) from `min_max_dir`; > +-+ > | EXPR$0 | > +-+ > | 0 | > +-+ > select count(dir1) from `min_max_dir`; > +-+ > | EXPR$0 | > +-+ > | 0 | > +-+ > If I put both dir0 and dir1 in the same select, it returns expected result: > select count(dir0), count(dir1) from `min_max_dir`; > +-+-+ > | EXPR$0 | EXPR$1 | > +-+-+ > | 600 | 600 | > +-+-+ > Here is the physical plan for count(dir0) query: > {code} > 00-00Screen : rowType = RecordType(BIGINT EXPR$0): rowcount = 20.0, > cumulative cost = {22.0 rows, 22.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id > = 1346 > 00-01 Project(EXPR$0=[$0]) : rowType = RecordType(BIGINT EXPR$0): > rowcount = 20.0, cumulative cost = {20.0 rows, 20.0 cpu, 0.0 io, 0.0 network, > 0.0 memory}, id = 1345 > 00-02Project(EXPR$0=[$0]) : rowType = RecordType(BIGINT EXPR$0): > rowcount = 20.0, cumulative cost = {20.0 rows, 20.0 cpu, 0.0 io, 0.0 network, > 0.0 memory}, id = 1344 > 00-03 > Scan(groupscan=[org.apache.drill.exec.store.pojo.PojoRecordReader@3da85d3b[columns > = null, isStarQuery = false, isSkipQuery = false]]) : rowType = > RecordType(BIGINT count): rowcount = 20.0, cumulative cost = {20.0 rows, 20.0 > cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 1343 > {code} > Here is part of the explain plan for the count(dir0) and count(dir1) in the > same select: > {code} > 00-00Screen : rowType = RecordType(BIGINT EXPR$0, BIGINT EXPR$1): > rowcount = 60.0, cumulative cost = {1206.0 rows, 15606.0 cpu, 0.0 io, 0.0 > network, 0.0 memory}, id = 1623 > 00-01 Project(EXPR$0=[$0], EXPR$1=[$1]) : rowType = RecordType(BIGINT > EXPR$0, BIGINT EXPR$1): rowcount = 60.0, cumulative cost = {1200.0 rows, > 15600.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 1622 > 00-02StreamAgg(group=[{}], EXPR$0=[COUNT($0)], EXPR$1=[COUNT($1)]) : > rowType = RecordType(BIGINT EXPR$0, BIGINT EXPR$1): rowcount = 60.0, > cumulative cost = {1200.0 rows, 15600.0 cpu, 0.0 io, 0.0 network, 0.0 > memory}, id = 1621 > 00-03 Scan(groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath > [path=maprfs:/drill/testdata/min_max_dir/1999/Apr/voter20.parquet/0_0_0.parquet], > ReadEntryWithPath > [path=maprfs:/drill/testdata/min_max_dir/1999/MAR/voter15.parquet/0_0_0.parquet], > ReadEntryWithPath > [path=maprfs:/drill/testdata/min_max_dir/1985/jan/voter5.parquet/0_0_0.parquet], > ReadEntryWithPath > [path=maprfs:/drill/testdata/min_max_dir/1985/apr/voter60.parquet/0_0_0.parquet],..., > ReadEntryWithPath > [path=maprfs:/drill/testdata/min_max_dir/2014/jul/voter35.parquet/0_0_0.parquet]], > selectionRoot=maprfs:/drill/testdata/min_max_dir, numFiles=16, > usedMetadataFile=false, columns=[`dir0`, `dir1`]]]) : rowType = > RecordType(ANY dir0, ANY dir1): rowcount = 600.0, cumulative cost = {600.0 > rows, 1200.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 1620 > {code} > Notice that in the first case, > "org.apache.drill.exec.store.pojo.PojoRecordReader" is used. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (DRILL-4211) Column aliases not pushed down to JDBC stores in some cases when Drill expects aliased columns to be returned.
[ https://issues.apache.org/jira/browse/DRILL-4211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122525#comment-16122525 ] Timothy Farkas edited comment on DRILL-4211 at 8/12/17 12:39 AM: - After discussing with Paul and Jinfeng, this issue can be fixed by pushing the whole select portion of the query down into the JDBC operator. And required the select portion of the query to alias potentially conflicting columns. was (Author: timothyfarkas): After discussing with Paul and Juinfeng, this issue can be fixed by pushing the whole select portion of the query down into the JDBC operator. And required the select portion of the query to alias potentially conflicting columns. > Column aliases not pushed down to JDBC stores in some cases when Drill > expects aliased columns to be returned. > -- > > Key: DRILL-4211 > URL: https://issues.apache.org/jira/browse/DRILL-4211 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Relational Operators >Affects Versions: 1.3.0, 1.11.0 > Environment: Postgres db storage >Reporter: Robert Hamilton-Smith >Assignee: Timothy Farkas > Labels: newbie > Fix For: 1.12.0 > > > When making an sql statement that incorporates a join to a table and then a > self join to that table to get a parent value , Drill brings back > inconsistent results. > Here is the sql in postgres with correct output: > {code:sql} > select trx.categoryguid, > cat.categoryname, w1.categoryname as parentcat > from transactions trx > join categories cat on (cat.CATEGORYGUID = trx.CATEGORYGUID) > join categories w1 on (cat.categoryparentguid = w1.categoryguid) > where cat.categoryparentguid IS NOT NULL; > {code} > Output: > ||categoryid||categoryname||parentcategory|| > |id1|restaurants|food| > |id1|restaurants|food| > |id2|Coffee Shops|food| > |id2|Coffee Shops|food| > When run in Drill with correct storage prefix: > {code:sql} > select trx.categoryguid, > cat.categoryname, w1.categoryname as parentcat > from db.schema.transactions trx > join db.schema.categories cat on (cat.CATEGORYGUID = trx.CATEGORYGUID) > join db.schema.wpfm_categories w1 on (cat.categoryparentguid = > w1.categoryguid) > where cat.categoryparentguid IS NOT NULL > {code} > Results are: > ||categoryid||categoryname||parentcategory|| > |id1|restaurants|null| > |id1|restaurants|null| > |id2|Coffee Shops|null| > |id2|Coffee Shops|null| > Physical plan is: > {code:sql} > 00-00Screen : rowType = RecordType(VARCHAR(50) categoryguid, VARCHAR(50) > categoryname, VARCHAR(50) parentcat): rowcount = 100.0, cumulative cost = > {110.0 rows, 110.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 64293 > 00-01 Project(categoryguid=[$0], categoryname=[$1], parentcat=[$2]) : > rowType = RecordType(VARCHAR(50) categoryguid, VARCHAR(50) categoryname, > VARCHAR(50) parentcat): rowcount = 100.0, cumulative cost = {100.0 rows, > 100.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 64292 > 00-02Project(categoryguid=[$9], categoryname=[$41], parentcat=[$47]) > : rowType = RecordType(VARCHAR(50) categoryguid, VARCHAR(50) categoryname, > VARCHAR(50) parentcat): rowcount = 100.0, cumulative cost = {100.0 rows, > 100.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 64291 > 00-03 Jdbc(sql=[SELECT * > FROM "public"."transactions" > INNER JOIN (SELECT * > FROM "public"."categories" > WHERE "categoryparentguid" IS NOT NULL) AS "t" ON > "transactions"."categoryguid" = "t"."categoryguid" > INNER JOIN "public"."categories" AS "categories0" ON "t"."categoryparentguid" > = "categories0"."categoryguid"]) : rowType = RecordType(VARCHAR(255) > transactionguid, VARCHAR(255) relatedtransactionguid, VARCHAR(255) > transactioncode, DECIMAL(1, 0) transactionpending, VARCHAR(50) > transactionrefobjecttype, VARCHAR(255) transactionrefobjectguid, > VARCHAR(1024) transactionrefobjectvalue, TIMESTAMP(6) transactiondate, > VARCHAR(256) transactiondescription, VARCHAR(50) categoryguid, VARCHAR(3) > transactioncurrency, DECIMAL(15, 3) transactionoldbalance, DECIMAL(13, 3) > transactionamount, DECIMAL(15, 3) transactionnewbalance, VARCHAR(512) > transactionnotes, DECIMAL(2, 0) transactioninstrumenttype, VARCHAR(20) > transactioninstrumentsubtype, VARCHAR(20) transactioninstrumentcode, > VARCHAR(50) transactionorigpartyguid, VARCHAR(255) > transactionorigaccountguid, VARCHAR(50) transactionrecpartyguid, VARCHAR(255) > transactionrecaccountguid, VARCHAR(256) transactionstatementdesc, DECIMAL(1, > 0) transactionsplit, DECIMAL(1, 0) transactionduplicated, DECIMAL(1, 0) > transactionrecategorized, TIMESTAMP(6) transactioncreatedat, TIMESTAMP(6) > transactionupdatedat, VARCHAR(50) transactionmatrulerefobjtype,
[jira] [Commented] (DRILL-4398) SYSTEM ERROR: IllegalStateException: Memory was leaked by query
[ https://issues.apache.org/jira/browse/DRILL-4398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124376#comment-16124376 ] Muhammad Gelbana commented on DRILL-4398: - Thanks [~zfong] for the response. I'll open a new issue shortly. {noformat} Fragment 0:0 [Error Id: 0403a63e-86cc-428e-929b-e8dcd561a6bf on iWebStitchFixDev:31010] at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543) at org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:293) at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160) at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:262) at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.drill.exec.rpc.ChannelClosedException: Channel closed /72.55.136.6:31010 <--> /72.55.136.6:40834. at org.apache.drill.exec.rpc.RpcBus$ChannelClosedHandler.operationComplete(RpcBus.java:166) at org.apache.drill.exec.rpc.RpcBus$ChannelClosedHandler.operationComplete(RpcBus.java:146) at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680) at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603) at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563) at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:406) at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82) at io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:943) at io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:592) at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:584) at io.netty.channel.DefaultChannelPipeline$HeadContext.close(DefaultChannelPipeline.java:1099) at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:615) at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:600) at io.netty.channel.ChannelOutboundHandlerAdapter.close(ChannelOutboundHandlerAdapter.java:71) at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:615) at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:600) at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:466) at io.netty.handler.timeout.ReadTimeoutHandler.readTimedOut(ReadTimeoutHandler.java:187) at org.apache.drill.exec.rpc.BasicServer$LogggingReadTimeoutHandler.readTimedOut(BasicServer.java:121) at io.netty.handler.timeout.ReadTimeoutHandler$ReadTimeoutTask.run(ReadTimeoutHandler.java:212) at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38) at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:120) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) ... 1 more Suppressed: org.apache.drill.exec.rpc.RpcException: Failure sending message. at org.apache.drill.exec.rpc.RpcBus.send(RpcBus.java:126) at org.apache.drill.exec.rpc.user.UserServer$UserClientConnectionImpl.sendData(UserServer.java:285) at org.apache.drill.exec.ops.AccountingUserConnection.sendData(AccountingUserConnection.java:42) at org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:118) at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94) at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:232) at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:226) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:226) at
[jira] [Commented] (DRILL-5714) Fix NPE when mapr-db plugin is used in table function
[ https://issues.apache.org/jira/browse/DRILL-5714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124235#comment-16124235 ] ASF GitHub Bot commented on DRILL-5714: --- Github user paul-rogers commented on the issue: https://github.com/apache/drill/pull/902 Thanks for the explanation. Looks good. +1 (again) > Fix NPE when mapr-db plugin is used in table function > - > > Key: DRILL-5714 > URL: https://issues.apache.org/jira/browse/DRILL-5714 > Project: Apache Drill > Issue Type: Bug > Components: Storage - MapRDB >Affects Versions: 1.11.0 >Reporter: Arina Ielchiieva >Assignee: Arina Ielchiieva > Labels: ready-to-commit > Fix For: 1.12.0 > > > Query: > {noformat} > select * from table(mapr.`/tmp/test`(type=>'maprdb', allTextMode=>true)) > {noformat} > Error: > {noformat} > 2017-08-03 13:13:56,527 [267cde6b-6327-11b4-a4f3-f59a8d82ae17:frag:0:0] ERROR > o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: NullPointerException > Fragment 0:0 > [Error Id: 07937ad2-90ce-43e3-b9c1-2540e1df05fa on master:31010] > org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: > NullPointerException > Fragment 0:0 > [Error Id: 07937ad2-90ce-43e3-b9c1-2540e1df05fa on master:31010] > at > org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:550) > ~[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:295) > [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160) > [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:264) > [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] > at > org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) > [drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > [na:1.7.0_95] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > [na:1.7.0_95] > at java.lang.Thread.run(Thread.java:745) [na:1.7.0_95] > Caused by: org.apache.drill.common.exceptions.ExecutionSetupException: > java.lang.NullPointerException > at > org.apache.drill.exec.store.mapr.db.MapRDBScanBatchCreator.getBatch(MapRDBScanBatchCreator.java:52) > ~[drill-format-mapr-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] > at > org.apache.drill.exec.store.mapr.db.MapRDBScanBatchCreator.getBatch(MapRDBScanBatchCreator.java:36) > ~[drill-format-mapr-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] > at > org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:156) > ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] > at > org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:179) > ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] > at > org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:136) > ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] > at > org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:179) > ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] > at > org.apache.drill.exec.physical.impl.ImplCreator.getRootExec(ImplCreator.java:109) > ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] > at > org.apache.drill.exec.physical.impl.ImplCreator.getExec(ImplCreator.java:87) > ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:207) > [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] > ... 4 common frames omitted > Caused by: java.lang.NullPointerException: null > at > org.apache.drill.exec.store.mapr.db.MapRDBScanBatchCreator.getBatch(MapRDBScanBatchCreator.java:46) > ~[drill-format-mapr-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] > ... 12 common frames omitted > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5717) date time test cases is not Local independent
[ https://issues.apache.org/jira/browse/DRILL-5717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124371#comment-16124371 ] ASF GitHub Bot commented on DRILL-5717: --- Github user weijietong commented on the issue: https://github.com/apache/drill/pull/904 Yes, thanks for the notice ,have noticed those cases before ,the work is in process.some of that will be part of this PR. Others like sql_to_timestamp function need to discuss whether it should have a Local parameter or not set the specific UTC timezone when get the date. > date time test cases is not Local independent > - > > Key: DRILL-5717 > URL: https://issues.apache.org/jira/browse/DRILL-5717 > Project: Apache Drill > Issue Type: Bug > Components: Tools, Build & Test >Affects Versions: 1.9.0, 1.11.0 >Reporter: weijie.tong > > Some date time test cases like JodaDateValidatorTest is not Local > independent .This will cause other Local's users's test phase to fail. We > should let these test cases to be Local env independent. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5709) Provide a value vector method to convert a vector to nullable
[ https://issues.apache.org/jira/browse/DRILL-5709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124396#comment-16124396 ] ASF GitHub Bot commented on DRILL-5709: --- Github user Ben-Zvi commented on a diff in the pull request: https://github.com/apache/drill/pull/901#discussion_r132806738 --- Diff: exec/vector/src/main/java/org/apache/drill/exec/vector/BaseValueVector.java --- @@ -133,5 +134,21 @@ public static boolean checkBufRefs(final ValueVector vv) { public BufferAllocator getAllocator() { return allocator; } + + public static void fillBitsVector(UInt1Vector bits, int valueCount) { + +// Create a new bits vector, all values non-null + +bits.allocateNew(valueCount); +UInt1Vector.Mutator bitsMutator = bits.getMutator(); +for (int i = 0; i < valueCount; i++) { + bitsMutator.set(i, 1); +} + } --- End diff -- And need to add: bitsMutator.setValueCount(valueCount); > Provide a value vector method to convert a vector to nullable > - > > Key: DRILL-5709 > URL: https://issues.apache.org/jira/browse/DRILL-5709 > Project: Apache Drill > Issue Type: Improvement >Reporter: Paul Rogers >Assignee: Paul Rogers >Priority: Minor > Fix For: 1.12.0 > > > The hash agg spill work has need to convert a non-null scalar vector to the > nullable equivalent. For efficiency, the code wishes to simply transfer the > underlying data buffer(s), and create the required "bits" vector, rather than > generating code that does the transfer row-by-row. > The solution is to add a {{toNullable(ValueVector nullableVector)}} method to > the {{ValueVector}} class, then implement it where needed. > Since the target code only works with scalars (that is, no arrays, no maps, > no lists), the code only handles these cases, throwing an > {{UnsupportedOperationException}} in other cases. > Usage: > {code} > ValueVector nonNullableVector = // your non-nullable vector > MajorType type = MajorType.newBuilder(nonNullableVector.getType) > .setMode(DataMode.OPTIONAL) > .build(); > MaterializedField field = MaterializedField.create(name, type); > ValueVector nullableVector = TypeHelper.getNewVector(field, > oContext.getAllocator()); > nonNullableVector.toNullable(nullableVector); > // Data is now in nullableVector > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5709) Provide a value vector method to convert a vector to nullable
[ https://issues.apache.org/jira/browse/DRILL-5709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124397#comment-16124397 ] ASF GitHub Bot commented on DRILL-5709: --- Github user Ben-Zvi commented on a diff in the pull request: https://github.com/apache/drill/pull/901#discussion_r132806694 --- Diff: exec/vector/src/main/java/org/apache/drill/exec/vector/BaseValueVector.java --- @@ -133,5 +134,21 @@ public static boolean checkBufRefs(final ValueVector vv) { public BufferAllocator getAllocator() { return allocator; } + + public static void fillBitsVector(UInt1Vector bits, int valueCount) { + +// Create a new bits vector, all values non-null + +bits.allocateNew(valueCount); +UInt1Vector.Mutator bitsMutator = bits.getMutator(); +for (int i = 0; i < valueCount; i++) { + bitsMutator.set(i, 1); +} --- End diff -- This loop may be a bit expensive; Is there a way to mark the whole bitmap with '1's "at once" ? A la "memset()". For example, keep a constant bitmap (of max length - 64K) someplace, and here just dup/copy it whole. > Provide a value vector method to convert a vector to nullable > - > > Key: DRILL-5709 > URL: https://issues.apache.org/jira/browse/DRILL-5709 > Project: Apache Drill > Issue Type: Improvement >Reporter: Paul Rogers >Assignee: Paul Rogers >Priority: Minor > Fix For: 1.12.0 > > > The hash agg spill work has need to convert a non-null scalar vector to the > nullable equivalent. For efficiency, the code wishes to simply transfer the > underlying data buffer(s), and create the required "bits" vector, rather than > generating code that does the transfer row-by-row. > The solution is to add a {{toNullable(ValueVector nullableVector)}} method to > the {{ValueVector}} class, then implement it where needed. > Since the target code only works with scalars (that is, no arrays, no maps, > no lists), the code only handles these cases, throwing an > {{UnsupportedOperationException}} in other cases. > Usage: > {code} > ValueVector nonNullableVector = // your non-nullable vector > MajorType type = MajorType.newBuilder(nonNullableVector.getType) > .setMode(DataMode.OPTIONAL) > .build(); > MaterializedField field = MaterializedField.create(name, type); > ValueVector nullableVector = TypeHelper.getNewVector(field, > oContext.getAllocator()); > nonNullableVector.toNullable(nullableVector); > // Data is now in nullableVector > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5663) Drillbit fails to start when only keystore path is provided without keystore password.
[ https://issues.apache.org/jira/browse/DRILL-5663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124260#comment-16124260 ] ASF GitHub Bot commented on DRILL-5663: --- Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/874#discussion_r132800051 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/SSLConfig.java --- @@ -0,0 +1,77 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.drill.exec; + +import com.typesafe.config.Config; +import org.apache.drill.common.exceptions.DrillException; + +public class SSLConfig { + + private final String keystorePath; + + private final String keystorePassword; + + private final String truststorePath; + + private final String truststorePassword; + + + public SSLConfig(Config config) throws DrillException { + +keystorePath = config.getString(ExecConstants.HTTP_KEYSTORE_PATH).trim(); + +keystorePassword = config.getString(ExecConstants.HTTP_KEYSTORE_PASSWORD).trim(); + +truststorePath = config.getString(ExecConstants.HTTP_TRUSTSTORE_PATH).trim(); + +truststorePassword = config.getString(ExecConstants.HTTP_TRUSTSTORE_PASSWORD).trim(); + +/*If keystorePath or keystorePassword is provided in the configuration file use that*/ +if (!keystorePath.isEmpty() || !keystorePassword.isEmpty()) { + if (keystorePath.isEmpty()) { +throw new DrillException(" *.ssl.keyStorePath in the configuration file is empty, but *.ssl.keyStorePassword is set"); + } + else if (keystorePassword.isEmpty()){ +throw new DrillException(" *.ssl.keyStorePassword in the configuration file is empty, but *.ssl.keyStorePath is set "); + } + +} + } + + public boolean isSslValid() { return !keystorePath.isEmpty() && !keystorePassword.isEmpty(); } + + public String getKeyStorePath() { +return keystorePath; + } + + public String getKeyStorePassword() { +return keystorePassword; + } + + public boolean hasTrustStorePath() { return !truststorePath.isEmpty(); } + + public boolean hasTrustStorePassword() {return !truststorePassword.isEmpty(); } + + public String getTrustStorePath() { +return truststorePath; + } + + public String getTrustStorePassword() { +return truststorePassword; + } --- End diff -- Very minor suggestion: these simple returns can also be one-liners like the "has" methods. > Drillbit fails to start when only keystore path is provided without keystore > password. > -- > > Key: DRILL-5663 > URL: https://issues.apache.org/jira/browse/DRILL-5663 > Project: Apache Drill > Issue Type: Bug >Reporter: Sorabh Hamirwasia >Assignee: Sindhuri Ramanarayan Rayavaram > > When we configure keystore path without keystore password inside > drill-override.conf for WebServer, then Drillbit fails to start. We should > explicitly check for either both being present or both being absent. If any > one of them is only present then throw startup exception for Drill. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5663) Drillbit fails to start when only keystore path is provided without keystore password.
[ https://issues.apache.org/jira/browse/DRILL-5663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124256#comment-16124256 ] ASF GitHub Bot commented on DRILL-5663: --- Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/874#discussion_r132800512 --- Diff: exec/java-exec/src/test/java/org/apache/drill/exec/TestSSLConfig.java --- @@ -0,0 +1,108 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.drill.exec; + + +import org.apache.drill.common.exceptions.DrillException; +import org.apache.drill.test.ConfigBuilder; +import org.junit.Test; +import static junit.framework.TestCase.fail; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; + +public class TestSSLConfig { + + @Test + public void testMissingKeystorePath() throws Exception { + +ConfigBuilder config = new ConfigBuilder(); +config.put(ExecConstants.HTTP_KEYSTORE_PATH, ""); +config.put(ExecConstants.HTTP_KEYSTORE_PASSWORD, "root"); +try { + SSLConfig sslv = new SSLConfig(config.build()); + fail(); + //Expected +} catch (Exception e) { + assertTrue(e instanceof DrillException); +} + } + + @Test + public void testMissingKeystorePassword() throws Exception { + +ConfigBuilder config = new ConfigBuilder(); +config.put(ExecConstants.HTTP_KEYSTORE_PATH, "/root"); +config.put(ExecConstants.HTTP_KEYSTORE_PASSWORD, ""); +try { + SSLConfig sslv = new SSLConfig(config.build()); + fail(); + //Expected +} catch (Exception e) { + assertTrue(e instanceof DrillException); +} + } + + @Test + public void testForKeystoreConfig() throws Exception { + +ConfigBuilder config = new ConfigBuilder(); +config.put(ExecConstants.HTTP_KEYSTORE_PATH, "/root"); +config.put(ExecConstants.HTTP_KEYSTORE_PASSWORD, "root"); +try { + SSLConfig sslv = new SSLConfig(config.build()); + assertEquals("/root", sslv.getKeyStorePath()); + assertEquals("root", sslv.getKeyStorePassword()); +} catch (Exception e) { + fail(); + +} + } + + @Test + public void testForBackwardCompatability() throws Exception { + +ConfigBuilder config = new ConfigBuilder(); +config.put("javax.net.ssl.keyStore", "/root"); +config.put("javax.net.ssl.keyStorePassword", "root"); +SSLConfig sslv = new SSLConfig(config.build()); +assertEquals("/root",sslv.getKeyStorePath()); +assertEquals("root", sslv.getKeyStorePassword()); + } + + @Test + public void testForTrustStore() throws Exception { + +ConfigBuilder config = new ConfigBuilder(); +config.put(ExecConstants.HTTP_TRUSTSTORE_PATH, "/root"); +config.put(ExecConstants.HTTP_TRUSTSTORE_PASSWORD, "root"); +SSLConfig sslv = new SSLConfig(config.build()); +assertEquals(true, sslv.hasTrustStorePath()); +assertEquals(true,sslv.hasTrustStorePassword()); +assertEquals("/root",sslv.getTrustStorePath()); +assertEquals("root",sslv.getTrustStorePassword()); + } +} --- End diff -- Thanks for the tests! Maybe delete the extra lines at the end of the file... > Drillbit fails to start when only keystore path is provided without keystore > password. > -- > > Key: DRILL-5663 > URL: https://issues.apache.org/jira/browse/DRILL-5663 > Project: Apache Drill > Issue Type: Bug >Reporter: Sorabh Hamirwasia >Assignee: Sindhuri Ramanarayan Rayavaram > > When we configure keystore path without keystore password inside > drill-override.conf for WebServer, then Drillbit fails to
[jira] [Commented] (DRILL-5663) Drillbit fails to start when only keystore path is provided without keystore password.
[ https://issues.apache.org/jira/browse/DRILL-5663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124258#comment-16124258 ] ASF GitHub Bot commented on DRILL-5663: --- Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/874#discussion_r132800128 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/WebServer.java --- @@ -263,18 +272,17 @@ private ServerConnector createHttpsConnector() throws Exception { logger.info("Setting up HTTPS connector for web server"); final SslContextFactory sslContextFactory = new SslContextFactory(); +SSLConfig ssl = new SSLConfig(config); -if (config.hasPath(ExecConstants.HTTP_KEYSTORE_PATH) && - !Strings.isNullOrEmpty(config.getString(ExecConstants.HTTP_KEYSTORE_PATH))) { +if(ssl.isSslValid()){ --- End diff -- {code} if (...) { {code} > Drillbit fails to start when only keystore path is provided without keystore > password. > -- > > Key: DRILL-5663 > URL: https://issues.apache.org/jira/browse/DRILL-5663 > Project: Apache Drill > Issue Type: Bug >Reporter: Sorabh Hamirwasia >Assignee: Sindhuri Ramanarayan Rayavaram > > When we configure keystore path without keystore password inside > drill-override.conf for WebServer, then Drillbit fails to start. We should > explicitly check for either both being present or both being absent. If any > one of them is only present then throw startup exception for Drill. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5663) Drillbit fails to start when only keystore path is provided without keystore password.
[ https://issues.apache.org/jira/browse/DRILL-5663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124254#comment-16124254 ] ASF GitHub Bot commented on DRILL-5663: --- Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/874#discussion_r132799898 --- Diff: distribution/src/resources/drill-override-example.conf --- @@ -93,6 +93,13 @@ drill.exec: { credentials: true } }, + # Below SSL parameters need to be set for custom transport layer settings. + ssl{ +keyStorePath: "/keystore.file", +keyStorePassword: "ks_passwd", +trustStorePath: "/truststore.file", +trustStorePassword: "ts_passwd" --- End diff -- ssl{ --> ssl: { > Drillbit fails to start when only keystore path is provided without keystore > password. > -- > > Key: DRILL-5663 > URL: https://issues.apache.org/jira/browse/DRILL-5663 > Project: Apache Drill > Issue Type: Bug >Reporter: Sorabh Hamirwasia >Assignee: Sindhuri Ramanarayan Rayavaram > > When we configure keystore path without keystore password inside > drill-override.conf for WebServer, then Drillbit fails to start. We should > explicitly check for either both being present or both being absent. If any > one of them is only present then throw startup exception for Drill. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5663) Drillbit fails to start when only keystore path is provided without keystore password.
[ https://issues.apache.org/jira/browse/DRILL-5663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124257#comment-16124257 ] ASF GitHub Bot commented on DRILL-5663: --- Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/874#discussion_r132800461 --- Diff: exec/java-exec/src/main/resources/drill-module.conf --- @@ -52,6 +52,14 @@ drill.client: { drill.tmp-dir: "/tmp" drill.tmp-dir: ${?DRILL_TMP_DIR} +javax.net.ssl : + { + keyStore = "", + keyStorePassword = "", + trustStore = "", + trustStorePassword = "" + }, --- End diff -- {code} javax.net.ssl: { keyStore = "", {code} That is, adjust formatting of the header line and indent the values. > Drillbit fails to start when only keystore path is provided without keystore > password. > -- > > Key: DRILL-5663 > URL: https://issues.apache.org/jira/browse/DRILL-5663 > Project: Apache Drill > Issue Type: Bug >Reporter: Sorabh Hamirwasia >Assignee: Sindhuri Ramanarayan Rayavaram > > When we configure keystore path without keystore password inside > drill-override.conf for WebServer, then Drillbit fails to start. We should > explicitly check for either both being present or both being absent. If any > one of them is only present then throw startup exception for Drill. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5663) Drillbit fails to start when only keystore path is provided without keystore password.
[ https://issues.apache.org/jira/browse/DRILL-5663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124255#comment-16124255 ] ASF GitHub Bot commented on DRILL-5663: --- Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/874#discussion_r132800101 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/WebServer.java --- @@ -139,7 +142,13 @@ public void start() throws Exception { final ServerConnector serverConnector; if (config.getBoolean(ExecConstants.HTTP_ENABLE_SSL)) { - serverConnector = createHttpsConnector(); + try { +serverConnector = createHttpsConnector(); + } + catch(DrillException e){ +throw new DrillbitStartupException(e.getMessage(),e); --- End diff -- Space after comma. > Drillbit fails to start when only keystore path is provided without keystore > password. > -- > > Key: DRILL-5663 > URL: https://issues.apache.org/jira/browse/DRILL-5663 > Project: Apache Drill > Issue Type: Bug >Reporter: Sorabh Hamirwasia >Assignee: Sindhuri Ramanarayan Rayavaram > > When we configure keystore path without keystore password inside > drill-override.conf for WebServer, then Drillbit fails to start. We should > explicitly check for either both being present or both being absent. If any > one of them is only present then throw startup exception for Drill. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5663) Drillbit fails to start when only keystore path is provided without keystore password.
[ https://issues.apache.org/jira/browse/DRILL-5663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124259#comment-16124259 ] ASF GitHub Bot commented on DRILL-5663: --- Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/874#discussion_r132800092 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/WebServer.java --- @@ -139,7 +142,13 @@ public void start() throws Exception { final ServerConnector serverConnector; if (config.getBoolean(ExecConstants.HTTP_ENABLE_SSL)) { - serverConnector = createHttpsConnector(); + try { +serverConnector = createHttpsConnector(); + } + catch(DrillException e){ --- End diff -- {code} catch (DrillbitException e) { {code} (note spaces) > Drillbit fails to start when only keystore path is provided without keystore > password. > -- > > Key: DRILL-5663 > URL: https://issues.apache.org/jira/browse/DRILL-5663 > Project: Apache Drill > Issue Type: Bug >Reporter: Sorabh Hamirwasia >Assignee: Sindhuri Ramanarayan Rayavaram > > When we configure keystore path without keystore password inside > drill-override.conf for WebServer, then Drillbit fails to start. We should > explicitly check for either both being present or both being absent. If any > one of them is only present then throw startup exception for Drill. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (DRILL-5712) Update the pom files with dependency exclusions for commons-codec
[ https://issues.apache.org/jira/browse/DRILL-5712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sindhuri Ramanarayan Rayavaram updated DRILL-5712: -- Labels: ready-to-commit (was: ) > Update the pom files with dependency exclusions for commons-codec > - > > Key: DRILL-5712 > URL: https://issues.apache.org/jira/browse/DRILL-5712 > Project: Apache Drill > Issue Type: Bug > Components: Tools, Build & Test >Reporter: Sindhuri Ramanarayan Rayavaram >Assignee: Sindhuri Ramanarayan Rayavaram > Labels: ready-to-commit > > In java-exec, we are adding a dependency for commons-codec of version 1.10. > Other dependencies like hadoop-common, parquet-column etc are trying to > download different versions for common codec. Exclusions should be added for > common-codec in these dependencies. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5477) String functions (lower, upper, initcap) should work for UTF-8
[ https://issues.apache.org/jira/browse/DRILL-5477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124313#comment-16124313 ] Kunal Khatua commented on DRILL-5477: - [~smopsy], we should scope work for this, along with documentation tasks. There is no reference of the need for setting system property {{saffron.default.charset}}. > String functions (lower, upper, initcap) should work for UTF-8 > -- > > Key: DRILL-5477 > URL: https://issues.apache.org/jira/browse/DRILL-5477 > Project: Apache Drill > Issue Type: Improvement > Components: Functions - Drill >Affects Versions: 1.10.0 >Reporter: Arina Ielchiieva > > Drill string functions lower / upper / initcap work only for ASCII, but not > for UTF-8. UTF-8 is a multi-byte code that requires special encoding/decoding > to convert to Unicode characters. Without that encoding, these functions > won't work for Cyrillic, Greek or any other character set with upper/lower > distinctions. > Currently, when user applies these functions for UTF-8, Drill returns the > same value as was given. > Example: > {noformat} > select upper('привет') from (values(1)) -> привет > {noformat} > There is disabled unit test in > https://github.com/arina-ielchiieva/drill/blob/master/exec/java-exec/src/test/java/org/apache/drill/exec/expr/fn/impl/TestStringFunctions.java#L33 > which should be enabled once issue is fixed. > Please note, by default Calcite does not allow to use UTF-8. Update system > property *saffron.default.charset* to *UTF-16LE* if you encounter the > following error: > {noformat} > org.apache.drill.exec.rpc.RpcException: > org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: > CalciteException: Failed to encode 'привет' in character set 'ISO-8859-1' > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-3640) Drill JDBC driver support Statement.setQueryTimeout(int)
[ https://issues.apache.org/jira/browse/DRILL-3640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124352#comment-16124352 ] ASF GitHub Bot commented on DRILL-3640: --- Github user parthchandra commented on the issue: https://github.com/apache/drill/pull/858 Been looking at this and the first thing that occurs to me is that we are not too clear about what the timeout means in the context of ResultSet. The API specification is rather silent on that topic. The only reference I could find to this question is this one: http://mail-archives.apache.org/mod_mbox/db-derby-dev/200504.mbox/%3c426909ba.1060...@sun.com%3E We have the same choices: > 1. setQueryTimeout() only affects Statement.execute() > 2. setQueryTimeout() affects Statement.execute() and ResultSet.next(),starting from zero for each invocation > 3. setQueryTimeout() affects Statement.execute() and ResultSet.next(),accumulating time spent in each invocation My own inclination was to select #2 as the appropriate behavior. In fact that is what I assumed before I looked at the code. Laurent's suggestion to implement the timeout in DrillCursor provides this behavior and is a little bit easier to implement. OTOH, Kunal has chosen #3 as the right behavior. MySQL implements this behavior, BTW, so it is not going to be a surprise to end users. And he has already done the work. I'm +0 on this so far. Let me see if I can get a quick prototype to test things out. > Drill JDBC driver support Statement.setQueryTimeout(int) > > > Key: DRILL-3640 > URL: https://issues.apache.org/jira/browse/DRILL-3640 > Project: Apache Drill > Issue Type: New Feature > Components: Client - JDBC >Affects Versions: 1.2.0 >Reporter: Chun Chang >Assignee: Kunal Khatua > Fix For: 1.12.0 > > > It would be nice if we have this implemented. Run away queries can be > automatically canceled by setting the timeout. > java.sql.SQLFeatureNotSupportedException: Setting network timeout is not > supported. > at > org.apache.drill.jdbc.impl.DrillStatementImpl.setQueryTimeout(DrillStatementImpl.java:152) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (DRILL-4211) Column aliases not pushed down to JDBC stores in some cases when Drill expects aliased columns to be returned.
[ https://issues.apache.org/jira/browse/DRILL-4211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pritesh Maker updated DRILL-4211: - Fix Version/s: 1.12.0 > Column aliases not pushed down to JDBC stores in some cases when Drill > expects aliased columns to be returned. > -- > > Key: DRILL-4211 > URL: https://issues.apache.org/jira/browse/DRILL-4211 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Relational Operators >Affects Versions: 1.3.0, 1.11.0 > Environment: Postgres db storage >Reporter: Robert Hamilton-Smith >Assignee: Timothy Farkas > Labels: newbie > Fix For: 1.12.0 > > > When making an sql statement that incorporates a join to a table and then a > self join to that table to get a parent value , Drill brings back > inconsistent results. > Here is the sql in postgres with correct output: > {code:sql} > select trx.categoryguid, > cat.categoryname, w1.categoryname as parentcat > from transactions trx > join categories cat on (cat.CATEGORYGUID = trx.CATEGORYGUID) > join categories w1 on (cat.categoryparentguid = w1.categoryguid) > where cat.categoryparentguid IS NOT NULL; > {code} > Output: > ||categoryid||categoryname||parentcategory|| > |id1|restaurants|food| > |id1|restaurants|food| > |id2|Coffee Shops|food| > |id2|Coffee Shops|food| > When run in Drill with correct storage prefix: > {code:sql} > select trx.categoryguid, > cat.categoryname, w1.categoryname as parentcat > from db.schema.transactions trx > join db.schema.categories cat on (cat.CATEGORYGUID = trx.CATEGORYGUID) > join db.schema.wpfm_categories w1 on (cat.categoryparentguid = > w1.categoryguid) > where cat.categoryparentguid IS NOT NULL > {code} > Results are: > ||categoryid||categoryname||parentcategory|| > |id1|restaurants|null| > |id1|restaurants|null| > |id2|Coffee Shops|null| > |id2|Coffee Shops|null| > Physical plan is: > {code:sql} > 00-00Screen : rowType = RecordType(VARCHAR(50) categoryguid, VARCHAR(50) > categoryname, VARCHAR(50) parentcat): rowcount = 100.0, cumulative cost = > {110.0 rows, 110.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 64293 > 00-01 Project(categoryguid=[$0], categoryname=[$1], parentcat=[$2]) : > rowType = RecordType(VARCHAR(50) categoryguid, VARCHAR(50) categoryname, > VARCHAR(50) parentcat): rowcount = 100.0, cumulative cost = {100.0 rows, > 100.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 64292 > 00-02Project(categoryguid=[$9], categoryname=[$41], parentcat=[$47]) > : rowType = RecordType(VARCHAR(50) categoryguid, VARCHAR(50) categoryname, > VARCHAR(50) parentcat): rowcount = 100.0, cumulative cost = {100.0 rows, > 100.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 64291 > 00-03 Jdbc(sql=[SELECT * > FROM "public"."transactions" > INNER JOIN (SELECT * > FROM "public"."categories" > WHERE "categoryparentguid" IS NOT NULL) AS "t" ON > "transactions"."categoryguid" = "t"."categoryguid" > INNER JOIN "public"."categories" AS "categories0" ON "t"."categoryparentguid" > = "categories0"."categoryguid"]) : rowType = RecordType(VARCHAR(255) > transactionguid, VARCHAR(255) relatedtransactionguid, VARCHAR(255) > transactioncode, DECIMAL(1, 0) transactionpending, VARCHAR(50) > transactionrefobjecttype, VARCHAR(255) transactionrefobjectguid, > VARCHAR(1024) transactionrefobjectvalue, TIMESTAMP(6) transactiondate, > VARCHAR(256) transactiondescription, VARCHAR(50) categoryguid, VARCHAR(3) > transactioncurrency, DECIMAL(15, 3) transactionoldbalance, DECIMAL(13, 3) > transactionamount, DECIMAL(15, 3) transactionnewbalance, VARCHAR(512) > transactionnotes, DECIMAL(2, 0) transactioninstrumenttype, VARCHAR(20) > transactioninstrumentsubtype, VARCHAR(20) transactioninstrumentcode, > VARCHAR(50) transactionorigpartyguid, VARCHAR(255) > transactionorigaccountguid, VARCHAR(50) transactionrecpartyguid, VARCHAR(255) > transactionrecaccountguid, VARCHAR(256) transactionstatementdesc, DECIMAL(1, > 0) transactionsplit, DECIMAL(1, 0) transactionduplicated, DECIMAL(1, 0) > transactionrecategorized, TIMESTAMP(6) transactioncreatedat, TIMESTAMP(6) > transactionupdatedat, VARCHAR(50) transactionmatrulerefobjtype, VARCHAR(50) > transactionmatrulerefobjguid, VARCHAR(50) transactionmatrulerefobjvalue, > VARCHAR(50) transactionuserruleguid, DECIMAL(2, 0) transactionsplitorder, > TIMESTAMP(6) transactionprocessedat, TIMESTAMP(6) > transactioncategoryassignat, VARCHAR(50) transactionsystemcategoryguid, > VARCHAR(50) transactionorigmandateid, VARCHAR(100) fingerprint, VARCHAR(50) > categoryguid0, VARCHAR(50) categoryparentguid, DECIMAL(3, 0) categorytype, > VARCHAR(50) categoryname, VARCHAR(50) categorydescription, VARCHAR(50) > partyguid, VARCHAR(50) categoryguid1,