[jira] [Updated] (DRILL-7107) Unable to connect to Drill 1.15 through ZK
[ https://issues.apache.org/jira/browse/DRILL-7107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sorabh Hamirwasia updated DRILL-7107: - Affects Version/s: 1.15.0 > Unable to connect to Drill 1.15 through ZK > -- > > Key: DRILL-7107 > URL: https://issues.apache.org/jira/browse/DRILL-7107 > Project: Apache Drill > Issue Type: Bug > Components: Client - JDBC >Affects Versions: 1.15.0 >Reporter: Karthikeyan Manivannan >Assignee: Karthikeyan Manivannan >Priority: Major > Labels: ready-to-commit > Fix For: 1.16.0 > > > After upgrading to Drill 1.15, users are seeing they are no longer able to > connect to Drill using ZK quorum. They are getting the following "Unable to > setup ZK for client" error. > [~]$ sqlline -u "jdbc:drill:zk=172.16.2.165:5181;auth=maprsasl" > Error: Failure in connecting to Drill: > org.apache.drill.exec.rpc.RpcException: Failure setting up ZK for client. > (state=,code=0) > java.sql.SQLNonTransientConnectionException: Failure in connecting to Drill: > org.apache.drill.exec.rpc.RpcException: Failure setting up ZK for client. > at > org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:174) > at > org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:67) > at > org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:67) > at > org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138) > at org.apache.drill.jdbc.Driver.connect(Driver.java:72) > at sqlline.DatabaseConnection.connect(DatabaseConnection.java:130) > at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:179) > at sqlline.Commands.connect(Commands.java:1247) > at sqlline.Commands.connect(Commands.java:1139) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:38) > at sqlline.SqlLine.dispatch(SqlLine.java:722) > at sqlline.SqlLine.initArgs(SqlLine.java:416) > at sqlline.SqlLine.begin(SqlLine.java:514) > at sqlline.SqlLine.start(SqlLine.java:264) > at sqlline.SqlLine.main(SqlLine.java:195) > Caused by: org.apache.drill.exec.rpc.RpcException: Failure setting up ZK for > client. > at org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:340) > at > org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:165) > ... 18 more > Caused by: java.lang.NullPointerException > at > org.apache.drill.exec.coord.zk.ZKACLProviderFactory.findACLProvider(ZKACLProviderFactory.java:68) > at > org.apache.drill.exec.coord.zk.ZKACLProviderFactory.getACLProvider(ZKACLProviderFactory.java:47) > at > org.apache.drill.exec.coord.zk.ZKClusterCoordinator.(ZKClusterCoordinator.java:114) > at > org.apache.drill.exec.coord.zk.ZKClusterCoordinator.(ZKClusterCoordinator.java:86) > at org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:337) > ... 19 more > Apache Drill 1.15.0.0 > "This isn't your grandfather's SQL." > sqlline> > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-7107) Unable to connect to Drill 1.15 through ZK
[ https://issues.apache.org/jira/browse/DRILL-7107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sorabh Hamirwasia updated DRILL-7107: - Labels: ready-to-commit (was: ) > Unable to connect to Drill 1.15 through ZK > -- > > Key: DRILL-7107 > URL: https://issues.apache.org/jira/browse/DRILL-7107 > Project: Apache Drill > Issue Type: Bug >Reporter: Karthikeyan Manivannan >Assignee: Karthikeyan Manivannan >Priority: Major > Labels: ready-to-commit > Fix For: 1.16.0 > > > After upgrading to Drill 1.15, users are seeing they are no longer able to > connect to Drill using ZK quorum. They are getting the following "Unable to > setup ZK for client" error. > [~]$ sqlline -u "jdbc:drill:zk=172.16.2.165:5181;auth=maprsasl" > Error: Failure in connecting to Drill: > org.apache.drill.exec.rpc.RpcException: Failure setting up ZK for client. > (state=,code=0) > java.sql.SQLNonTransientConnectionException: Failure in connecting to Drill: > org.apache.drill.exec.rpc.RpcException: Failure setting up ZK for client. > at > org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:174) > at > org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:67) > at > org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:67) > at > org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138) > at org.apache.drill.jdbc.Driver.connect(Driver.java:72) > at sqlline.DatabaseConnection.connect(DatabaseConnection.java:130) > at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:179) > at sqlline.Commands.connect(Commands.java:1247) > at sqlline.Commands.connect(Commands.java:1139) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:38) > at sqlline.SqlLine.dispatch(SqlLine.java:722) > at sqlline.SqlLine.initArgs(SqlLine.java:416) > at sqlline.SqlLine.begin(SqlLine.java:514) > at sqlline.SqlLine.start(SqlLine.java:264) > at sqlline.SqlLine.main(SqlLine.java:195) > Caused by: org.apache.drill.exec.rpc.RpcException: Failure setting up ZK for > client. > at org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:340) > at > org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:165) > ... 18 more > Caused by: java.lang.NullPointerException > at > org.apache.drill.exec.coord.zk.ZKACLProviderFactory.findACLProvider(ZKACLProviderFactory.java:68) > at > org.apache.drill.exec.coord.zk.ZKACLProviderFactory.getACLProvider(ZKACLProviderFactory.java:47) > at > org.apache.drill.exec.coord.zk.ZKClusterCoordinator.(ZKClusterCoordinator.java:114) > at > org.apache.drill.exec.coord.zk.ZKClusterCoordinator.(ZKClusterCoordinator.java:86) > at org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:337) > ... 19 more > Apache Drill 1.15.0.0 > "This isn't your grandfather's SQL." > sqlline> > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-7127) Update hbase version for mapr profile
[
https://issues.apache.org/jira/browse/DRILL-7127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sorabh Hamirwasia updated DRILL-7127:
-
Labels: (was: ready-to-commit)
> Update hbase version for mapr profile
> -
>
> Key: DRILL-7127
> URL: https://issues.apache.org/jira/browse/DRILL-7127
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - HBase, Tools, Build & Test
>Affects Versions: 1.16.0
>Reporter: Abhishek Girish
>Assignee: Abhishek Girish
>Priority: Major
> Fix For: 1.16.0
>
>
> Current hbase version for mapr profile is {{1.1.1-mapr-1602-m7-5.2.0}} -
> which is over 3 years old. Needs to be updated to {{1.1.8-mapr-1808}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-7127) Update hbase version for mapr profile
[
https://issues.apache.org/jira/browse/DRILL-7127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sorabh Hamirwasia updated DRILL-7127:
-
Labels: ready-to-commit (was: )
> Update hbase version for mapr profile
> -
>
> Key: DRILL-7127
> URL: https://issues.apache.org/jira/browse/DRILL-7127
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - HBase, Tools, Build & Test
>Affects Versions: 1.16.0
>Reporter: Abhishek Girish
>Assignee: Abhishek Girish
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.16.0
>
>
> Current hbase version for mapr profile is {{1.1.1-mapr-1602-m7-5.2.0}} -
> which is over 3 years old. Needs to be updated to {{1.1.8-mapr-1808}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-7126) Contrib format-ltsv is not being included in distribution
[ https://issues.apache.org/jira/browse/DRILL-7126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sorabh Hamirwasia updated DRILL-7126: - Labels: ready-to-commit (was: ) > Contrib format-ltsv is not being included in distribution > - > > Key: DRILL-7126 > URL: https://issues.apache.org/jira/browse/DRILL-7126 > Project: Apache Drill > Issue Type: Bug > Components: Tools, Build & Test >Affects Versions: 1.16.0 >Reporter: Abhishek Girish >Assignee: Abhishek Girish >Priority: Major > Labels: ready-to-commit > Fix For: 1.16.0 > > > Unable to add the ltsv format in the dfs storage plugin. Looks like it's a > build distribution issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-7125) REFRESH TABLE METADATA fails after upgrade from Drill 1.13.0 to Drill 1.15.0
[
https://issues.apache.org/jira/browse/DRILL-7125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sorabh Hamirwasia updated DRILL-7125:
-
Reviewer: Aman Sinha
> REFRESH TABLE METADATA fails after upgrade from Drill 1.13.0 to Drill 1.15.0
>
>
> Key: DRILL-7125
> URL: https://issues.apache.org/jira/browse/DRILL-7125
> Project: Apache Drill
> Issue Type: Bug
> Components: Metadata
>Affects Versions: 1.14.0, 1.15.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sorabh Hamirwasia
>Priority: Major
> Fix For: 1.16.0
>
>
> REFRESH TABLE METADATA command worked successfully on Drill 1.13.0, however
> after upgrade Drill 1.15.0 there are errors sometime.
> {code:java}
> In sqlline logging in as regular user "alice" or Drill process user "admin"
> gives the same error (permission denied)
> If this helps, here's also what I am seeing on sqlline
> Error message contains random but valid user's names other than the user
> (Alice) that logged in to refresh the metadata. Looks like during the refresh
> metadata drillbits seems to incorrectly try the metadata generation as some
> random user which obviously does not have write access
> 2019-03-12 15:27:20,564 [2377cdd9-dd6e-d213-de1a-70b50d3641d7:frag:0:0] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 2377cdd9-dd6e-d213-de1a-70b50d3641d7:0:0: State change requested RUNNING -->
> FINISHED
> 2019-03-12 15:27:20,564 [2377cdd9-dd6e-d213-de1a-70b50d3641d7:frag:0:0] INFO
> o.a.d.e.w.f.FragmentStatusReporter -
> 2377cdd9-dd6e-d213-de1a-70b50d3641d7:0:0: State to report: FINISHED
> 2019-03-12 15:27:23,032 [2377cdb3-86cc-438d-8ada-787d2a84df9a:foreman] INFO
> o.a.drill.exec.work.foreman.Foreman - Query text for query with id
> 2377cdb3-86cc-438d-8ada-787d2a84df9a issued by alice: REFRESH TABLE METADATA
> dfs.root.`/user/alice/logs/hive/warehouse/detail`
> 2019-03-12 15:27:23,350 [2377cdb3-86cc-438d-8ada-787d2a84df9a:foreman] ERROR
> o.a.d.e.s.parquet.metadata.Metadata - Failed to read
> 'file://user/alice/logs/hive/warehouse/detail/.drill.parquet_metadata_directories'
> metadata file
> java.io.IOException: 2879.5854742.1036302960
> /user/alice/logs/hive/warehouse/detail/file1/.drill.parquet_metadata
> (Permission denied)
> at com.mapr.fs.Inode.throwIfFailed(Inode.java:390)
> ~[maprfs-6.1.0-mapr.jar:na]
> at com.mapr.fs.Inode.flushPages(Inode.java:505)
> ~[maprfs-6.1.0-mapr.jar:na]
> at com.mapr.fs.Inode.releaseDirty(Inode.java:583)
> ~[maprfs-6.1.0-mapr.jar:na]
> at
> com.mapr.fs.MapRFsOutStream.dropCurrentPage(MapRFsOutStream.java:73)
> ~[maprfs-6.1.0-mapr.jar:na]
> at com.mapr.fs.MapRFsOutStream.write(MapRFsOutStream.java:85)
> ~[maprfs-6.1.0-mapr.jar:na]
> at
> com.mapr.fs.MapRFsDataOutputStream.write(MapRFsDataOutputStream.java:39)
> ~[maprfs-6.1.0-mapr.jar:na]
> at
> com.fasterxml.jackson.core.json.UTF8JsonGenerator._flushBuffer(UTF8JsonGenerator.java:2085)
> ~[jackson-core-2.9.5.jar:2.9.5]
> at
> com.fasterxml.jackson.core.json.UTF8JsonGenerator.flush(UTF8JsonGenerator.java:1097)
> ~[jackson-core-2.9.5.jar:2.9.5]
> at
> com.fasterxml.jackson.databind.ObjectMapper.writeValue(ObjectMapper.java:2645)
> ~[jackson-databind-2.9.5.jar:2.9.5]
> at
> com.fasterxml.jackson.core.base.GeneratorBase.writeObject(GeneratorBase.java:381)
> ~[jackson-core-2.9.5.jar:2.9.5]
> at
> com.fasterxml.jackson.core.JsonGenerator.writeObjectField(JsonGenerator.java:1726)
> ~[jackson-core-2.9.5.jar:2.9.5]
> at
> org.apache.drill.exec.store.parquet.metadata.Metadata_V3$ColumnMetadata_v3$Serializer.serialize(Metadata_V3.java:448)
> ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
> at
> org.apache.drill.exec.store.parquet.metadata.Metadata_V3$ColumnMetadata_v3$Serializer.serialize(Metadata_V3.java:417)
> ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
> at
> com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119)
> ~[jackson-databind-2.9.5.jar:2.9.5]
> at
> com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79)
> ~[jackson-databind-2.9.5.jar:2.9.5]
> at
> com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18)
> ~[jackson-databind-2.9.5.jar:2.9.5]
> at
> com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:727)
> ~[jackson-databind-2.9.5.jar:2.9.5]
> at
> com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719)
> ~[jackson-databind-2.9.5.jar:2.9.5]
> at
> com.fasterxml.jackson.databind.ser.BeanSerializer.serializ
[jira] [Updated] (DRILL-7125) REFRESH TABLE METADATA fails after upgrade from Drill 1.13.0 to Drill 1.15.0
[
https://issues.apache.org/jira/browse/DRILL-7125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sorabh Hamirwasia updated DRILL-7125:
-
Labels: ready-to-commit (was: )
> REFRESH TABLE METADATA fails after upgrade from Drill 1.13.0 to Drill 1.15.0
>
>
> Key: DRILL-7125
> URL: https://issues.apache.org/jira/browse/DRILL-7125
> Project: Apache Drill
> Issue Type: Bug
> Components: Metadata
>Affects Versions: 1.14.0, 1.15.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sorabh Hamirwasia
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.16.0
>
>
> REFRESH TABLE METADATA command worked successfully on Drill 1.13.0, however
> after upgrade Drill 1.15.0 there are errors sometime.
> {code:java}
> In sqlline logging in as regular user "alice" or Drill process user "admin"
> gives the same error (permission denied)
> If this helps, here's also what I am seeing on sqlline
> Error message contains random but valid user's names other than the user
> (Alice) that logged in to refresh the metadata. Looks like during the refresh
> metadata drillbits seems to incorrectly try the metadata generation as some
> random user which obviously does not have write access
> 2019-03-12 15:27:20,564 [2377cdd9-dd6e-d213-de1a-70b50d3641d7:frag:0:0] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 2377cdd9-dd6e-d213-de1a-70b50d3641d7:0:0: State change requested RUNNING -->
> FINISHED
> 2019-03-12 15:27:20,564 [2377cdd9-dd6e-d213-de1a-70b50d3641d7:frag:0:0] INFO
> o.a.d.e.w.f.FragmentStatusReporter -
> 2377cdd9-dd6e-d213-de1a-70b50d3641d7:0:0: State to report: FINISHED
> 2019-03-12 15:27:23,032 [2377cdb3-86cc-438d-8ada-787d2a84df9a:foreman] INFO
> o.a.drill.exec.work.foreman.Foreman - Query text for query with id
> 2377cdb3-86cc-438d-8ada-787d2a84df9a issued by alice: REFRESH TABLE METADATA
> dfs.root.`/user/alice/logs/hive/warehouse/detail`
> 2019-03-12 15:27:23,350 [2377cdb3-86cc-438d-8ada-787d2a84df9a:foreman] ERROR
> o.a.d.e.s.parquet.metadata.Metadata - Failed to read
> 'file://user/alice/logs/hive/warehouse/detail/.drill.parquet_metadata_directories'
> metadata file
> java.io.IOException: 2879.5854742.1036302960
> /user/alice/logs/hive/warehouse/detail/file1/.drill.parquet_metadata
> (Permission denied)
> at com.mapr.fs.Inode.throwIfFailed(Inode.java:390)
> ~[maprfs-6.1.0-mapr.jar:na]
> at com.mapr.fs.Inode.flushPages(Inode.java:505)
> ~[maprfs-6.1.0-mapr.jar:na]
> at com.mapr.fs.Inode.releaseDirty(Inode.java:583)
> ~[maprfs-6.1.0-mapr.jar:na]
> at
> com.mapr.fs.MapRFsOutStream.dropCurrentPage(MapRFsOutStream.java:73)
> ~[maprfs-6.1.0-mapr.jar:na]
> at com.mapr.fs.MapRFsOutStream.write(MapRFsOutStream.java:85)
> ~[maprfs-6.1.0-mapr.jar:na]
> at
> com.mapr.fs.MapRFsDataOutputStream.write(MapRFsDataOutputStream.java:39)
> ~[maprfs-6.1.0-mapr.jar:na]
> at
> com.fasterxml.jackson.core.json.UTF8JsonGenerator._flushBuffer(UTF8JsonGenerator.java:2085)
> ~[jackson-core-2.9.5.jar:2.9.5]
> at
> com.fasterxml.jackson.core.json.UTF8JsonGenerator.flush(UTF8JsonGenerator.java:1097)
> ~[jackson-core-2.9.5.jar:2.9.5]
> at
> com.fasterxml.jackson.databind.ObjectMapper.writeValue(ObjectMapper.java:2645)
> ~[jackson-databind-2.9.5.jar:2.9.5]
> at
> com.fasterxml.jackson.core.base.GeneratorBase.writeObject(GeneratorBase.java:381)
> ~[jackson-core-2.9.5.jar:2.9.5]
> at
> com.fasterxml.jackson.core.JsonGenerator.writeObjectField(JsonGenerator.java:1726)
> ~[jackson-core-2.9.5.jar:2.9.5]
> at
> org.apache.drill.exec.store.parquet.metadata.Metadata_V3$ColumnMetadata_v3$Serializer.serialize(Metadata_V3.java:448)
> ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
> at
> org.apache.drill.exec.store.parquet.metadata.Metadata_V3$ColumnMetadata_v3$Serializer.serialize(Metadata_V3.java:417)
> ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
> at
> com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119)
> ~[jackson-databind-2.9.5.jar:2.9.5]
> at
> com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79)
> ~[jackson-databind-2.9.5.jar:2.9.5]
> at
> com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18)
> ~[jackson-databind-2.9.5.jar:2.9.5]
> at
> com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:727)
> ~[jackson-databind-2.9.5.jar:2.9.5]
> at
> com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719)
> ~[jackson-databind-2.9.5.jar:2.9.5]
> at
> com.fas
[jira] [Commented] (DRILL-7125) REFRESH TABLE METADATA fails after upgrade from Drill 1.13.0 to Drill 1.15.0
[ https://issues.apache.org/jira/browse/DRILL-7125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16797522#comment-16797522 ] Sorabh Hamirwasia commented on DRILL-7125: -- *Sorabh:* I was looking into this customer issue where *Refresh Metadata* query is failing after upgrade from 1.13.0 to 1.15.0. Below are my findings: 1) Until 1.13 all the metadata cache file related access was done using process user. This was enforced by DRILL-4143 [PR: Link|https://github.com/apache/drill/pull/470/files] .The way it was done is all the filesystem instance passed to Metadata class was ignore and instead FileSystem instance created using process user context was only used. 2) In 1.14 with DRILL-6331, [PR: Link|https://github.com/apache/drill/commit/c6549e58859397c88cb1de61b4f6eee52a07ed0c#diff-45bfe599bc468a9fd85319266b1222dd] there was some refactoring done to support filter/limit push down in cases where Hive Native Parquet reader is used. This change removed the fix done in DRILL-4143 above. I guess it expected that all the caller will call with correct FileSystem object references which was actually not the case. With change 2 above from 1.14 it looks like metadata refresh call (both explicit and auto refresh) will not work if issued with a user other than Drill process user. Probably all the tests are run as process user and hence the issue was never caught. Arina - Do you remember if there was any other reason to remove the fix in DRILL-6331 which was made as part of DRILL-4143 ? If not then to revert back to older behavior quick fix will be again to ignore the filesystem instances passed in all the Metadata class api's and just use the process user FileSystem instance. But this has security issues which were never fixed and needs to be thought through to address it. *For example:* 1) REFRESH Metadata query can be executed by any user. Even the ones who doesn't own's the actual data. 2) Metadata file itself is not encrypted hence any user can read this file. 3) If we encrypt the metadata file such that only Drill can understand. Still a user who doesn't have access to actual data can run query which is totally served by Metadata file and can get access to those information. Like count(*) and min/max. *Arina:* Hi Sorabh, The original logic was not removed. Metadata classes started to accept file system object but for Drill tables it is still created under process user, its just ParquetGroupScan is responsible for this. For the refresh metadata use case, I guess we need to create file system under process user in the RefreshMetadataHandler [2]. Some background: Metadata class is responsible for creating metadata files and reading metadata from footers. While for Drill tables having file system under process user is acceptable, for Hive table special file system is created for each path [3]. That's why responsibility of creating proper file system was moved from Metadata class to Group Scan classes. I guess I just missed that RefreshMetadataHandler also calls methods from Metadata class and passes file system. [1] [https://github.com/apache/drill/blob/3d29faf81da593035f6bd38dd56d48e719afe7d4/exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetGroupScan.java#L241] [2] [https://github.com/apache/drill/blob/5aa38a51d90998234b4ca46434ce225df72addc5/exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/RefreshMetadataHandler.java#L119] [3] [https://github.com/apache/drill/blob/3d29faf81da593035f6bd38dd56d48e719afe7d4/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveDrillNativeParquetScan.java#L207] *Sorabh:* Hi Arina*,* Thanks for your input. With DRILL-6331 the expectation was all the caller of Metadata class api needs to pass filesystem created under correct user context. But Chunhui's fix basically just made a change in Metadata class itself to create a process user related filesystem ignoring the passed object from caller (probably to avoid changing in all the caller. Kind of a hack). I know that RefreshMetadataHandler code doesn't create process user file system, but that will still not fix the issue. Because in that handler it calls getTable() [1] which internally calls WorkspaceSchemaFactory::create ---> isReadable ---> readBlockMeta using the query user (need not be process user). Since readBlockMeta is used by both autoRefresh code path and this RefreshMetadata code path internally it tries to actually write metadata file if modification time is obsolete. Since this write happens using the query user file system object it fails. Please see the stack trace below. Just creating file system object with process user in RefreshMetadataHandler [2] will help to resolve the case when REFRESH command was executed by a process user only. Now for all other queries when same *isReada
[jira] [Created] (DRILL-7125) REFRESH TABLE METADATA fails after upgrade from Drill 1.13.0 to Drill 1.15.0
Sorabh Hamirwasia created DRILL-7125:
Summary: REFRESH TABLE METADATA fails after upgrade from Drill
1.13.0 to Drill 1.15.0
Key: DRILL-7125
URL: https://issues.apache.org/jira/browse/DRILL-7125
Project: Apache Drill
Issue Type: Bug
Components: Metadata
Affects Versions: 1.15.0, 1.14.0
Reporter: Sorabh Hamirwasia
Assignee: Sorabh Hamirwasia
Fix For: 1.16.0
REFRESH TABLE METADATA command worked successfully on Drill 1.13.0, however
after upgrade Drill 1.15.0 there are errors sometime.
{code:java}
In sqlline logging in as regular user "alice" or Drill process user "admin"
gives the same error (permission denied)
If this helps, here's also what I am seeing on sqlline
Error message contains random but valid user's names other than the user
(Alice) that logged in to refresh the metadata. Looks like during the refresh
metadata drillbits seems to incorrectly try the metadata generation as some
random user which obviously does not have write access
2019-03-12 15:27:20,564 [2377cdd9-dd6e-d213-de1a-70b50d3641d7:frag:0:0] INFO
o.a.d.e.w.fragment.FragmentExecutor - 2377cdd9-dd6e-d213-de1a-70b50d3641d7:0:0:
State change requested RUNNING --> FINISHED
2019-03-12 15:27:20,564 [2377cdd9-dd6e-d213-de1a-70b50d3641d7:frag:0:0] INFO
o.a.d.e.w.f.FragmentStatusReporter - 2377cdd9-dd6e-d213-de1a-70b50d3641d7:0:0:
State to report: FINISHED
2019-03-12 15:27:23,032 [2377cdb3-86cc-438d-8ada-787d2a84df9a:foreman] INFO
o.a.drill.exec.work.foreman.Foreman - Query text for query with id
2377cdb3-86cc-438d-8ada-787d2a84df9a issued by alice: REFRESH TABLE METADATA
dfs.root.`/user/alice/logs/hive/warehouse/detail`
2019-03-12 15:27:23,350 [2377cdb3-86cc-438d-8ada-787d2a84df9a:foreman] ERROR
o.a.d.e.s.parquet.metadata.Metadata - Failed to read
'file://user/alice/logs/hive/warehouse/detail/.drill.parquet_metadata_directories'
metadata file
java.io.IOException: 2879.5854742.1036302960
/user/alice/logs/hive/warehouse/detail/file1/.drill.parquet_metadata
(Permission denied)
at com.mapr.fs.Inode.throwIfFailed(Inode.java:390)
~[maprfs-6.1.0-mapr.jar:na]
at com.mapr.fs.Inode.flushPages(Inode.java:505)
~[maprfs-6.1.0-mapr.jar:na]
at com.mapr.fs.Inode.releaseDirty(Inode.java:583)
~[maprfs-6.1.0-mapr.jar:na]
at com.mapr.fs.MapRFsOutStream.dropCurrentPage(MapRFsOutStream.java:73)
~[maprfs-6.1.0-mapr.jar:na]
at com.mapr.fs.MapRFsOutStream.write(MapRFsOutStream.java:85)
~[maprfs-6.1.0-mapr.jar:na]
at
com.mapr.fs.MapRFsDataOutputStream.write(MapRFsDataOutputStream.java:39)
~[maprfs-6.1.0-mapr.jar:na]
at
com.fasterxml.jackson.core.json.UTF8JsonGenerator._flushBuffer(UTF8JsonGenerator.java:2085)
~[jackson-core-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.core.json.UTF8JsonGenerator.flush(UTF8JsonGenerator.java:1097)
~[jackson-core-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.databind.ObjectMapper.writeValue(ObjectMapper.java:2645)
~[jackson-databind-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.core.base.GeneratorBase.writeObject(GeneratorBase.java:381)
~[jackson-core-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.core.JsonGenerator.writeObjectField(JsonGenerator.java:1726)
~[jackson-core-2.9.5.jar:2.9.5]
at
org.apache.drill.exec.store.parquet.metadata.Metadata_V3$ColumnMetadata_v3$Serializer.serialize(Metadata_V3.java:448)
~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at
org.apache.drill.exec.store.parquet.metadata.Metadata_V3$ColumnMetadata_v3$Serializer.serialize(Metadata_V3.java:417)
~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at
com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119)
~[jackson-databind-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79)
~[jackson-databind-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18)
~[jackson-databind-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:727)
~[jackson-databind-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719)
~[jackson-databind-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155)
~[jackson-databind-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119)
~[jackson-databind-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79)
~[jackson-databind-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.
[jira] [Updated] (DRILL-6501) Revert/modify fix for DRILL-6212 after CALCITE-2223 is fixed
[ https://issues.apache.org/jira/browse/DRILL-6501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sorabh Hamirwasia updated DRILL-6501: - Fix Version/s: (was: 1.16.0) 1.17.0 > Revert/modify fix for DRILL-6212 after CALCITE-2223 is fixed > > > Key: DRILL-6501 > URL: https://issues.apache.org/jira/browse/DRILL-6501 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.14.0 >Reporter: Gautam Parai >Assignee: Gautam Parai >Priority: Major > Fix For: 1.17.0 > > Original Estimate: 48h > Remaining Estimate: 48h > > DRILL-6212 is a temporary fix to alleviate issues due to CALCITE-2223. Once, > CALCITE-2223 is fixed this change needs to be reverted back which would > require DrillProjectMergeRule to go back to extending the ProjectMergeRule. > Please take a look at how CALCITE-2223 is eventually fixed (as of now it is > still not clear which fix is the way to do). Depending on the fix we may need > to additional work to integrate these changes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-7032) Ignore corrupt rows in a PCAP file
[ https://issues.apache.org/jira/browse/DRILL-7032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sorabh Hamirwasia updated DRILL-7032: - Reviewer: Arina Ielchiieva > Ignore corrupt rows in a PCAP file > -- > > Key: DRILL-7032 > URL: https://issues.apache.org/jira/browse/DRILL-7032 > Project: Apache Drill > Issue Type: Improvement > Components: Functions - Drill >Affects Versions: 1.15.0 > Environment: OS: Ubuntu 18.4 > Drill version: 1.15.0 > Java(TM) SE Runtime Environment (build 1.8.0_191-b12) >Reporter: Giovanni Conte >Assignee: Charles Givre >Priority: Major > Fix For: 1.16.0 > > > Would be useful for Drill to have some ability to ignore corrupt rows in a > PCAP file instead of trow the java exception. > This is because there are many pcap files with corrupted lines and this > funcionality will avoid to do a pre-fixing of the packet-captures (example > attached file). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6992) Support column histogram statistics
[ https://issues.apache.org/jira/browse/DRILL-6992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sorabh Hamirwasia updated DRILL-6992: - Fix Version/s: 1.16.0 > Support column histogram statistics > --- > > Key: DRILL-6992 > URL: https://issues.apache.org/jira/browse/DRILL-6992 > Project: Apache Drill > Issue Type: New Feature > Components: Query Planning & Optimization >Affects Versions: 1.15.0 >Reporter: Aman Sinha >Assignee: Aman Sinha >Priority: Major > Fix For: 1.16.0 > > > As a follow-up to > [DRILL-1328|https://issues.apache.org/jira/browse/DRILL-1328] which is adding > NDV (num distinct values) support and creating the framework for statistics, > we also need Histograms. These are needed for range predicates selectivity > estimation as well as equality predicates when there is non-uniform > distribution of data. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6956) Maintain a single entry for Drill Version in the pom file
[ https://issues.apache.org/jira/browse/DRILL-6956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sorabh Hamirwasia updated DRILL-6956: - Fix Version/s: (was: 1.16.0) 1.17.0 > Maintain a single entry for Drill Version in the pom file > - > > Key: DRILL-6956 > URL: https://issues.apache.org/jira/browse/DRILL-6956 > Project: Apache Drill > Issue Type: Improvement > Components: Tools, Build & Test >Affects Versions: 1.15.0 >Reporter: Kunal Khatua >Assignee: Kunal Khatua >Priority: Major > Fix For: 1.17.0 > > > Currently, updating the version information for a Drill release involves > updating 30+ pom files. > The right way would be to use the Multi Module Setup for Maven CI. > https://maven.apache.org/maven-ci-friendly.html#Multi_Module_Setup -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6899) Fix timestamp issues in unit tests ignored with DRILL-6833
[
https://issues.apache.org/jira/browse/DRILL-6899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sorabh Hamirwasia updated DRILL-6899:
-
Fix Version/s: (was: 1.16.0)
1.17.0
> Fix timestamp issues in unit tests ignored with DRILL-6833
> --
>
> Key: DRILL-6899
> URL: https://issues.apache.org/jira/browse/DRILL-6899
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.15.0
>Reporter: Gautam Parai
>Assignee: Gautam Parai
>Priority: Major
> Fix For: 1.17.0
>
> Original Estimate: 96h
> Remaining Estimate: 96h
>
> {{The following tests were disabled in the PR for DRILL-6833}}
> {{IndexPlanTest.testCastTimestampPlan() - Re-enable after the MapRDB format
> plugin issue is fixed.}}
> {{IndexPlanTest.testRowkeyJoinPushdown_13() - Re-enable the testcase after
> fixing the execution issue with HashJoin used as Rowkeyjoin.}}
> {{IndexPlanTest.testRowkeyJoinPushdown_12() - Remove the testcase since the
> SemiJoin transformation makes the rowkeyjoinpushdown transformation invalid.}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-7089) Implement caching of BaseMetadata classes
[
https://issues.apache.org/jira/browse/DRILL-7089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sorabh Hamirwasia updated DRILL-7089:
-
Reviewer: Aman Sinha
> Implement caching of BaseMetadata classes
> -
>
> Key: DRILL-7089
> URL: https://issues.apache.org/jira/browse/DRILL-7089
> Project: Apache Drill
> Issue Type: Sub-task
>Affects Versions: 1.16.0
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.16.0
>
>
> In the scope of DRILL-6852 were introduced new classes for metadata usage.
> These classes may be reused in other GroupScan instances to preserve heap
> usage for the case when metadata is large.
> The idea is to store {{BaseMetadata}} inheritors in {{DrillTable}} and pass
> them to the {{GroupScan}}, so in the scope of the single query, it will be
> possible to reuse them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6032) Use RecordBatchSizer to estimate size of columns in HashAgg
[ https://issues.apache.org/jira/browse/DRILL-6032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sorabh Hamirwasia updated DRILL-6032: - Fix Version/s: (was: 1.16.0) 1.17.0 > Use RecordBatchSizer to estimate size of columns in HashAgg > --- > > Key: DRILL-6032 > URL: https://issues.apache.org/jira/browse/DRILL-6032 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Fix For: 1.17.0 > > > We need to use the RecordBatchSize to estimate the size of columns in the > Partition batches created by HashAgg. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-7107) Unable to connect to Drill 1.15 through ZK
[ https://issues.apache.org/jira/browse/DRILL-7107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sorabh Hamirwasia updated DRILL-7107: - Reviewer: Sorabh Hamirwasia > Unable to connect to Drill 1.15 through ZK > -- > > Key: DRILL-7107 > URL: https://issues.apache.org/jira/browse/DRILL-7107 > Project: Apache Drill > Issue Type: Bug >Reporter: Karthikeyan Manivannan >Assignee: Karthikeyan Manivannan >Priority: Major > Fix For: 1.16.0 > > > After upgrading to Drill 1.15, users are seeing they are no longer able to > connect to Drill using ZK quorum. They are getting the following "Unable to > setup ZK for client" error. > [~]$ sqlline -u "jdbc:drill:zk=172.16.2.165:5181;auth=maprsasl" > Error: Failure in connecting to Drill: > org.apache.drill.exec.rpc.RpcException: Failure setting up ZK for client. > (state=,code=0) > java.sql.SQLNonTransientConnectionException: Failure in connecting to Drill: > org.apache.drill.exec.rpc.RpcException: Failure setting up ZK for client. > at > org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:174) > at > org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:67) > at > org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:67) > at > org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138) > at org.apache.drill.jdbc.Driver.connect(Driver.java:72) > at sqlline.DatabaseConnection.connect(DatabaseConnection.java:130) > at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:179) > at sqlline.Commands.connect(Commands.java:1247) > at sqlline.Commands.connect(Commands.java:1139) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:38) > at sqlline.SqlLine.dispatch(SqlLine.java:722) > at sqlline.SqlLine.initArgs(SqlLine.java:416) > at sqlline.SqlLine.begin(SqlLine.java:514) > at sqlline.SqlLine.start(SqlLine.java:264) > at sqlline.SqlLine.main(SqlLine.java:195) > Caused by: org.apache.drill.exec.rpc.RpcException: Failure setting up ZK for > client. > at org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:340) > at > org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:165) > ... 18 more > Caused by: java.lang.NullPointerException > at > org.apache.drill.exec.coord.zk.ZKACLProviderFactory.findACLProvider(ZKACLProviderFactory.java:68) > at > org.apache.drill.exec.coord.zk.ZKACLProviderFactory.getACLProvider(ZKACLProviderFactory.java:47) > at > org.apache.drill.exec.coord.zk.ZKClusterCoordinator.(ZKClusterCoordinator.java:114) > at > org.apache.drill.exec.coord.zk.ZKClusterCoordinator.(ZKClusterCoordinator.java:86) > at org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:337) > ... 19 more > Apache Drill 1.15.0.0 > "This isn't your grandfather's SQL." > sqlline> > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6540) Upgrade to HADOOP-3.1 libraries
[ https://issues.apache.org/jira/browse/DRILL-6540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sorabh Hamirwasia updated DRILL-6540: - Fix Version/s: (was: 1.16.0) 1.17.0 > Upgrade to HADOOP-3.1 libraries > > > Key: DRILL-6540 > URL: https://issues.apache.org/jira/browse/DRILL-6540 > Project: Apache Drill > Issue Type: Improvement > Components: Tools, Build & Test >Affects Versions: 1.14.0 >Reporter: Vitalii Diravka >Assignee: Vitalii Diravka >Priority: Major > Fix For: 1.17.0 > > > Currently Drill uses 2.7.4 version of hadoop libraries (hadoop-common, > hadoop-hdfs, hadoop-annotations, hadoop-aws, hadoop-yarn-api, hadoop-client, > hadoop-yarn-client). > Half of year ago the [Hadoop > 3.0|https://hadoop.apache.org/docs/r3.0.0/index.html] was released and > recently it was an update - [Hadoop > 3.2.0|https://hadoop.apache.org/docs/r3.2.0/]. > To use Drill under Hadoop3.0 distribution we need this upgrade. Also the > newer version includes new features, which can be useful for Drill. > This upgrade is also needed to leverage the newest version of Zookeeper > libraries and Hive 3.1 version. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-7098) File Metadata Metastore Plugin
[ https://issues.apache.org/jira/browse/DRILL-7098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sorabh Hamirwasia updated DRILL-7098: - Fix Version/s: (was: 1.16.0) 2.0.0 > File Metadata Metastore Plugin > -- > > Key: DRILL-7098 > URL: https://issues.apache.org/jira/browse/DRILL-7098 > Project: Apache Drill > Issue Type: Sub-task > Components: Server, Metadata >Reporter: Vitalii Diravka >Assignee: Vitalii Diravka >Priority: Major > Labels: Metastore > Fix For: 2.0.0 > > > DRILL-6852 introduces Drill Metastore API. > The second step is to create internal Drill Metastore mechanism (and File > Metastore Plugin), which will involve Metastore API and can be extended for > using by other Storage Plugins. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-7113) Issue with filtering null values from MapRDB-JSON
[
https://issues.apache.org/jira/browse/DRILL-7113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sorabh Hamirwasia updated DRILL-7113:
-
Reviewer: Hanumath Rao Maduri
> Issue with filtering null values from MapRDB-JSON
> -
>
> Key: DRILL-7113
> URL: https://issues.apache.org/jira/browse/DRILL-7113
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.15.0
>Reporter: Hanumath Rao Maduri
>Assignee: Aman Sinha
>Priority: Major
> Fix For: 1.16.0
>
>
> When the Drill is querying documents from MapRDBJSON that contain fields with
> null value, it returns the wrong result.
> The issue is locally reproduced.
> Please find the repro steps:
> [1] Create a MaprDBJSON table. Say '/tmp/dmdb2/'.
> [2] Insert the following sample records to table:
> {code:java}
> insert --table /tmp/dmdb2/ --value '{"_id": "1", "label": "person",
> "confidence": 0.24}'
> insert --table /tmp/dmdb2/ --value '{"_id": "2", "label": "person2"}'
> insert --table /tmp/dmdb2/ --value '{"_id": "3", "label": "person3",
> "confidence": 0.54}'
> insert --table /tmp/dmdb2/ --value '{"_id": "4", "label": "person4",
> "confidence": null}'
> {code}
> We can see that for field 'confidence' document 1 has value 0.24, document 3
> has value 0.54, document 2 does not have the field and document 4 has the
> field with value null.
> [3] Query the table from DRILL.
> *Query 1:*
> {code:java}
> 0: jdbc:drill:> select label,confidence from dfs.tmp.dmdb2;
> +--+-+
> | label | confidence |
> +--+-+
> | person | 0.24|
> | person2 | null|
> | person3 | 0.54|
> | person4 | null|
> +--+-+
> 4 rows selected (0.2 seconds)
> {code}
> *Query 2:*
> {code:java}
> 0: jdbc:drill:> select * from dfs.tmp.dmdb2;
> +--+-+--+
> | _id | confidence | label |
> +--+-+--+
> | 1| 0.24| person |
> | 2| null| person2 |
> | 3| 0.54| person3 |
> | 4| null| person4 |
> +--+-+--+
> 4 rows selected (0.174 seconds)
> {code}
> *Query 3:*
> {code:java}
> 0: jdbc:drill:> select label,confidence from dfs.tmp.dmdb2 where confidence
> is not null;
> +--+-+
> | label | confidence |
> +--+-+
> | person | 0.24|
> | person3 | 0.54|
> | person4 | null|
> +--+-+
> 3 rows selected (0.192 seconds)
> {code}
> *Query 4:*
> {code:java}
> 0: jdbc:drill:> select label,confidence from dfs.tmp.dmdb2 where confidence
> is null;
> +--+-+
> | label | confidence |
> +--+-+
> | person2 | null|
> +--+-+
> 1 row selected (0.262 seconds)
> {code}
> As you can see, Query 3 which queries for all documents with confidence value
> 'is not null', returns a document with null value.
> *Other observation:*
> Querying the same data using DRILL without MapRDB provides the correct
> result.
> For example, create 4 different JSON files with following data:
> {"label": "person", "confidence": 0.24} \{"label": "person2"} \{"label":
> "person3", "confidence": 0.54} \{"label": "person4", "confidence": null}
> Query it directly using DRILL:
> *Query 5:*
> {code:java}
> 0: jdbc:drill:> select label,confidence from dfs.tmp.t2;
> +--+-+
> | label | confidence |
> +--+-+
> | person4 | null|
> | person3 | 0.54|
> | person2 | null|
> | person | 0.24|
> +--+-+
> 4 rows selected (0.203 seconds)
> {code}
> *Query 6:*
> {code:java}
> 0: jdbc:drill:> select label,confidence from dfs.tmp.t2 where confidence is
> null;
> +--+-+
> | label | confidence |
> +--+-+
> | person4 | null|
> | person2 | null|
> +--+-+
> 2 rows selected (0.352 seconds)
> {code}
> *Query 7:*
> {code:java}
> 0: jdbc:drill:> select label,confidence from dfs.tmp.t2 where confidence is
> not null;
> +--+-+
> | label | confidence |
> +--+-+
> | person3 | 0.54|
> | person | 0.24|
> +--+-+
> 2 rows selected (0.265 seconds)
> {code}
> As seen in query 6 & 7, it returns the correct result.
> I believe the issue is at the MapRDB layer where it is fetching the results.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-7021) HTTPD Throws NPE and Doesn't Recognize Timeformat
[ https://issues.apache.org/jira/browse/DRILL-7021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16795149#comment-16795149 ] Sorabh Hamirwasia commented on DRILL-7021: -- Merged into master with commit id: bf1bdec6069f6fdd2132608450357edea47d328c > HTTPD Throws NPE and Doesn't Recognize Timeformat > - > > Key: DRILL-7021 > URL: https://issues.apache.org/jira/browse/DRILL-7021 > Project: Apache Drill > Issue Type: Bug > Components: Storage - Other >Affects Versions: 1.16.0 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Labels: ready-to-commit > Fix For: 1.16.0 > > > The HTTPD format plugin throws a NPE when you attempt to query all fields and > enumerate them in the query. > The query below will throw the NPE: > ``` > SELECT `request_referer_query`, > . . . . . . .semicolon> `request_referer_ref`, > . . . . . . .semicolon> `request_firstline_uri_port`, > . . . . . . .semicolon> `request_firstline_method`, > . . . . . . .semicolon> `request_firstline_uri_protocol`, > . . . . . . .semicolon> `request_receive_time_millisecond`, > . . . . . . .semicolon> `request_receive_time_day__utc`, > . . . . . . .semicolon> `request_receive_time_year__utc`, > . . . . . . .semicolon> `request_receive_time_weekofweekyear__utc`, > . . . . . . .semicolon> `request_receive_time_second`, > . . . . . . .semicolon> `connection_client_host`, > . . . . . . .semicolon> `request_referer_userinfo`, > . . . . . . .semicolon> `request_referer_path`, > . . . . . . .semicolon> `request_referer_host`, > . . . . . . .semicolon> `request_receive_time_monthname`, > . . . . . . .semicolon> `request_receive_time_month__utc`, > . . . . . . .semicolon> `request_firstline_uri_ref`, > . . . . . . .semicolon> `request_firstline_protocol_version`, > . . . . . . .semicolon> `request_receive_time_minute`, > . . . . . . .semicolon> `request_firstline`, > . . . . . . .semicolon> `request_receive_time_day`, > . . . . . . .semicolon> `request_referer_port`, > . . . . . . .semicolon> `request_receive_time_year`, > . . . . . . .semicolon> `request_referer_query_$`, > . . . . . . .semicolon> `request_firstline_uri_query_$`, > . . . . . . .semicolon> `request_firstline_uri`, > . . . . . . .semicolon> `request_receive_time_month`, > . . . . . . .semicolon> `request_receive_time_weekofweekyear`, > . . . . . . .semicolon> `request_firstline_uri_userinfo`, > . . . . . . .semicolon> `request_referer`, > . . . . . . .semicolon> `request_receive_time_epoch`, > . . . . . . .semicolon> `request_referer_protocol`, > . . . . . . .semicolon> `request_receive_time_monthname__utc`, > . . . . . . .semicolon> `connection_client_logname`, > . . . . . . .semicolon> `request_receive_time`, > . . . . . . .semicolon> `request_firstline_protocol`, > . . . . . . .semicolon> `request_receive_time_hour`, > . . . . . . .semicolon> `request_firstline_uri_host`, > . . . . . . .semicolon> `request_firstline_uri_path`, > . . . . . . .semicolon> `request_user-agent`, > . . . . . . .semicolon> `request_receive_time_hour__utc`, > . . . . . . .semicolon> `request_receive_time_second__utc`, > . . . . . . .semicolon> `request_receive_time_weekyear`, > . . . . . . .semicolon> `request_receive_time_timezone`, > . . . . . . .semicolon> `request_receive_time_weekyear__utc`, > . . . . . . .semicolon> `response_body_bytesclf`, > . . . . . . .semicolon> `connection_client_user`, > . . . . . . .semicolon> `request_receive_time_millisecond__utc`, > . . . . . . .semicolon> `request_status_last`, > . . . . . . .semicolon> `request_firstline_uri_query`, > . . . . . . .semicolon> `request_receive_time_minute__utc` > . . . . . . .semicolon> FROM `dfs.drillclass`.`hackers-access.httpd` > ``` > The cause for the NPE is that several fields were missing from a type map in > the format plugin. > Separately, the format plugin is not recognizing the time stamp and is not > parsing dates as time formats. > Oh... and the unit tests suck. Sorry. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-7105) Error while building the Drill native client
[
https://issues.apache.org/jira/browse/DRILL-7105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sorabh Hamirwasia updated DRILL-7105:
-
Component/s: Client - C++
> Error while building the Drill native client
>
>
> Key: DRILL-7105
> URL: https://issues.apache.org/jira/browse/DRILL-7105
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - C++
>Affects Versions: 1.16.0
>Reporter: Anton Gozhiy
>Assignee: Anton Gozhiy
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.16.0
>
>
> *Steps:*
> # cd contrib/native/client
> # mkdir build
> # cd build && cmake -std=c++11 -G "Unix Makefiles" -D CMAKE_BUILD_TYPE=Debug
> ..
> # make
> *Expected result:*
> The native client is built successfully.
> *Actual result:*
> Error happens:
> {noformat}
> [ 4%] Built target y2038
> [ 7%] Building CXX object
> src/protobuf/CMakeFiles/protomsgs.dir/BitControl.pb.cc.o
> In file included from /usr/include/c++/5/mutex:35:0,
> from /usr/local/include/google/protobuf/stubs/mutex.h:33,
> from /usr/local/include/google/protobuf/stubs/common.h:52,
> from
> /home/agozhiy/git_repo/drill/contrib/native/client/src/protobuf/BitControl.pb.h:9,
> from
> /home/agozhiy/git_repo/drill/contrib/native/client/src/protobuf/BitControl.pb.cc:4:
> /usr/include/c++/5/bits/c++0x_warning.h:32:2: error: #error This file
> requires compiler and library support for the ISO C++ 2011 standard. This
> support must be enabled with the -std=c++11 or -std=gnu++11 compiler options.
> #error This file requires compiler and library support \
> ^
> In file included from /usr/local/include/google/protobuf/stubs/common.h:52:0,
> from
> /home/agozhiy/git_repo/drill/contrib/native/client/src/protobuf/BitControl.pb.h:9,
> from
> /home/agozhiy/git_repo/drill/contrib/native/client/src/protobuf/BitControl.pb.cc:4:
> /usr/local/include/google/protobuf/stubs/mutex.h:58:8: error: 'mutex' in
> namespace 'std' does not name a type
>std::mutex mu_;
> ^
> /usr/local/include/google/protobuf/stubs/mutex.h: In member function 'void
> google::protobuf::internal::WrappedMutex::Lock()':
> /usr/local/include/google/protobuf/stubs/mutex.h:51:17: error: 'mu_' was not
> declared in this scope
>void Lock() { mu_.lock(); }
> ^
> /usr/local/include/google/protobuf/stubs/mutex.h: In member function 'void
> google::protobuf::internal::WrappedMutex::Unlock()':
> /usr/local/include/google/protobuf/stubs/mutex.h:52:19: error: 'mu_' was not
> declared in this scope
>void Unlock() { mu_.unlock(); }
>^
> /usr/local/include/google/protobuf/stubs/mutex.h: At global scope:
> /usr/local/include/google/protobuf/stubs/mutex.h:61:7: error: expected
> nested-name-specifier before 'Mutex'
> using Mutex = WrappedMutex;
>^
> /usr/local/include/google/protobuf/stubs/mutex.h:66:28: error: expected ')'
> before '*' token
>explicit MutexLock(Mutex *mu) : mu_(mu) { this->mu_->Lock(); }
> ^
> /usr/local/include/google/protobuf/stubs/mutex.h:69:3: error: 'Mutex' does
> not name a type
>Mutex *const mu_;
>^
> /usr/local/include/google/protobuf/stubs/mutex.h: In destructor
> 'google::protobuf::internal::MutexLock::~MutexLock()':
> /usr/local/include/google/protobuf/stubs/mutex.h:67:24: error: 'class
> google::protobuf::internal::MutexLock' has no member named 'mu_'
>~MutexLock() { this->mu_->Unlock(); }
> ^
> /usr/local/include/google/protobuf/stubs/mutex.h: At global scope:
> /usr/local/include/google/protobuf/stubs/mutex.h:80:33: error: expected ')'
> before '*' token
>explicit MutexLockMaybe(Mutex *mu) :
> ^
> In file included from /usr/local/include/google/protobuf/arena.h:48:0,
> from
> /home/agozhiy/git_repo/drill/contrib/native/client/src/protobuf/BitControl.pb.h:23,
> from
> /home/agozhiy/git_repo/drill/contrib/native/client/src/protobuf/BitControl.pb.cc:4:
> /usr/include/c++/5/typeinfo:39:37: error: expected '}' before end of line
> /usr/include/c++/5/typeinfo:39:37: error: expected unqualified-id before end
> of line
> /usr/include/c++/5/typeinfo:39:37: error: expected '}' before end of line
> /usr/include/c++/5/typeinfo:39:37: error: expected '}' before end of line
> /usr/include/c++/5/typeinfo:39:37: error: expected '}' before end of line
> /usr/include/c++/5/typeinfo:39:37: error: expected declaration before end of
> line
> src/protobuf/CMakeFiles/protomsgs.dir/build.make:62: recipe for target
> 'src/protobuf/CMakeFiles/protomsgs.dir/BitControl.pb.cc.o' failed
> make[2]: *** [src/protobuf/CMakeFiles/protomsgs.dir/BitCont
[jira] [Updated] (DRILL-6707) Query with 10-way merge join fails with IllegalArgumentException
[
https://issues.apache.org/jira/browse/DRILL-6707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sorabh Hamirwasia updated DRILL-6707:
-
Labels: ready-to-commit (was: )
> Query with 10-way merge join fails with IllegalArgumentException
>
>
> Key: DRILL-6707
> URL: https://issues.apache.org/jira/browse/DRILL-6707
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators, Query Planning &
> Optimization
>Affects Versions: 1.15.0
>Reporter: Abhishek Girish
>Assignee: Boaz Ben-Zvi
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.16.0
>
> Attachments: drillbit.zip
>
>
> Query
> {code}
> SELECT *
> FROM si.tpch_sf1_parquet.customer C,
> si.tpch_sf1_parquet.orders O,
> si.tpch_sf1_parquet.lineitem L,
> si.tpch_sf1_parquet.part P,
> si.tpch_sf1_parquet.supplier S,
> si.tpch_sf1_parquet.partsupp PS,
> si.tpch_sf1_parquet.nation S_N,
> si.tpch_sf1_parquet.region S_R,
> si.tpch_sf1_parquet.nation C_N,
> si.tpch_sf1_parquet.region C_R
> WHEREC.C_CUSTKEY = O.O_CUSTKEY
> AND O.O_ORDERKEY = L.L_ORDERKEY
> AND L.L_PARTKEY = P.P_PARTKEY
> AND L.L_SUPPKEY = S.S_SUPPKEY
> AND P.P_PARTKEY = PS.PS_PARTKEY
> AND P.P_SUPPKEY = PS.PS_SUPPKEY
> AND S.S_NATIONKEY = S_N.N_NATIONKEY
> AND S_N.N_REGIONKEY = S_R.R_REGIONKEY
> AND C.C_NATIONKEY = C_N.N_NATIONKEY
> AND C_N.N_REGIONKEY = C_R.R_REGIONKEY
> {code}
> Plan
> {code}
> 00-00Screen : rowType = RecordType(DYNAMIC_STAR **, DYNAMIC_STAR **0,
> DYNAMIC_STAR **1, DYNAMIC_STAR **2, DYNAMIC_STAR **3, DYNAMIC_STAR **4,
> DYNAMIC_STAR **5, DYNAMIC_STAR **6, DYNAMIC_STAR **7, DYNAMIC_STAR **8):
> rowcount = 6001215.0, cumulative cost = {1.151087965E8 rows,
> 2.66710261332395E9 cpu, 3.198503E7 io, 5.172844544E11 network, 1.87681384E9
> memory}, id = 419943
> 00-01 ProjectAllowDup(**=[$0], **0=[$1], **1=[$2], **2=[$3], **3=[$4],
> **4=[$5], **5=[$6], **6=[$7], **7=[$8], **8=[$9]) : rowType =
> RecordType(DYNAMIC_STAR **, DYNAMIC_STAR **0, DYNAMIC_STAR **1, DYNAMIC_STAR
> **2, DYNAMIC_STAR **3, DYNAMIC_STAR **4, DYNAMIC_STAR **5, DYNAMIC_STAR **6,
> DYNAMIC_STAR **7, DYNAMIC_STAR **8): rowcount = 6001215.0, cumulative cost =
> {1.14508675E8 rows, 2.66650249182395E9 cpu, 3.198503E7 io, 5.172844544E11
> network, 1.87681384E9 memory}, id = 419942
> 00-02UnionExchange : rowType = RecordType(DYNAMIC_STAR T19¦¦**,
> DYNAMIC_STAR T18¦¦**, DYNAMIC_STAR T12¦¦**, DYNAMIC_STAR T17¦¦**,
> DYNAMIC_STAR T13¦¦**, DYNAMIC_STAR T16¦¦**, DYNAMIC_STAR T14¦¦**,
> DYNAMIC_STAR T15¦¦**, DYNAMIC_STAR T20¦¦**, DYNAMIC_STAR T21¦¦**): rowcount =
> 6001215.0, cumulative cost = {1.0850746E8 rows, 2.60649034182395E9 cpu,
> 3.198503E7 io, 5.172844544E11 network, 1.87681384E9 memory}, id = 419941
> 01-01 Project(T19¦¦**=[$0], T18¦¦**=[$3], T12¦¦**=[$6],
> T17¦¦**=[$10], T13¦¦**=[$13], T16¦¦**=[$16], T14¦¦**=[$19], T15¦¦**=[$22],
> T20¦¦**=[$24], T21¦¦**=[$27]) : rowType = RecordType(DYNAMIC_STAR T19¦¦**,
> DYNAMIC_STAR T18¦¦**, DYNAMIC_STAR T12¦¦**, DYNAMIC_STAR T17¦¦**,
> DYNAMIC_STAR T13¦¦**, DYNAMIC_STAR T16¦¦**, DYNAMIC_STAR T14¦¦**,
> DYNAMIC_STAR T15¦¦**, DYNAMIC_STAR T20¦¦**, DYNAMIC_STAR T21¦¦**): rowcount =
> 6001215.0, cumulative cost = {1.02506245E8 rows, 2.55848062182395E9 cpu,
> 3.198503E7 io, 2.71474688E11 network, 1.87681384E9 memory}, id = 419940
> 01-02Project(T19¦¦**=[$21], C_CUSTKEY=[$22], C_NATIONKEY=[$23],
> T18¦¦**=[$18], O_CUSTKEY=[$19], O_ORDERKEY=[$20], T12¦¦**=[$0],
> L_ORDERKEY=[$1], L_PARTKEY=[$2], L_SUPPKEY=[$3], T17¦¦**=[$15],
> P_PARTKEY=[$16], P_SUPPKEY=[$17], T13¦¦**=[$4], S_SUPPKEY=[$5],
> S_NATIONKEY=[$6], T16¦¦**=[$12], PS_PARTKEY=[$13], PS_SUPPKEY=[$14],
> T14¦¦**=[$7], N_NATIONKEY=[$8], N_REGIONKEY=[$9], T15¦¦**=[$10],
> R_REGIONKEY=[$11], T20¦¦**=[$24], N_NATIONKEY0=[$25], N_REGIONKEY0=[$26],
> T21¦¦**=[$27], R_REGIONKEY0=[$28]) : rowType = RecordType(DYNAMIC_STAR
> T19¦¦**, ANY C_CUSTKEY, ANY C_NATIONKEY, DYNAMIC_STAR T18¦¦**, ANY O_CUSTKEY,
> ANY O_ORDERKEY, DYNAMIC_STAR T12¦¦**, ANY L_ORDERKEY, ANY L_PARTKEY, ANY
> L_SUPPKEY, DYNAMIC_STAR T17¦¦**, ANY P_PARTKEY, ANY P_SUPPKEY, DYNAMIC_STAR
> T13¦¦**, ANY S_SUPPKEY, ANY S_NATIONKEY, DYNAMIC_STAR T16¦¦**, ANY
> PS_PARTKEY, ANY PS_SUPPKEY, DYNAMIC_STAR T14¦¦**, ANY N_NATIONKEY, ANY
> N_REGIONKEY, DYNAMIC_STAR T15¦¦**, ANY R_REGIONKEY, DYNAMIC_STAR T20¦¦**, ANY
> N_NATIONKEY0, ANY N_REGIONKEY0, DYNAMIC_STAR T21¦¦**, ANY R_REGIONKEY0):
> rowcount = 6001215.0, cumulative cost = {9.650503E7 rows, 2.49846847182395E9
> cpu, 3.198503E7 io, 2.71474688E11 network, 1.87681384E9 memory}, id = 419939
> 01-03 MergeJo
[jira] [Commented] (DRILL-6970) Issue with LogRegex format plugin where drillbuf was overflowing
[
https://issues.apache.org/jira/browse/DRILL-6970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16793167#comment-16793167
]
Sorabh Hamirwasia commented on DRILL-6970:
--
[~jccote] - Can you please re-open the PR ? Looks like Vitalli was asking for
squashing and rebasing it off master to start review.
> Issue with LogRegex format plugin where drillbuf was overflowing
> -
>
> Key: DRILL-6970
> URL: https://issues.apache.org/jira/browse/DRILL-6970
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.15.0
>Reporter: jean-claude
>Assignee: jean-claude
>Priority: Major
> Fix For: 1.16.0
>
>
> The log format plugin does re-allocate the drillbuf when it fills up. You can
> query small log files but larger ones will fail with this error:
> 0: jdbc:drill:zk=local> select * from dfs.root.`/prog/test.log`;
> Error: INTERNAL_ERROR ERROR: index: 32724, length: 108 (expected: range(0,
> 32768))
> Fragment 0:0
> Please, refer to logs for more information.
>
> I'm running drill-embeded. The log storage plugin is configured like so
> {code:java}
> "log": {
> "type": "logRegex",
> "regex": "(.+)",
> "extension": "log",
> "maxErrors": 10,
> "schema": [
> {
> "fieldName": "line"
> }
> ]
> },
> {code}
> The log files is very simple
> {code:java}
> jdsaljfldaksjfldsajfldasjflkjdsfldsjfljsdalfk
> jdsaljfldaksjfldsajfldasjflkjdsfldsjfljsdalfk
> jdsaljfldaksjfldsajfldasjflkjdsfldsjfljsdalfk
> jdsaljfldaksjfldsajfldasjflkjdsfldsjfljsdalfk
> jdsaljfldaksjfldsajfldasjflkjdsfldsjfljsdalfk
> jdsaljfldaksjfldsajfldasjflkjdsfldsjfljsdalfk
> ...{code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6970) Issue with LogRegex format plugin where drillbuf was overflowing
[
https://issues.apache.org/jira/browse/DRILL-6970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sorabh Hamirwasia updated DRILL-6970:
-
Labels: (was: ready-to-commit)
> Issue with LogRegex format plugin where drillbuf was overflowing
> -
>
> Key: DRILL-6970
> URL: https://issues.apache.org/jira/browse/DRILL-6970
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.15.0
>Reporter: jean-claude
>Assignee: jean-claude
>Priority: Major
> Fix For: 1.16.0
>
>
> The log format plugin does re-allocate the drillbuf when it fills up. You can
> query small log files but larger ones will fail with this error:
> 0: jdbc:drill:zk=local> select * from dfs.root.`/prog/test.log`;
> Error: INTERNAL_ERROR ERROR: index: 32724, length: 108 (expected: range(0,
> 32768))
> Fragment 0:0
> Please, refer to logs for more information.
>
> I'm running drill-embeded. The log storage plugin is configured like so
> {code:java}
> "log": {
> "type": "logRegex",
> "regex": "(.+)",
> "extension": "log",
> "maxErrors": 10,
> "schema": [
> {
> "fieldName": "line"
> }
> ]
> },
> {code}
> The log files is very simple
> {code:java}
> jdsaljfldaksjfldsajfldasjflkjdsfldsjfljsdalfk
> jdsaljfldaksjfldsajfldasjflkjdsfldsjfljsdalfk
> jdsaljfldaksjfldsajfldasjflkjdsfldsjfljsdalfk
> jdsaljfldaksjfldsajfldasjflkjdsfldsjfljsdalfk
> jdsaljfldaksjfldsajfldasjflkjdsfldsjfljsdalfk
> jdsaljfldaksjfldsajfldasjflkjdsfldsjfljsdalfk
> ...{code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6970) Issue with LogRegex format plugin where drillbuf was overflowing
[
https://issues.apache.org/jira/browse/DRILL-6970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16793130#comment-16793130
]
Sorabh Hamirwasia commented on DRILL-6970:
--
[~cgivre] - Both of the PR linked to this JIRA are closed. So I am not sure
what we want to merge in master ? Removing ready-to-commit label for now.
> Issue with LogRegex format plugin where drillbuf was overflowing
> -
>
> Key: DRILL-6970
> URL: https://issues.apache.org/jira/browse/DRILL-6970
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.15.0
>Reporter: jean-claude
>Assignee: jean-claude
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.16.0
>
>
> The log format plugin does re-allocate the drillbuf when it fills up. You can
> query small log files but larger ones will fail with this error:
> 0: jdbc:drill:zk=local> select * from dfs.root.`/prog/test.log`;
> Error: INTERNAL_ERROR ERROR: index: 32724, length: 108 (expected: range(0,
> 32768))
> Fragment 0:0
> Please, refer to logs for more information.
>
> I'm running drill-embeded. The log storage plugin is configured like so
> {code:java}
> "log": {
> "type": "logRegex",
> "regex": "(.+)",
> "extension": "log",
> "maxErrors": 10,
> "schema": [
> {
> "fieldName": "line"
> }
> ]
> },
> {code}
> The log files is very simple
> {code:java}
> jdsaljfldaksjfldsajfldasjflkjdsfldsjfljsdalfk
> jdsaljfldaksjfldsajfldasjflkjdsfldsjfljsdalfk
> jdsaljfldaksjfldsajfldasjflkjdsfldsjfljsdalfk
> jdsaljfldaksjfldsajfldasjflkjdsfldsjfljsdalfk
> jdsaljfldaksjfldsajfldasjflkjdsfldsjfljsdalfk
> jdsaljfldaksjfldsajfldasjflkjdsfldsjfljsdalfk
> ...{code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6852) Adapt current Parquet Metadata cache implementation to use Drill Metastore API
[ https://issues.apache.org/jira/browse/DRILL-6852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sorabh Hamirwasia updated DRILL-6852: - Labels: (was: ready-to-commit) > Adapt current Parquet Metadata cache implementation to use Drill Metastore API > -- > > Key: DRILL-6852 > URL: https://issues.apache.org/jira/browse/DRILL-6852 > Project: Apache Drill > Issue Type: Sub-task >Reporter: Volodymyr Vysotskyi >Assignee: Volodymyr Vysotskyi >Priority: Major > Fix For: 1.16.0 > > > According to the design document for DRILL-6552, existing metadata cache API > should be adapted to use generalized API for metastore and parquet metadata > cache will be presented as the implementation of metastore API. > The aim of this Jira is to refactor Parquet Metadata cache implementation and > adapt it to use Drill Metastore API. > Execution plan: > - Refactor AbstractParquetGroupScan and its implementations to use metastore > metadata classes. Store Drill data types in metadata files for Parquet tables. > - Storing the least restrictive type instead of current first file’s column > data type. > - Rework logic in AbstractParquetGroupScan to allow filtering at different > metadata layers: partition, file, row group, etc. The same for pushing the > limit. > - Implement logic to convert existing parquet metadata to metastore metadata > to preserve backward compatibility. > - Implement fetching metadata only when it is needed (for filtering, limit, > count(*) etc.) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-7061) Selecting option to limit results to 1000 on web UI causes parse error
[
https://issues.apache.org/jira/browse/DRILL-7061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sorabh Hamirwasia updated DRILL-7061:
-
Labels: ready-to-commit (was: )
> Selecting option to limit results to 1000 on web UI causes parse error
> --
>
> Key: DRILL-7061
> URL: https://issues.apache.org/jira/browse/DRILL-7061
> Project: Apache Drill
> Issue Type: Bug
> Components: Web Server
>Affects Versions: 1.16.0
>Reporter: Khurram Faraaz
>Assignee: Kunal Khatua
>Priority: Critical
> Labels: ready-to-commit
> Fix For: 1.16.0
>
> Attachments: image-2019-02-27-14-17-24-348.png
>
>
> Selecting option to Limit results to 1,000 causes a parse error on web UI,
> screen shot is attached. Browser used was Chrome.
> Drill version => 1.16.0-SNAPSHOT
> commit = e342ff5
> Error reported on web UI when we press Submit button on web UI
> {noformat}
> Query Failed: An Error Occurred
> org.apache.drill.common.exceptions.UserRemoteException: PARSE ERROR: 'LIMIT
> start, count' is not allowed under the current SQL conformance level SQL
> Query -- [autoLimit: 1,000 rows] select * from ( select length(varStr) from
> dfs.`/root/many_json_files` ) limit 1,000 [Error Id:
> e252d1cc-54d4-4530-837c-a1726a5be89f on qa102-45.qa.lab:31010]{noformat}
> Stack trace from drillbit.log
> {noformat}
> 2019-02-27 21:59:18,428 [2388f7c9-2cb4-0ef8-4088-3ffcab1f0ed2:foreman] INFO
> o.a.drill.exec.work.foreman.Foreman - Query text for query with id
> 2388f7c9-2cb4-0ef8-4088-3ffcab1f0ed2 issued by anonymous: -- [autoLimit:
> 1,000 rows]
> select * from (
> select length(varStr) from dfs.`/root/many_json_files`
> ) limit 1,000
> 2019-02-27 21:59:18,438 [2388f7c9-2cb4-0ef8-4088-3ffcab1f0ed2:foreman] INFO
> o.a.d.exec.planner.sql.SqlConverter - User Error Occurred: 'LIMIT start,
> count' is not allowed under the current SQL conformance level ('LIMIT start,
> count' is not allowed under the current SQL conformance level)
> org.apache.drill.common.exceptions.UserException: PARSE ERROR: 'LIMIT start,
> count' is not allowed under the current SQL conformance level
> SQL Query -- [autoLimit: 1,000 rows]
> select * from (
> select length(varStr) from dfs.`/root/many_json_files`
> ) limit 1,000
> [Error Id: 286b7236-bafd-4ddc-ab10-aaac07e5c088 ]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
> ~[drill-common-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.SqlConverter.parse(SqlConverter.java:193)
> [drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan(DrillSqlWorker.java:138)
> [drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.DrillSqlWorker.convertPlan(DrillSqlWorker.java:110)
> [drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:76)
> [drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:584)
> [drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:272)
> [drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [na:1.8.0_191]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [na:1.8.0_191]
> at java.lang.Thread.run(Thread.java:748) [na:1.8.0_191]
> Caused by: org.apache.calcite.sql.parser.SqlParseException: 'LIMIT start,
> count' is not allowed under the current SQL conformance level
> at
> org.apache.drill.exec.planner.sql.parser.impl.DrillParserImpl.convertException(DrillParserImpl.java:357)
> ~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.parser.impl.DrillParserImpl.normalizeException(DrillParserImpl.java:145)
> ~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at org.apache.calcite.sql.parser.SqlParser.parseQuery(SqlParser.java:156)
> ~[calcite-core-1.18.0-drill-r0.jar:1.18.0-drill-r0]
> at org.apache.calcite.sql.parser.SqlParser.parseStmt(SqlParser.java:181)
> ~[calcite-core-1.18.0-drill-r0.jar:1.18.0-drill-r0]
> at
> org.apache.drill.exec.planner.sql.SqlConverter.parse(SqlConverter.java:185)
> [drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> ... 8 common frames omitted
> Caused by: org.apache.drill.exec.planner.sql.parser.impl.ParseException:
> 'LIMIT start, count' is not allowed under the current SQL conformance level
> at
> org.apache.drill.exec.planner.sql.parser.impl.DrillParserImpl.OrderedQueryOrExpr(DrillParserImpl.java:489)
> ~[drill-java-exec
[jira] [Commented] (DRILL-7061) Selecting option to limit results to 1000 on web UI causes parse error
[
https://issues.apache.org/jira/browse/DRILL-7061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16790799#comment-16790799
]
Sorabh Hamirwasia commented on DRILL-7061:
--
Should we remove the link for DRILL-6960 since this issue is different ?
> Selecting option to limit results to 1000 on web UI causes parse error
> --
>
> Key: DRILL-7061
> URL: https://issues.apache.org/jira/browse/DRILL-7061
> Project: Apache Drill
> Issue Type: Bug
> Components: Web Server
>Affects Versions: 1.16.0
>Reporter: Khurram Faraaz
>Assignee: Kunal Khatua
>Priority: Critical
> Fix For: 1.16.0
>
> Attachments: image-2019-02-27-14-17-24-348.png
>
>
> Selecting option to Limit results to 1,000 causes a parse error on web UI,
> screen shot is attached. Browser used was Chrome.
> Drill version => 1.16.0-SNAPSHOT
> commit = e342ff5
> Error reported on web UI when we press Submit button on web UI
> {noformat}
> Query Failed: An Error Occurred
> org.apache.drill.common.exceptions.UserRemoteException: PARSE ERROR: 'LIMIT
> start, count' is not allowed under the current SQL conformance level SQL
> Query -- [autoLimit: 1,000 rows] select * from ( select length(varStr) from
> dfs.`/root/many_json_files` ) limit 1,000 [Error Id:
> e252d1cc-54d4-4530-837c-a1726a5be89f on qa102-45.qa.lab:31010]{noformat}
> Stack trace from drillbit.log
> {noformat}
> 2019-02-27 21:59:18,428 [2388f7c9-2cb4-0ef8-4088-3ffcab1f0ed2:foreman] INFO
> o.a.drill.exec.work.foreman.Foreman - Query text for query with id
> 2388f7c9-2cb4-0ef8-4088-3ffcab1f0ed2 issued by anonymous: -- [autoLimit:
> 1,000 rows]
> select * from (
> select length(varStr) from dfs.`/root/many_json_files`
> ) limit 1,000
> 2019-02-27 21:59:18,438 [2388f7c9-2cb4-0ef8-4088-3ffcab1f0ed2:foreman] INFO
> o.a.d.exec.planner.sql.SqlConverter - User Error Occurred: 'LIMIT start,
> count' is not allowed under the current SQL conformance level ('LIMIT start,
> count' is not allowed under the current SQL conformance level)
> org.apache.drill.common.exceptions.UserException: PARSE ERROR: 'LIMIT start,
> count' is not allowed under the current SQL conformance level
> SQL Query -- [autoLimit: 1,000 rows]
> select * from (
> select length(varStr) from dfs.`/root/many_json_files`
> ) limit 1,000
> [Error Id: 286b7236-bafd-4ddc-ab10-aaac07e5c088 ]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
> ~[drill-common-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.SqlConverter.parse(SqlConverter.java:193)
> [drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan(DrillSqlWorker.java:138)
> [drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.DrillSqlWorker.convertPlan(DrillSqlWorker.java:110)
> [drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:76)
> [drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:584)
> [drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:272)
> [drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [na:1.8.0_191]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [na:1.8.0_191]
> at java.lang.Thread.run(Thread.java:748) [na:1.8.0_191]
> Caused by: org.apache.calcite.sql.parser.SqlParseException: 'LIMIT start,
> count' is not allowed under the current SQL conformance level
> at
> org.apache.drill.exec.planner.sql.parser.impl.DrillParserImpl.convertException(DrillParserImpl.java:357)
> ~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.parser.impl.DrillParserImpl.normalizeException(DrillParserImpl.java:145)
> ~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at org.apache.calcite.sql.parser.SqlParser.parseQuery(SqlParser.java:156)
> ~[calcite-core-1.18.0-drill-r0.jar:1.18.0-drill-r0]
> at org.apache.calcite.sql.parser.SqlParser.parseStmt(SqlParser.java:181)
> ~[calcite-core-1.18.0-drill-r0.jar:1.18.0-drill-r0]
> at
> org.apache.drill.exec.planner.sql.SqlConverter.parse(SqlConverter.java:185)
> [drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> ... 8 common frames omitted
> Caused by: org.apache.drill.exec.planner.sql.parser.impl.ParseException:
> 'LIMIT start, count' is not allowed under the current SQL conformance level
> at
> org.apache.drill.exec.planner.sql.parser.impl.DrillParserImpl.OrderedQue
[jira] [Updated] (DRILL-6825) Applying different hash function according to data types and data size
[ https://issues.apache.org/jira/browse/DRILL-6825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sorabh Hamirwasia updated DRILL-6825: - Reviewer: Boaz Ben-Zvi > Applying different hash function according to data types and data size > -- > > Key: DRILL-6825 > URL: https://issues.apache.org/jira/browse/DRILL-6825 > Project: Apache Drill > Issue Type: Improvement > Components: Execution - Codegen >Reporter: weijie.tong >Assignee: weijie.tong >Priority: Major > Fix For: 1.16.0 > > > Different hash functions have different performance according to different > data types and data size. We should choose a right one to apply not just > Murmurhash. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6845) Eliminate duplicates for Semi Hash Join
[ https://issues.apache.org/jira/browse/DRILL-6845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sorabh Hamirwasia updated DRILL-6845: - Reviewer: Timothy Farkas > Eliminate duplicates for Semi Hash Join > --- > > Key: DRILL-6845 > URL: https://issues.apache.org/jira/browse/DRILL-6845 > Project: Apache Drill > Issue Type: Improvement > Components: Execution - Relational Operators >Affects Versions: 1.14.0 >Reporter: Boaz Ben-Zvi >Assignee: Boaz Ben-Zvi >Priority: Minor > Fix For: 1.16.0 > > > Following DRILL-6735: The performance of the new Semi Hash Join may degrade > if the build side contains excessive number of join-key duplicate rows; this > mainly a result of the need to store all those rows first, before the hash > table is built. > Proposed solution: For Semi, the Hash Agg would create a Hash-Table > initially, and use it to eliminate key-duplicate rows as they arrive. > Proposed extra: That Hash-Table has an added cost (e.g. resizing). So > perform "runtime stats" – Check initial number of incoming rows (e.g. 32k), > and if the number of duplicates is less than some threshold (e.g. %20) – > cancel that "early" hash table. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-7061) Selecting option to limit results to 1000 on web UI causes parse error
[
https://issues.apache.org/jira/browse/DRILL-7061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sorabh Hamirwasia updated DRILL-7061:
-
Reviewer: Sorabh Hamirwasia
> Selecting option to limit results to 1000 on web UI causes parse error
> --
>
> Key: DRILL-7061
> URL: https://issues.apache.org/jira/browse/DRILL-7061
> Project: Apache Drill
> Issue Type: Bug
> Components: Web Server
>Affects Versions: 1.16.0
>Reporter: Khurram Faraaz
>Assignee: Kunal Khatua
>Priority: Critical
> Fix For: 1.16.0
>
> Attachments: image-2019-02-27-14-17-24-348.png
>
>
> Selecting option to Limit results to 1,000 causes a parse error on web UI,
> screen shot is attached. Browser used was Chrome.
> Drill version => 1.16.0-SNAPSHOT
> commit = e342ff5
> Error reported on web UI when we press Submit button on web UI
> {noformat}
> Query Failed: An Error Occurred
> org.apache.drill.common.exceptions.UserRemoteException: PARSE ERROR: 'LIMIT
> start, count' is not allowed under the current SQL conformance level SQL
> Query -- [autoLimit: 1,000 rows] select * from ( select length(varStr) from
> dfs.`/root/many_json_files` ) limit 1,000 [Error Id:
> e252d1cc-54d4-4530-837c-a1726a5be89f on qa102-45.qa.lab:31010]{noformat}
> Stack trace from drillbit.log
> {noformat}
> 2019-02-27 21:59:18,428 [2388f7c9-2cb4-0ef8-4088-3ffcab1f0ed2:foreman] INFO
> o.a.drill.exec.work.foreman.Foreman - Query text for query with id
> 2388f7c9-2cb4-0ef8-4088-3ffcab1f0ed2 issued by anonymous: -- [autoLimit:
> 1,000 rows]
> select * from (
> select length(varStr) from dfs.`/root/many_json_files`
> ) limit 1,000
> 2019-02-27 21:59:18,438 [2388f7c9-2cb4-0ef8-4088-3ffcab1f0ed2:foreman] INFO
> o.a.d.exec.planner.sql.SqlConverter - User Error Occurred: 'LIMIT start,
> count' is not allowed under the current SQL conformance level ('LIMIT start,
> count' is not allowed under the current SQL conformance level)
> org.apache.drill.common.exceptions.UserException: PARSE ERROR: 'LIMIT start,
> count' is not allowed under the current SQL conformance level
> SQL Query -- [autoLimit: 1,000 rows]
> select * from (
> select length(varStr) from dfs.`/root/many_json_files`
> ) limit 1,000
> [Error Id: 286b7236-bafd-4ddc-ab10-aaac07e5c088 ]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
> ~[drill-common-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.SqlConverter.parse(SqlConverter.java:193)
> [drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan(DrillSqlWorker.java:138)
> [drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.DrillSqlWorker.convertPlan(DrillSqlWorker.java:110)
> [drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:76)
> [drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:584)
> [drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:272)
> [drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [na:1.8.0_191]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [na:1.8.0_191]
> at java.lang.Thread.run(Thread.java:748) [na:1.8.0_191]
> Caused by: org.apache.calcite.sql.parser.SqlParseException: 'LIMIT start,
> count' is not allowed under the current SQL conformance level
> at
> org.apache.drill.exec.planner.sql.parser.impl.DrillParserImpl.convertException(DrillParserImpl.java:357)
> ~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.parser.impl.DrillParserImpl.normalizeException(DrillParserImpl.java:145)
> ~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at org.apache.calcite.sql.parser.SqlParser.parseQuery(SqlParser.java:156)
> ~[calcite-core-1.18.0-drill-r0.jar:1.18.0-drill-r0]
> at org.apache.calcite.sql.parser.SqlParser.parseStmt(SqlParser.java:181)
> ~[calcite-core-1.18.0-drill-r0.jar:1.18.0-drill-r0]
> at
> org.apache.drill.exec.planner.sql.SqlConverter.parse(SqlConverter.java:185)
> [drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> ... 8 common frames omitted
> Caused by: org.apache.drill.exec.planner.sql.parser.impl.ParseException:
> 'LIMIT start, count' is not allowed under the current SQL conformance level
> at
> org.apache.drill.exec.planner.sql.parser.impl.DrillParserImpl.OrderedQueryOrExpr(DrillParserImpl.java:489)
> ~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at
[jira] [Commented] (DRILL-7081) Upgrade GlassFish Jersey and Javax Servlet dependecies
[ https://issues.apache.org/jira/browse/DRILL-7081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16788435#comment-16788435 ] Sorabh Hamirwasia commented on DRILL-7081: -- [~vitalii] - The status of JIRA is in progress ? I have attached the tag of ready-to-commit to it. Do you want to move it to Reviewable ? > Upgrade GlassFish Jersey and Javax Servlet dependecies > -- > > Key: DRILL-7081 > URL: https://issues.apache.org/jira/browse/DRILL-7081 > Project: Apache Drill > Issue Type: Improvement > Components: Web Server >Affects Versions: 1.15.0 >Reporter: Vitalii Diravka >Assignee: Vitalii Diravka >Priority: Minor > Labels: ready-to-commit > Fix For: 1.16.0 > > > DRILL-7051 updates Eclipse Jetty dependencies. > There are more dependencies related to Drill WebUI, which are old and should > be updated: > 1. Set of GlassFish Jersey dependecies > 2. Set of Javax Servlet dependecies > It requires almost 2M increasing of drill-jdbc-all jar. > It is necessary to investigate whether it is possible to exclude some > dependencies to reduce the size of the driver. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-7081) Upgrade GlassFish Jersey and Javax Servlet dependecies
[ https://issues.apache.org/jira/browse/DRILL-7081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sorabh Hamirwasia updated DRILL-7081: - Labels: ready-to-commit (was: ) > Upgrade GlassFish Jersey and Javax Servlet dependecies > -- > > Key: DRILL-7081 > URL: https://issues.apache.org/jira/browse/DRILL-7081 > Project: Apache Drill > Issue Type: Improvement > Components: Web Server >Affects Versions: 1.15.0 >Reporter: Vitalii Diravka >Assignee: Vitalii Diravka >Priority: Minor > Labels: ready-to-commit > Fix For: 1.16.0 > > > DRILL-7051 updates Eclipse Jetty dependencies. > There are more dependencies related to Drill WebUI, which are old and should > be updated: > 1. Set of GlassFish Jersey dependecies > 2. Set of Javax Servlet dependecies > It requires almost 2M increasing of drill-jdbc-all jar. > It is necessary to investigate whether it is possible to exclude some > dependencies to reduce the size of the driver. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-7046) Support for loading and parsing new RM config file
[ https://issues.apache.org/jira/browse/DRILL-7046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sorabh Hamirwasia updated DRILL-7046: - Labels: ready-to-commit (was: ) > Support for loading and parsing new RM config file > -- > > Key: DRILL-7046 > URL: https://issues.apache.org/jira/browse/DRILL-7046 > Project: Apache Drill > Issue Type: Sub-task > Components: Execution - Flow >Affects Versions: 1.16.0 >Reporter: Sorabh Hamirwasia >Assignee: Sorabh Hamirwasia >Priority: Major > Labels: ready-to-commit > > This Jira will help to add support for loading new RM specific configuration > file if needed and also parsing it to create all the required in-memory > objects. The details of the new configuration is defined in [Function > Spec|https://docs.google.com/document/d/1cX0lPLL-QzBGUwcekAPvUBgubdcNlWRdy7IG_IFt-Ok/edit?usp=sharing] > RM configuration file will also support override, default and distrib > specific files. It will only be loaded when RM is enabled which will be > controlled by *drill.exec.rm.enabled* parameter in Drill main configuration > file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-7060) Support JsonParser Feature 'ALLOW_BACKSLASH_ESCAPING_ANY_CHARACTER' in JsonReader
[
https://issues.apache.org/jira/browse/DRILL-7060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sorabh Hamirwasia updated DRILL-7060:
-
Labels: ready-to-commit (was: )
> Support JsonParser Feature 'ALLOW_BACKSLASH_ESCAPING_ANY_CHARACTER' in
> JsonReader
> -
>
> Key: DRILL-7060
> URL: https://issues.apache.org/jira/browse/DRILL-7060
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - JSON
>Affects Versions: 1.15.0, 1.16.0
>Reporter: Abhishek Girish
>Assignee: Abhishek Girish
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.16.0
>
>
> Some JSON files may have strings with backslashes - which are read as escape
> characters. By default only standard escape characters are allowed. So
> querying such files would fail. For example see:
> Data
> {code}
> {"file":"C:\Sfiles\escape.json"}
> {code}
> Error
> {code}
> (com.fasterxml.jackson.core.JsonParseException) Unrecognized character escape
> 'S' (code 83)
> at [Source: (org.apache.drill.exec.store.dfs.DrillFSDataInputStream); line:
> 1, column: 178]
> com.fasterxml.jackson.core.JsonParser._constructError():1804
> com.fasterxml.jackson.core.base.ParserMinimalBase._reportError():663
>
> com.fasterxml.jackson.core.base.ParserMinimalBase._handleUnrecognizedCharacterEscape():640
> com.fasterxml.jackson.core.json.UTF8StreamJsonParser._decodeEscaped():3243
> com.fasterxml.jackson.core.json.UTF8StreamJsonParser._skipString():2537
> com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken():683
> org.apache.drill.exec.vector.complex.fn.JsonReader.writeData():342
> org.apache.drill.exec.vector.complex.fn.JsonReader.writeDataSwitch():298
> org.apache.drill.exec.vector.complex.fn.JsonReader.writeToVector():246
> org.apache.drill.exec.vector.complex.fn.JsonReader.write():205
> org.apache.drill.exec.store.easy.json.JSONRecordReader.next():216
> org.apache.drill.exec.physical.impl.ScanBatch.internalNext():223
> ...
> ...
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-7035) Drill C++ Client crashes on multiple SaslAuthenticatorImpl Destruction due to communication error
[
https://issues.apache.org/jira/browse/DRILL-7035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16782881#comment-16782881
]
Sorabh Hamirwasia commented on DRILL-7035:
--
[~debraj92] - Can you please move JIRA to Reviewable state otherwise it will be
missed by batch committer to merge it in.
> Drill C++ Client crashes on multiple SaslAuthenticatorImpl Destruction due to
> communication error
> --
>
> Key: DRILL-7035
> URL: https://issues.apache.org/jira/browse/DRILL-7035
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - C++
>Affects Versions: 1.12.0
>Reporter: Rob Wu
>Assignee: Debraj Ray
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.16.0
>
>
> [~debraj92] found that when under some circumstance the
> SaslAuthenticatorImpl's sasl_dispose() function will crash out at
> destruction. The incident seems to be random and only when certain
> authentication and encryption combinations are used during connection.
> After digging a little deeper, I found that when BOOST communication error
> occurs, the shutdownSocket (eventually triggering sasl_dispose()) could be
> called from various threads resulting in a race condition of freeing the
> handle. This can be reproduced with the querysubmitter. This is reproducible
> since 1.12.0+.
> [~debraj92] will be adding a patch to resolve this incident.
>
> {code:java}
> 2019-Feb-11 10:44:01 : TRACE : 2d74 : DrillClientImpl::handleRead: Handle
> Read from buffer 04E1D850
> 2019-Feb-11 10:44:01 : TRACE : 2d74 : DrillClientImpl::handleRead: Cancel
> deadline timer.
> 2019-Feb-11 10:44:01 : TRACE : 2d74 : DrillClientImpl::handleRead:
> ERR_QRY_COMMERR. Boost Communication Error: End of file
> 2019-Feb-11 10:44:31 : TRACE : 3df8 : Disposing 1: +++ ENTER +++
> 2019-Feb-11 10:44:31 : TRACE : 2d74 : Disposing 2: +++ ENTER +++
> 2019-Feb-11 10:44:31 : TRACE : 2d74 : Disposing 2: --- EXIT ---
> 2019-Feb-11 10:44:31 : TRACE : 2d74 : Socket shutdown
> 2019-Feb-11 10:44:31 : TRACE : 3df8 : Disposing 1: --- EXIT ---
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (DRILL-6991) Kerberos ticket is being dumped in the log if log level is "debug" for stdout
[
https://issues.apache.org/jira/browse/DRILL-6991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sorabh Hamirwasia resolved DRILL-6991.
--
Resolution: Not A Problem
> Kerberos ticket is being dumped in the log if log level is "debug" for stdout
> --
>
> Key: DRILL-6991
> URL: https://issues.apache.org/jira/browse/DRILL-6991
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.15.0
>Reporter: Anton Gozhiy
>Assignee: Sorabh Hamirwasia
>Priority: Major
> Fix For: 1.16.0
>
>
> *Prerequisites:*
> # Drill is installed on cluster with Kerberos security
> # Into conf/logback.xml, set the following log level:
> {code:xml}
>
>
>
>
> {code}
> *Steps:*
> # Start Drill
> # Connect using sqlline using the following string:
> {noformat}
> bin/sqlline -u "jdbc:drill:zk=;principal="
> {noformat}
> *Expected result:*
> No sensitive information should be displayed
> *Actual result:*
> Kerberos ticket and session key are being dumped into console output:
> {noformat}
> 14:35:38.806 [TGT Renewer for mapr/node1.cluster.com@NODE1] DEBUG
> o.a.h.security.UserGroupInformation - Found tgt Ticket (hex) =
> : 61 82 01 3D 30 82 01 39 A0 03 02 01 05 A1 07 1B a..=0..9
> 0010: 05 4E 4F 44 45 31 A2 1A 30 18 A0 03 02 01 02 A1 .NODE1..0...
> 0020: 11 30 0F 1B 06 6B 72 62 74 67 74 1B 05 4E 4F 44 .0...krbtgt..NOD
> 0030: 45 31 A3 82 01 0B 30 82 01 07 A0 03 02 01 12 A1 E10.
> 0040: 03 02 01 01 A2 81 FA 04 81 F7 03 8D A9 FA 7D 89
> 0050: 1B DF 37 B7 4D E6 6C 99 3E 8F FA 48 D9 9A 79 F3 ..7.M.l.>..H..y.
> 0060: 92 34 7F BF 67 1E 77 4A 2F C9 AF 82 93 4E 46 1D .4..g.wJ/NF.
> 0070: 41 74 B0 AF 41 A8 8B 02 71 83 CC 14 51 72 60 EE At..A...q...Qr`.
> 0080: 29 67 14 F0 A6 33 63 07 41 AA 8D DC 7B 5B 41 F3 )g...3c.A[A.
> 0090: 83 48 8B 2A 0B 4D 6D 57 9A 6E CF 6B DC 0B C0 D1 .H.*.MmW.n.k
> 00A0: 83 BB 27 40 88 7E 9F 2B D1 FD A8 6A E1 BF F6 CC ..'@...+...j
> 00B0: 0E 0C FB 93 5D 69 9A 8B 11 88 0C F2 7C E1 FD 04 ]i..
> 00C0: F5 AB 66 0C A4 A4 7B 30 D1 7F F1 2D D6 A1 52 D1 ..f0...-..R.
> 00D0: 79 59 F2 06 CB 65 FB 73 63 1D 5B E9 4F 28 73 EB yY...e.sc.[.O(s.
> 00E0: 72 7F 04 46 34 56 F4 40 6C C0 2C 39 C0 5B C6 25 r..F4V.@l.,9.[.%
> 00F0: ED EF 64 07 CE ED 35 9D D7 91 6C 8F C9 CE 16 F5 ..d...5...l.
> 0100: CA 5E 6F DE 08 D2 68 30 C7 03 97 E7 C0 FF D9 52 .^o...h0...R
> 0110: F8 1D 2F DB 63 6D 12 4A CD 60 AD D0 BA FA 4B CF ../.cm.J.`K.
> 0120: 2C B9 8C CA 5A E6 EC 10 5A 0A 1F 84 B0 80 BD 39 ,...Z...Z..9
> 0130: 42 2C 33 EB C0 AA 0D 44 F0 F4 E9 87 24 43 BB 9A B,3D$C..
> 0140: 52 R
> Client Principal = mapr/node1.cluster.com@NODE1
> Server Principal = krbtgt/NODE1@NODE1
> Session Key = EncryptionKey: keyType=18 keyBytes (hex dump)=
> : 50 DA D1 D7 91 D3 64 BE 45 7B D8 02 25 81 18 25 P.d.E...%..%
> 0010: DA 59 4F BA 76 67 BB 39 9C F7 17 46 A7 C5 00 E2 .YO.vg.9...F
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6991) Kerberos ticket is being dumped in the log if log level is "debug" for stdout
[
https://issues.apache.org/jira/browse/DRILL-6991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16781936#comment-16781936
]
Sorabh Hamirwasia commented on DRILL-6991:
--
[~angozhiy] - I am closing the JIRA for now since I don't see any threat here.
Please re-open if you think otherwise.
> Kerberos ticket is being dumped in the log if log level is "debug" for stdout
> --
>
> Key: DRILL-6991
> URL: https://issues.apache.org/jira/browse/DRILL-6991
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.15.0
>Reporter: Anton Gozhiy
>Assignee: Sorabh Hamirwasia
>Priority: Major
> Fix For: 1.16.0
>
>
> *Prerequisites:*
> # Drill is installed on cluster with Kerberos security
> # Into conf/logback.xml, set the following log level:
> {code:xml}
>
>
>
>
> {code}
> *Steps:*
> # Start Drill
> # Connect using sqlline using the following string:
> {noformat}
> bin/sqlline -u "jdbc:drill:zk=;principal="
> {noformat}
> *Expected result:*
> No sensitive information should be displayed
> *Actual result:*
> Kerberos ticket and session key are being dumped into console output:
> {noformat}
> 14:35:38.806 [TGT Renewer for mapr/node1.cluster.com@NODE1] DEBUG
> o.a.h.security.UserGroupInformation - Found tgt Ticket (hex) =
> : 61 82 01 3D 30 82 01 39 A0 03 02 01 05 A1 07 1B a..=0..9
> 0010: 05 4E 4F 44 45 31 A2 1A 30 18 A0 03 02 01 02 A1 .NODE1..0...
> 0020: 11 30 0F 1B 06 6B 72 62 74 67 74 1B 05 4E 4F 44 .0...krbtgt..NOD
> 0030: 45 31 A3 82 01 0B 30 82 01 07 A0 03 02 01 12 A1 E10.
> 0040: 03 02 01 01 A2 81 FA 04 81 F7 03 8D A9 FA 7D 89
> 0050: 1B DF 37 B7 4D E6 6C 99 3E 8F FA 48 D9 9A 79 F3 ..7.M.l.>..H..y.
> 0060: 92 34 7F BF 67 1E 77 4A 2F C9 AF 82 93 4E 46 1D .4..g.wJ/NF.
> 0070: 41 74 B0 AF 41 A8 8B 02 71 83 CC 14 51 72 60 EE At..A...q...Qr`.
> 0080: 29 67 14 F0 A6 33 63 07 41 AA 8D DC 7B 5B 41 F3 )g...3c.A[A.
> 0090: 83 48 8B 2A 0B 4D 6D 57 9A 6E CF 6B DC 0B C0 D1 .H.*.MmW.n.k
> 00A0: 83 BB 27 40 88 7E 9F 2B D1 FD A8 6A E1 BF F6 CC ..'@...+...j
> 00B0: 0E 0C FB 93 5D 69 9A 8B 11 88 0C F2 7C E1 FD 04 ]i..
> 00C0: F5 AB 66 0C A4 A4 7B 30 D1 7F F1 2D D6 A1 52 D1 ..f0...-..R.
> 00D0: 79 59 F2 06 CB 65 FB 73 63 1D 5B E9 4F 28 73 EB yY...e.sc.[.O(s.
> 00E0: 72 7F 04 46 34 56 F4 40 6C C0 2C 39 C0 5B C6 25 r..F4V.@l.,9.[.%
> 00F0: ED EF 64 07 CE ED 35 9D D7 91 6C 8F C9 CE 16 F5 ..d...5...l.
> 0100: CA 5E 6F DE 08 D2 68 30 C7 03 97 E7 C0 FF D9 52 .^o...h0...R
> 0110: F8 1D 2F DB 63 6D 12 4A CD 60 AD D0 BA FA 4B CF ../.cm.J.`K.
> 0120: 2C B9 8C CA 5A E6 EC 10 5A 0A 1F 84 B0 80 BD 39 ,...Z...Z..9
> 0130: 42 2C 33 EB C0 AA 0D 44 F0 F4 E9 87 24 43 BB 9A B,3D$C..
> 0140: 52 R
> Client Principal = mapr/node1.cluster.com@NODE1
> Server Principal = krbtgt/NODE1@NODE1
> Session Key = EncryptionKey: keyType=18 keyBytes (hex dump)=
> : 50 DA D1 D7 91 D3 64 BE 45 7B D8 02 25 81 18 25 P.d.E...%..%
> 0010: DA 59 4F BA 76 67 BB 39 9C F7 17 46 A7 C5 00 E2 .YO.vg.9...F
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6979) Add autofocus attribute to username on login page, and to query textbox on Query tab.
[ https://issues.apache.org/jira/browse/DRILL-6979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sorabh Hamirwasia updated DRILL-6979: - Labels: ready-to-commit user-experience (was: user-experience) > Add autofocus attribute to username on login page, and to query textbox on > Query tab. > - > > Key: DRILL-6979 > URL: https://issues.apache.org/jira/browse/DRILL-6979 > Project: Apache Drill > Issue Type: Improvement > Components: Web Server >Affects Versions: 1.16.0 >Reporter: Khurram Faraaz >Assignee: Khurram Faraaz >Priority: Minor > Labels: ready-to-commit, user-experience > > Add autofocus attribute to username on login page, and to query textbox on > Query tab. > The two text boxes that need the change are in these files > ./exec/java-exec/src/main/resources/rest/query/query.ftl > ./exec/java-exec/src/main/resources/rest/login.ftl -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6979) Add autofocus attribute to username on login page, and to query textbox on Query tab.
[ https://issues.apache.org/jira/browse/DRILL-6979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sorabh Hamirwasia updated DRILL-6979: - Reviewer: Sorabh Hamirwasia > Add autofocus attribute to username on login page, and to query textbox on > Query tab. > - > > Key: DRILL-6979 > URL: https://issues.apache.org/jira/browse/DRILL-6979 > Project: Apache Drill > Issue Type: Improvement > Components: Web Server >Affects Versions: 1.16.0 >Reporter: Khurram Faraaz >Assignee: Khurram Faraaz >Priority: Minor > Labels: ready-to-commit, user-experience > > Add autofocus attribute to username on login page, and to query textbox on > Query tab. > The two text boxes that need the change are in these files > ./exec/java-exec/src/main/resources/rest/query/query.ftl > ./exec/java-exec/src/main/resources/rest/login.ftl -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-7035) Drill C++ Client crashes on multiple SaslAuthenticatorImpl Destruction due to communication error
[
https://issues.apache.org/jira/browse/DRILL-7035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sorabh Hamirwasia updated DRILL-7035:
-
Reviewer: Sorabh Hamirwasia
> Drill C++ Client crashes on multiple SaslAuthenticatorImpl Destruction due to
> communication error
> --
>
> Key: DRILL-7035
> URL: https://issues.apache.org/jira/browse/DRILL-7035
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - C++
>Affects Versions: 1.12.0
>Reporter: Rob Wu
>Assignee: Debraj Ray
>Priority: Major
> Fix For: 1.16.0
>
>
> [~debraj92] found that when under some circumstance the
> SaslAuthenticatorImpl's sasl_dispose() function will crash out at
> destruction. The incident seems to be random and only when certain
> authentication and encryption combinations are used during connection.
> After digging a little deeper, I found that when BOOST communication error
> occurs, the shutdownSocket (eventually triggering sasl_dispose()) could be
> called from various threads resulting in a race condition of freeing the
> handle. This can be reproduced with the querysubmitter. This is reproducible
> since 1.12.0+.
> [~debraj92] will be adding a patch to resolve this incident.
>
> {code:java}
> 2019-Feb-11 10:44:01 : TRACE : 2d74 : DrillClientImpl::handleRead: Handle
> Read from buffer 04E1D850
> 2019-Feb-11 10:44:01 : TRACE : 2d74 : DrillClientImpl::handleRead: Cancel
> deadline timer.
> 2019-Feb-11 10:44:01 : TRACE : 2d74 : DrillClientImpl::handleRead:
> ERR_QRY_COMMERR. Boost Communication Error: End of file
> 2019-Feb-11 10:44:31 : TRACE : 3df8 : Disposing 1: +++ ENTER +++
> 2019-Feb-11 10:44:31 : TRACE : 2d74 : Disposing 2: +++ ENTER +++
> 2019-Feb-11 10:44:31 : TRACE : 2d74 : Disposing 2: --- EXIT ---
> 2019-Feb-11 10:44:31 : TRACE : 2d74 : Socket shutdown
> 2019-Feb-11 10:44:31 : TRACE : 3df8 : Disposing 1: --- EXIT ---
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-7047) Drill C++ Client crash due to Dangling stack ptr to sasl_callback_t
[
https://issues.apache.org/jira/browse/DRILL-7047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sorabh Hamirwasia updated DRILL-7047:
-
Labels: ready-to-commit (was: )
> Drill C++ Client crash due to Dangling stack ptr to sasl_callback_t
>
>
> Key: DRILL-7047
> URL: https://issues.apache.org/jira/browse/DRILL-7047
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - C++
>Affects Versions: 1.14.0
>Reporter: Rob Wu
>Assignee: Debraj Ray
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.16.0
>
>
> The sasl_client_new does not copy its callback argument array, resulting in a
> pointer to transient stack memory.
>
> [~debraj92] will be supplying a patch to resolve this issue. This patch moves
> the callbacks array into the member variable m_callbacks which has the same
> lifetime as the sasl impl instance and thus will remain valid until the end
> of life.
>
> Trace:
> {code:java}
> #0 0x0080 in ?? ()
> #1 0xb38c04bc in _sasl_canon_user ()
> from libdrillClient.so
> #2 0xb38c0611 in _sasl_canon_user_lookup ()
> from libdrillClient.so
> #3 0xb2c0824e in gssapi_client_mech_step () from /usr/lib/sasl2/libgssapiv2.so
> #4 0xb38ad244 in sasl_client_step ()
> from libdrillClient.so
> #5 0xb37fddde in Drill::SaslAuthenticatorImpl::step(exec::shared::SaslMessage
> const&, exec::shared::SaslMessage&) const ()
> from libdrillClient.so
> #6 0xb37bdf16 in
> Drill::DrillClientImpl::processSaslChallenge(Drill::AllocatedBuffer*,
> Drill::rpc::InBoundRpcMessage const&) ()
> from libdrillClient.so
> #7 0xb37bfa17 in Drill::DrillClientImpl::handleRead(unsigned char*,
> boost_sb::system::error_code const&, unsigned int) ()
> from libdrillClient.so
> #8 0xb37c0955 in
> boost_sb::detail::function::void_function_obj_invoker2 boost_sb::_mfi::mf3 boost_sb::system::error_code const&, unsigned int>,
> boost_sb::_bi::list4,
> boost_sb::_bi::value, boost_sb::arg<1> (*)(),
> boost_sb::arg<2> (*)()> >, void, boost_sb::system::error_code const&,
> unsigned int>::invoke(boost_sb::detail::function::function_buffer&,
> boost_sb::system::error_code const&, unsigned int) ()
> from libdrillClient.so
> #9 0xb378f17d in boost_sb::function2 const&, unsigned int>::operator()(boost_sb::system::error_code const&,
> unsigned int) const
> () from libdrillClient.so
> #10 0xb3799bc8 in boost_sb::asio::detail::read_op boost_sb::asio::mutable_buffers_1, boost_sb::asio::mutable_buffer const*,
> boost_sb::asio::detail::transfer_all_t, boost_sb::function (boost_sb::system::error_code const&, unsigned int)>
> >::operator()(boost_sb::system::error_code const&, unsigned int, int) ()
> from libdrillClient.so
> #11 0xb379a1c3 in
> boost_sb::asio::detail::reactive_socket_recv_op boost_sb::asio::detail::read_op boost_sb::asio::mutable_buffers_1, boost_sb::asio::mutable_buffer const*,
> boost_sb::asio::detail::transfer_all_t, boost_sb::function (boost_sb::system::error_code const&, unsigned int)> > >::do_complete(void*,
> boost_sb::asio::detail::scheduler_operation*, boost_sb::system::error_code
> const&, unsigned int) ()
> from libdrillClient.so
> #12 0xb3788fb8 in
> boost_sb::asio::detail::epoll_reactor::descriptor_state::do_complete(void*,
> boost_sb::asio::detail::scheduler_operation*, boost_sb::system::error_code
> const&, unsigned int) ()
> from libdrillClient.so
> #13 0xb3791948 in boost_sb::asio::io_context::run() ()
> from libdrillClient.so
> #14 0xb37c0e67 in
> boost_sb::detail::thread_data boost_sb::_mfi::mf0,
> boost_sb::_bi::list1 > >
> >::run() ()
> from libdrillClient.so
> #15 0xb3825f5a in thread_proxy ()
> from libdrillClient.so
> #16 0xb6730b3c in start_thread () from /lib/libpthread.so.0
> #17 0xb64db44e in clone () from /lib/libc.so.6
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-7035) Drill C++ Client crashes on multiple SaslAuthenticatorImpl Destruction due to communication error
[
https://issues.apache.org/jira/browse/DRILL-7035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sorabh Hamirwasia updated DRILL-7035:
-
Labels: ready-to-commit (was: )
> Drill C++ Client crashes on multiple SaslAuthenticatorImpl Destruction due to
> communication error
> --
>
> Key: DRILL-7035
> URL: https://issues.apache.org/jira/browse/DRILL-7035
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - C++
>Affects Versions: 1.12.0
>Reporter: Rob Wu
>Assignee: Debraj Ray
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.16.0
>
>
> [~debraj92] found that when under some circumstance the
> SaslAuthenticatorImpl's sasl_dispose() function will crash out at
> destruction. The incident seems to be random and only when certain
> authentication and encryption combinations are used during connection.
> After digging a little deeper, I found that when BOOST communication error
> occurs, the shutdownSocket (eventually triggering sasl_dispose()) could be
> called from various threads resulting in a race condition of freeing the
> handle. This can be reproduced with the querysubmitter. This is reproducible
> since 1.12.0+.
> [~debraj92] will be adding a patch to resolve this incident.
>
> {code:java}
> 2019-Feb-11 10:44:01 : TRACE : 2d74 : DrillClientImpl::handleRead: Handle
> Read from buffer 04E1D850
> 2019-Feb-11 10:44:01 : TRACE : 2d74 : DrillClientImpl::handleRead: Cancel
> deadline timer.
> 2019-Feb-11 10:44:01 : TRACE : 2d74 : DrillClientImpl::handleRead:
> ERR_QRY_COMMERR. Boost Communication Error: End of file
> 2019-Feb-11 10:44:31 : TRACE : 3df8 : Disposing 1: +++ ENTER +++
> 2019-Feb-11 10:44:31 : TRACE : 2d74 : Disposing 2: +++ ENTER +++
> 2019-Feb-11 10:44:31 : TRACE : 2d74 : Disposing 2: --- EXIT ---
> 2019-Feb-11 10:44:31 : TRACE : 2d74 : Socket shutdown
> 2019-Feb-11 10:44:31 : TRACE : 3df8 : Disposing 1: --- EXIT ---
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-7035) Drill C++ Client crashes on multiple SaslAuthenticatorImpl Destruction due to communication error
[
https://issues.apache.org/jira/browse/DRILL-7035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16778769#comment-16778769
]
Sorabh Hamirwasia commented on DRILL-7035:
--
Please move the JIRA to reviewable state
> Drill C++ Client crashes on multiple SaslAuthenticatorImpl Destruction due to
> communication error
> --
>
> Key: DRILL-7035
> URL: https://issues.apache.org/jira/browse/DRILL-7035
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - C++
>Affects Versions: 1.12.0
>Reporter: Rob Wu
>Assignee: Debraj Ray
>Priority: Major
> Fix For: 1.16.0
>
>
> [~debraj92] found that when under some circumstance the
> SaslAuthenticatorImpl's sasl_dispose() function will crash out at
> destruction. The incident seems to be random and only when certain
> authentication and encryption combinations are used during connection.
> After digging a little deeper, I found that when BOOST communication error
> occurs, the shutdownSocket (eventually triggering sasl_dispose()) could be
> called from various threads resulting in a race condition of freeing the
> handle. This can be reproduced with the querysubmitter. This is reproducible
> since 1.12.0+.
> [~debraj92] will be adding a patch to resolve this incident.
>
> {code:java}
> 2019-Feb-11 10:44:01 : TRACE : 2d74 : DrillClientImpl::handleRead: Handle
> Read from buffer 04E1D850
> 2019-Feb-11 10:44:01 : TRACE : 2d74 : DrillClientImpl::handleRead: Cancel
> deadline timer.
> 2019-Feb-11 10:44:01 : TRACE : 2d74 : DrillClientImpl::handleRead:
> ERR_QRY_COMMERR. Boost Communication Error: End of file
> 2019-Feb-11 10:44:31 : TRACE : 3df8 : Disposing 1: +++ ENTER +++
> 2019-Feb-11 10:44:31 : TRACE : 2d74 : Disposing 2: +++ ENTER +++
> 2019-Feb-11 10:44:31 : TRACE : 2d74 : Disposing 2: --- EXIT ---
> 2019-Feb-11 10:44:31 : TRACE : 2d74 : Socket shutdown
> 2019-Feb-11 10:44:31 : TRACE : 3df8 : Disposing 1: --- EXIT ---
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-7047) Drill C++ Client crash due to Dangling stack ptr to sasl_callback_t
[
https://issues.apache.org/jira/browse/DRILL-7047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sorabh Hamirwasia updated DRILL-7047:
-
Reviewer: Sorabh Hamirwasia
> Drill C++ Client crash due to Dangling stack ptr to sasl_callback_t
>
>
> Key: DRILL-7047
> URL: https://issues.apache.org/jira/browse/DRILL-7047
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - C++
>Affects Versions: 1.14.0
>Reporter: Rob Wu
>Assignee: Debraj Ray
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.16.0
>
>
> The sasl_client_new does not copy its callback argument array, resulting in a
> pointer to transient stack memory.
>
> [~debraj92] will be supplying a patch to resolve this issue. This patch moves
> the callbacks array into the member variable m_callbacks which has the same
> lifetime as the sasl impl instance and thus will remain valid until the end
> of life.
>
> Trace:
> {code:java}
> #0 0x0080 in ?? ()
> #1 0xb38c04bc in _sasl_canon_user ()
> from libdrillClient.so
> #2 0xb38c0611 in _sasl_canon_user_lookup ()
> from libdrillClient.so
> #3 0xb2c0824e in gssapi_client_mech_step () from /usr/lib/sasl2/libgssapiv2.so
> #4 0xb38ad244 in sasl_client_step ()
> from libdrillClient.so
> #5 0xb37fddde in Drill::SaslAuthenticatorImpl::step(exec::shared::SaslMessage
> const&, exec::shared::SaslMessage&) const ()
> from libdrillClient.so
> #6 0xb37bdf16 in
> Drill::DrillClientImpl::processSaslChallenge(Drill::AllocatedBuffer*,
> Drill::rpc::InBoundRpcMessage const&) ()
> from libdrillClient.so
> #7 0xb37bfa17 in Drill::DrillClientImpl::handleRead(unsigned char*,
> boost_sb::system::error_code const&, unsigned int) ()
> from libdrillClient.so
> #8 0xb37c0955 in
> boost_sb::detail::function::void_function_obj_invoker2 boost_sb::_mfi::mf3 boost_sb::system::error_code const&, unsigned int>,
> boost_sb::_bi::list4,
> boost_sb::_bi::value, boost_sb::arg<1> (*)(),
> boost_sb::arg<2> (*)()> >, void, boost_sb::system::error_code const&,
> unsigned int>::invoke(boost_sb::detail::function::function_buffer&,
> boost_sb::system::error_code const&, unsigned int) ()
> from libdrillClient.so
> #9 0xb378f17d in boost_sb::function2 const&, unsigned int>::operator()(boost_sb::system::error_code const&,
> unsigned int) const
> () from libdrillClient.so
> #10 0xb3799bc8 in boost_sb::asio::detail::read_op boost_sb::asio::mutable_buffers_1, boost_sb::asio::mutable_buffer const*,
> boost_sb::asio::detail::transfer_all_t, boost_sb::function (boost_sb::system::error_code const&, unsigned int)>
> >::operator()(boost_sb::system::error_code const&, unsigned int, int) ()
> from libdrillClient.so
> #11 0xb379a1c3 in
> boost_sb::asio::detail::reactive_socket_recv_op boost_sb::asio::detail::read_op boost_sb::asio::mutable_buffers_1, boost_sb::asio::mutable_buffer const*,
> boost_sb::asio::detail::transfer_all_t, boost_sb::function (boost_sb::system::error_code const&, unsigned int)> > >::do_complete(void*,
> boost_sb::asio::detail::scheduler_operation*, boost_sb::system::error_code
> const&, unsigned int) ()
> from libdrillClient.so
> #12 0xb3788fb8 in
> boost_sb::asio::detail::epoll_reactor::descriptor_state::do_complete(void*,
> boost_sb::asio::detail::scheduler_operation*, boost_sb::system::error_code
> const&, unsigned int) ()
> from libdrillClient.so
> #13 0xb3791948 in boost_sb::asio::io_context::run() ()
> from libdrillClient.so
> #14 0xb37c0e67 in
> boost_sb::detail::thread_data boost_sb::_mfi::mf0,
> boost_sb::_bi::list1 > >
> >::run() ()
> from libdrillClient.so
> #15 0xb3825f5a in thread_proxy ()
> from libdrillClient.so
> #16 0xb6730b3c in start_thread () from /lib/libpthread.so.0
> #17 0xb64db44e in clone () from /lib/libc.so.6
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6758) Hash Join should not return the join columns when they are not needed downstream
[ https://issues.apache.org/jira/browse/DRILL-6758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sorabh Hamirwasia updated DRILL-6758: - Fix Version/s: (was: 1.16.0) 1.17.0 > Hash Join should not return the join columns when they are not needed > downstream > > > Key: DRILL-6758 > URL: https://issues.apache.org/jira/browse/DRILL-6758 > Project: Apache Drill > Issue Type: Improvement > Components: Execution - Relational Operators, Query Planning & > Optimization >Affects Versions: 1.14.0 >Reporter: Boaz Ben-Zvi >Assignee: Hanumath Rao Maduri >Priority: Minor > Fix For: 1.17.0 > > > Currently the Hash-Join operator returns all its (both sides) incoming > columns. In cases where the join columns are not used further downstream, > this is a waste (allocating vectors, copying each value, etc). > Suggestion: Have the planner pass this information to the Hash-Join > operator, to enable skipping the return of these columns. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (DRILL-6871) Enabling runtime filter eliminates more incoming rows than it should.
[
https://issues.apache.org/jira/browse/DRILL-6871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sorabh Hamirwasia resolved DRILL-6871.
--
Resolution: Fixed
> Enabling runtime filter eliminates more incoming rows than it should.
> -
>
> Key: DRILL-6871
> URL: https://issues.apache.org/jira/browse/DRILL-6871
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.15.0
>Reporter: Kunal Khatua
>Assignee: weijie.tong
>Priority: Major
> Fix For: 1.16.0
>
> Attachments: 24036b35-7c41-12c9-8c08-9327c8d8fdf2.sys.drill,
> 24036bbf-ede2-690a-93b7-9b43c28eed52.sys.drill,
> 24036c15-54a9-7b5d-07be-4d6de4f63731.sys.drill,
> 24036c22-7f39-ae8b-f8f4-000ecc2572cc.sys.drill
>
>
> When testing with the following combination on TPC-H dataset (scale factor
> 100) using a 4 node setup...
> {code:java}
> exec.hashjoin.bloom_filter.fpp=0.2
> exec.hashjoin.enable.runtime_filter=true
> exec.hashjoin.runtime_filter.max.waiting.time=2
> {code}
> It was observed that the filter eliminates more rows than it should.
>
> {code:java}
> 0: jdbc:drill:schema=dfs.par100> select count(*) from (select * from lineitem
> l, supplier s where l.l_suppkey = s.s_suppkey and s.s_acctbal <1000);
> +-+
> | EXPR$0 |
> +-+
> | 405566 |
> +-+
> 1 row selected (10.565 seconds)
> 0: jdbc:drill:schema=dfs.par100> select count(*) from (select * from lineitem
> l, supplier s where l.l_suppkey = s.s_suppkey and s.s_acctbal <1000);
> +-+
> | EXPR$0 |
> +-+
> | 405769 |
> +-+
> 1 row selected (9.845 seconds)
> {code}
> The expected row count for the above (broadcast-join) query should have been
> *109307880*
>
> {code:java}
> 0: jdbc:drill:schema=dfs.par100> select count(*) from (select * from lineitem
> l, orders o where o.o_orderkey = l.l_orderkey and o.o_totalprice < 10);
> +---+
> | EXPR$0 |
> +---+
> | 37338355 |
> +---+
> 1 row selected (44.698 seconds)
> 0: jdbc:drill:schema=dfs.par100> select count(*) from (select * from lineitem
> l, orders o where o.o_orderkey = l.l_orderkey and o.o_totalprice < 10);
> +---+
> | EXPR$0 |
> +---+
> | 38044874 |
> +---+
> 1 row selected (44.871 seconds)
> {code}
> The expected row count for the above (hash partition-join) query should have
> been *96176495*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6871) Enabling runtime filter eliminates more incoming rows than it should.
[
https://issues.apache.org/jira/browse/DRILL-6871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16778608#comment-16778608
]
Sorabh Hamirwasia commented on DRILL-6871:
--
[~aravi5] - Thanks for confirmation. I will mark this JIRA as verified.
> Enabling runtime filter eliminates more incoming rows than it should.
> -
>
> Key: DRILL-6871
> URL: https://issues.apache.org/jira/browse/DRILL-6871
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.15.0
>Reporter: Kunal Khatua
>Assignee: weijie.tong
>Priority: Major
> Fix For: 1.16.0
>
> Attachments: 24036b35-7c41-12c9-8c08-9327c8d8fdf2.sys.drill,
> 24036bbf-ede2-690a-93b7-9b43c28eed52.sys.drill,
> 24036c15-54a9-7b5d-07be-4d6de4f63731.sys.drill,
> 24036c22-7f39-ae8b-f8f4-000ecc2572cc.sys.drill
>
>
> When testing with the following combination on TPC-H dataset (scale factor
> 100) using a 4 node setup...
> {code:java}
> exec.hashjoin.bloom_filter.fpp=0.2
> exec.hashjoin.enable.runtime_filter=true
> exec.hashjoin.runtime_filter.max.waiting.time=2
> {code}
> It was observed that the filter eliminates more rows than it should.
>
> {code:java}
> 0: jdbc:drill:schema=dfs.par100> select count(*) from (select * from lineitem
> l, supplier s where l.l_suppkey = s.s_suppkey and s.s_acctbal <1000);
> +-+
> | EXPR$0 |
> +-+
> | 405566 |
> +-+
> 1 row selected (10.565 seconds)
> 0: jdbc:drill:schema=dfs.par100> select count(*) from (select * from lineitem
> l, supplier s where l.l_suppkey = s.s_suppkey and s.s_acctbal <1000);
> +-+
> | EXPR$0 |
> +-+
> | 405769 |
> +-+
> 1 row selected (9.845 seconds)
> {code}
> The expected row count for the above (broadcast-join) query should have been
> *109307880*
>
> {code:java}
> 0: jdbc:drill:schema=dfs.par100> select count(*) from (select * from lineitem
> l, orders o where o.o_orderkey = l.l_orderkey and o.o_totalprice < 10);
> +---+
> | EXPR$0 |
> +---+
> | 37338355 |
> +---+
> 1 row selected (44.698 seconds)
> 0: jdbc:drill:schema=dfs.par100> select count(*) from (select * from lineitem
> l, orders o where o.o_orderkey = l.l_orderkey and o.o_totalprice < 10);
> +---+
> | EXPR$0 |
> +---+
> | 38044874 |
> +---+
> 1 row selected (44.871 seconds)
> {code}
> The expected row count for the above (hash partition-join) query should have
> been *96176495*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-7046) Support for loading and parsing new RM config file
[ https://issues.apache.org/jira/browse/DRILL-7046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sorabh Hamirwasia updated DRILL-7046: - Description: This Jira will help to add support for loading new RM specific configuration file if needed and also parsing it to create all the required in-memory objects. The details of the new configuration is defined in [Function Spec|https://docs.google.com/document/d/1cX0lPLL-QzBGUwcekAPvUBgubdcNlWRdy7IG_IFt-Ok/edit?usp=sharing] RM configuration file will also support override, default and distrib specific files. It will only be loaded when RM is enabled which will be controlled by *drill.exec.rm.enabled* parameter in Drill main configuration file. (was: This Jira will help to add support for loading new RM specific configuration file if needed and also parsing it to create all the required in-memory objects. The details of the new configuration is defined in [Function Spec|[https://docs.google.com/document/d/1cX0lPLL-QzBGUwcekAPvUBgubdcNlWRdy7IG_IFt-Ok/edit?usp=sharing]|https://docs.google.com/document/d/1cX0lPLL-QzBGUwcekAPvUBgubdcNlWRdy7IG_IFt-Ok/edit?usp=sharing].] RM configuration file will also support override, default and distrib specific files. It will only be loaded when RM is enabled which will be controlled by *drill.exec.rm.enabled* parameter in Drill main configuration file.) > Support for loading and parsing new RM config file > -- > > Key: DRILL-7046 > URL: https://issues.apache.org/jira/browse/DRILL-7046 > Project: Apache Drill > Issue Type: Sub-task > Components: Execution - Flow >Affects Versions: 1.16.0 >Reporter: Sorabh Hamirwasia >Assignee: Sorabh Hamirwasia >Priority: Major > > This Jira will help to add support for loading new RM specific configuration > file if needed and also parsing it to create all the required in-memory > objects. The details of the new configuration is defined in [Function > Spec|https://docs.google.com/document/d/1cX0lPLL-QzBGUwcekAPvUBgubdcNlWRdy7IG_IFt-Ok/edit?usp=sharing] > RM configuration file will also support override, default and distrib > specific files. It will only be loaded when RM is enabled which will be > controlled by *drill.exec.rm.enabled* parameter in Drill main configuration > file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-7035) Drill C++ Client crashes on multiple SaslAuthenticatorImpl Destruction due to communication error
[
https://issues.apache.org/jira/browse/DRILL-7035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777397#comment-16777397
]
Sorabh Hamirwasia commented on DRILL-7035:
--
Hi [~debraj92],
Are you planning to submit a patch for this ?
> Drill C++ Client crashes on multiple SaslAuthenticatorImpl Destruction due to
> communication error
> --
>
> Key: DRILL-7035
> URL: https://issues.apache.org/jira/browse/DRILL-7035
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - C++
>Affects Versions: 1.12.0
>Reporter: Rob Wu
>Assignee: Debraj Ray
>Priority: Major
> Fix For: 1.16.0
>
>
> [~debraj92] found that when under some circumstance the
> SaslAuthenticatorImpl's sasl_dispose() function will crash out at
> destruction. The incident seems to be random and only when certain
> authentication and encryption combinations are used during connection.
> After digging a little deeper, I found that when BOOST communication error
> occurs, the shutdownSocket (eventually triggering sasl_dispose()) could be
> called from various threads resulting in a race condition of freeing the
> handle. This can be reproduced with the querysubmitter. This is reproducible
> since 1.12.0+.
> [~debraj92] will be adding a patch to resolve this incident.
>
> {code:java}
> 2019-Feb-11 10:44:01 : TRACE : 2d74 : DrillClientImpl::handleRead: Handle
> Read from buffer 04E1D850
> 2019-Feb-11 10:44:01 : TRACE : 2d74 : DrillClientImpl::handleRead: Cancel
> deadline timer.
> 2019-Feb-11 10:44:01 : TRACE : 2d74 : DrillClientImpl::handleRead:
> ERR_QRY_COMMERR. Boost Communication Error: End of file
> 2019-Feb-11 10:44:31 : TRACE : 3df8 : Disposing 1: +++ ENTER +++
> 2019-Feb-11 10:44:31 : TRACE : 2d74 : Disposing 2: +++ ENTER +++
> 2019-Feb-11 10:44:31 : TRACE : 2d74 : Disposing 2: --- EXIT ---
> 2019-Feb-11 10:44:31 : TRACE : 2d74 : Socket shutdown
> 2019-Feb-11 10:44:31 : TRACE : 3df8 : Disposing 1: --- EXIT ---
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6871) Enabling runtime filter eliminates more incoming rows than it should.
[
https://issues.apache.org/jira/browse/DRILL-6871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777407#comment-16777407
]
Sorabh Hamirwasia commented on DRILL-6871:
--
[~kkhatua] / [~aravi5] - Did you get a chance to verify this issue with all the
latest fixes ?
> Enabling runtime filter eliminates more incoming rows than it should.
> -
>
> Key: DRILL-6871
> URL: https://issues.apache.org/jira/browse/DRILL-6871
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.15.0
>Reporter: Kunal Khatua
>Assignee: weijie.tong
>Priority: Major
> Fix For: 1.16.0
>
> Attachments: 24036b35-7c41-12c9-8c08-9327c8d8fdf2.sys.drill,
> 24036bbf-ede2-690a-93b7-9b43c28eed52.sys.drill,
> 24036c15-54a9-7b5d-07be-4d6de4f63731.sys.drill,
> 24036c22-7f39-ae8b-f8f4-000ecc2572cc.sys.drill
>
>
> When testing with the following combination on TPC-H dataset (scale factor
> 100) using a 4 node setup...
> {code:java}
> exec.hashjoin.bloom_filter.fpp=0.2
> exec.hashjoin.enable.runtime_filter=true
> exec.hashjoin.runtime_filter.max.waiting.time=2
> {code}
> It was observed that the filter eliminates more rows than it should.
>
> {code:java}
> 0: jdbc:drill:schema=dfs.par100> select count(*) from (select * from lineitem
> l, supplier s where l.l_suppkey = s.s_suppkey and s.s_acctbal <1000);
> +-+
> | EXPR$0 |
> +-+
> | 405566 |
> +-+
> 1 row selected (10.565 seconds)
> 0: jdbc:drill:schema=dfs.par100> select count(*) from (select * from lineitem
> l, supplier s where l.l_suppkey = s.s_suppkey and s.s_acctbal <1000);
> +-+
> | EXPR$0 |
> +-+
> | 405769 |
> +-+
> 1 row selected (9.845 seconds)
> {code}
> The expected row count for the above (broadcast-join) query should have been
> *109307880*
>
> {code:java}
> 0: jdbc:drill:schema=dfs.par100> select count(*) from (select * from lineitem
> l, orders o where o.o_orderkey = l.l_orderkey and o.o_totalprice < 10);
> +---+
> | EXPR$0 |
> +---+
> | 37338355 |
> +---+
> 1 row selected (44.698 seconds)
> 0: jdbc:drill:schema=dfs.par100> select count(*) from (select * from lineitem
> l, orders o where o.o_orderkey = l.l_orderkey and o.o_totalprice < 10);
> +---+
> | EXPR$0 |
> +---+
> | 38044874 |
> +---+
> 1 row selected (44.871 seconds)
> {code}
> The expected row count for the above (hash partition-join) query should have
> been *96176495*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6245) Clicking on anything redirects to main login page
[ https://issues.apache.org/jira/browse/DRILL-6245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777405#comment-16777405 ] Sorabh Hamirwasia commented on DRILL-6245: -- [~harisekhon] - Can you provide exact steps and behavior to reproduce this issue. As explained by [~vdonapati] we see this issue only when Drill's webserver is moved from https to http endpoint. > Clicking on anything redirects to main login page > - > > Key: DRILL-6245 > URL: https://issues.apache.org/jira/browse/DRILL-6245 > Project: Apache Drill > Issue Type: Bug >Reporter: Venkata Jyothsna Donapati >Assignee: Venkata Jyothsna Donapati >Priority: Minor > Fix For: 1.17.0 > > > When the Drill Web UI is accessed using https and then by http protocol, the > Web UI is always trying to redirect to main login page if anything is clicked > on index page. However, this works fine if the cookies are cleared. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6245) Clicking on anything redirects to main login page
[ https://issues.apache.org/jira/browse/DRILL-6245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sorabh Hamirwasia updated DRILL-6245: - Fix Version/s: (was: 1.16.0) 1.17.0 > Clicking on anything redirects to main login page > - > > Key: DRILL-6245 > URL: https://issues.apache.org/jira/browse/DRILL-6245 > Project: Apache Drill > Issue Type: Bug >Reporter: Venkata Jyothsna Donapati >Assignee: Venkata Jyothsna Donapati >Priority: Minor > Fix For: 1.17.0 > > > When the Drill Web UI is accessed using https and then by http protocol, the > Web UI is always trying to redirect to main login page if anything is clicked > on index page. However, this works fine if the cookies are cleared. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6615) To prevent the limit operator after topN operator if there is a single fragment
[
https://issues.apache.org/jira/browse/DRILL-6615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sorabh Hamirwasia updated DRILL-6615:
-
Fix Version/s: (was: 1.16.0)
1.17.0
> To prevent the limit operator after topN operator if there is a single
> fragment
> ---
>
> Key: DRILL-6615
> URL: https://issues.apache.org/jira/browse/DRILL-6615
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.14.0
>Reporter: Kedar Sankar Behera
>Assignee: Sorabh Hamirwasia
>Priority: Major
> Fix For: 1.17.0
>
> Attachments: topnNlimit.pdf
>
>
> The limit operator is called after topN operator which is not needed if u
> have only 1 fragment .
> For eg :-
> {code}
> 00-00 Screen : rowType = RecordType(ANY c_custkey, ANY c_name, ANY EXPR$2,
> ANY EXPR$3): rowcount = 50.0, cumulative cost = \{116621.0 rows,
> 2787624.201857771 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 7854 00-01
> Project(c_custkey=[$0], c_name=[$1], EXPR$2=[$2], EXPR$3=[$3]) : rowType =
> RecordType(ANY c_custkey, ANY c_name, ANY EXPR$2, ANY EXPR$3): rowcount =
> 50.0, cumulative cost = \{116616.0 rows, 2787619.201857771 cpu, 0.0 io, 0.0
> network, 0.0 memory}, id = 7853 00-02 SelectionVectorRemover : rowType =
> RecordType(ANY c_custkey, ANY c_name, ANY ITEM, ANY ITEM1): rowcount = 50.0,
> cumulative cost = \{116566.0 rows, 2787419.201857771 cpu, 0.0 io, 0.0
> network, 0.0 memory}, id = 7852 00-03 Limit(fetch=[50]) : rowType =
> RecordType(ANY c_custkey, ANY c_name, ANY ITEM, ANY ITEM1): rowcount = 50.0,
> cumulative cost = \{116516.0 rows, 2787369.201857771 cpu, 0.0 io, 0.0
> network, 0.0 memory}, id = 7851 00-04 SelectionVectorRemover : rowType =
> RecordType(ANY c_custkey, ANY c_name, ANY ITEM, ANY ITEM1): rowcount =
> 29116.0, cumulative cost = \{116466.0 rows, 2787169.201857771 cpu, 0.0 io,
> 0.0 network, 0.0 memory}, id = 7850 00-05 TopN(limit=[50]) : rowType =
> RecordType(ANY c_custkey, ANY c_name, ANY ITEM, ANY ITEM1): rowcount =
> 29116.0, cumulative cost = \{87350.0 rows, 2758053.201857771 cpu, 0.0 io, 0.0
> network, 0.0 memory}, id = 7849 00-06 LateralJoin(correlation=[$cor2],
> joinType=[inner], requiredColumns=[\{0}], column excluded from output:
> =[`c_orders`]) : rowType = RecordType(ANY c_custkey, ANY c_name, ANY ITEM,
> ANY ITEM1): rowcount = 29116.0, cumulative cost = \{58234.0 rows, 786135.0
> cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 7848 00-08
> Scan(groupscan=[EasyGroupScan
> [selectionRoot=maprfs:/drill/testdata/lateralUnnest/sf0dot01/json/customer,
> numFiles=1, columns=[`c_orders`, `c_custkey`, `c_name`],
> files=[maprfs:///drill/testdata/lateralUnnest/sf0dot01/json/customer/customer.json]]])
> : rowType = RecordType(ANY c_orders, ANY c_custkey, ANY c_name): rowcount =
> 29116.0, cumulative cost = \{29116.0 rows, 87348.0 cpu, 0.0 io, 0.0 network,
> 0.0 memory}, id = 7846 00-07 Project(ITEM=[ITEM($0, 'o_orderkey')],
> ITEM1=[ITEM($0, 'o_totalprice')]) : rowType = RecordType(ANY ITEM, ANY
> ITEM1): rowcount = 1.0, cumulative cost = \{2.0 rows, 3.0 cpu, 0.0 io, 0.0
> network, 0.0 memory}, id = 7847 00-09 Unnest [srcOp=00-06] : rowType =
> RecordType(ANY c_orders): rowcount = 1.0, cumulative cost = \{1.0 rows, 1.0
> cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 7700
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6991) Kerberos ticket is being dumped in the log if log level is "debug" for stdout
[
https://issues.apache.org/jira/browse/DRILL-6991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777400#comment-16777400
]
Sorabh Hamirwasia commented on DRILL-6991:
--
[~angozhiy] - Can you please close the Jira if you agree this is not an issue ?
> Kerberos ticket is being dumped in the log if log level is "debug" for stdout
> --
>
> Key: DRILL-6991
> URL: https://issues.apache.org/jira/browse/DRILL-6991
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.15.0
>Reporter: Anton Gozhiy
>Assignee: Sorabh Hamirwasia
>Priority: Major
> Fix For: 1.16.0
>
>
> *Prerequisites:*
> # Drill is installed on cluster with Kerberos security
> # Into conf/logback.xml, set the following log level:
> {code:xml}
>
>
>
>
> {code}
> *Steps:*
> # Start Drill
> # Connect using sqlline using the following string:
> {noformat}
> bin/sqlline -u "jdbc:drill:zk=;principal="
> {noformat}
> *Expected result:*
> No sensitive information should be displayed
> *Actual result:*
> Kerberos ticket and session key are being dumped into console output:
> {noformat}
> 14:35:38.806 [TGT Renewer for mapr/node1.cluster.com@NODE1] DEBUG
> o.a.h.security.UserGroupInformation - Found tgt Ticket (hex) =
> : 61 82 01 3D 30 82 01 39 A0 03 02 01 05 A1 07 1B a..=0..9
> 0010: 05 4E 4F 44 45 31 A2 1A 30 18 A0 03 02 01 02 A1 .NODE1..0...
> 0020: 11 30 0F 1B 06 6B 72 62 74 67 74 1B 05 4E 4F 44 .0...krbtgt..NOD
> 0030: 45 31 A3 82 01 0B 30 82 01 07 A0 03 02 01 12 A1 E10.
> 0040: 03 02 01 01 A2 81 FA 04 81 F7 03 8D A9 FA 7D 89
> 0050: 1B DF 37 B7 4D E6 6C 99 3E 8F FA 48 D9 9A 79 F3 ..7.M.l.>..H..y.
> 0060: 92 34 7F BF 67 1E 77 4A 2F C9 AF 82 93 4E 46 1D .4..g.wJ/NF.
> 0070: 41 74 B0 AF 41 A8 8B 02 71 83 CC 14 51 72 60 EE At..A...q...Qr`.
> 0080: 29 67 14 F0 A6 33 63 07 41 AA 8D DC 7B 5B 41 F3 )g...3c.A[A.
> 0090: 83 48 8B 2A 0B 4D 6D 57 9A 6E CF 6B DC 0B C0 D1 .H.*.MmW.n.k
> 00A0: 83 BB 27 40 88 7E 9F 2B D1 FD A8 6A E1 BF F6 CC ..'@...+...j
> 00B0: 0E 0C FB 93 5D 69 9A 8B 11 88 0C F2 7C E1 FD 04 ]i..
> 00C0: F5 AB 66 0C A4 A4 7B 30 D1 7F F1 2D D6 A1 52 D1 ..f0...-..R.
> 00D0: 79 59 F2 06 CB 65 FB 73 63 1D 5B E9 4F 28 73 EB yY...e.sc.[.O(s.
> 00E0: 72 7F 04 46 34 56 F4 40 6C C0 2C 39 C0 5B C6 25 r..F4V.@l.,9.[.%
> 00F0: ED EF 64 07 CE ED 35 9D D7 91 6C 8F C9 CE 16 F5 ..d...5...l.
> 0100: CA 5E 6F DE 08 D2 68 30 C7 03 97 E7 C0 FF D9 52 .^o...h0...R
> 0110: F8 1D 2F DB 63 6D 12 4A CD 60 AD D0 BA FA 4B CF ../.cm.J.`K.
> 0120: 2C B9 8C CA 5A E6 EC 10 5A 0A 1F 84 B0 80 BD 39 ,...Z...Z..9
> 0130: 42 2C 33 EB C0 AA 0D 44 F0 F4 E9 87 24 43 BB 9A B,3D$C..
> 0140: 52 R
> Client Principal = mapr/node1.cluster.com@NODE1
> Server Principal = krbtgt/NODE1@NODE1
> Session Key = EncryptionKey: keyType=18 keyBytes (hex dump)=
> : 50 DA D1 D7 91 D3 64 BE 45 7B D8 02 25 81 18 25 P.d.E...%..%
> 0010: DA 59 4F BA 76 67 BB 39 9C F7 17 46 A7 C5 00 E2 .YO.vg.9...F
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-7046) Support for loading and parsing new RM config file
Sorabh Hamirwasia created DRILL-7046: Summary: Support for loading and parsing new RM config file Key: DRILL-7046 URL: https://issues.apache.org/jira/browse/DRILL-7046 Project: Apache Drill Issue Type: Sub-task Components: Execution - Flow Affects Versions: 1.16.0 Reporter: Sorabh Hamirwasia Assignee: Sorabh Hamirwasia This Jira will help to add support for loading new RM specific configuration file if needed and also parsing it to create all the required in-memory objects. The details of the new configuration is defined in [Function Spec|[https://docs.google.com/document/d/1cX0lPLL-QzBGUwcekAPvUBgubdcNlWRdy7IG_IFt-Ok/edit?usp=sharing]|https://docs.google.com/document/d/1cX0lPLL-QzBGUwcekAPvUBgubdcNlWRdy7IG_IFt-Ok/edit?usp=sharing].] RM configuration file will also support override, default and distrib specific files. It will only be loaded when RM is enabled which will be controlled by *drill.exec.rm.enabled* parameter in Drill main configuration file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6855) Query from non-existent proxy user fails with "No default schema selected" when impersonation is enabled
[
https://issues.apache.org/jira/browse/DRILL-6855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sorabh Hamirwasia updated DRILL-6855:
-
Reviewer: Sorabh Hamirwasia
> Query from non-existent proxy user fails with "No default schema selected"
> when impersonation is enabled
>
>
> Key: DRILL-6855
> URL: https://issues.apache.org/jira/browse/DRILL-6855
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.15.0
>Reporter: Abhishek Ravi
>Assignee: Abhishek Ravi
>Priority: Major
> Fix For: 1.16.0
>
>
> Query from a *proxy user* fails with following error when *impersonation* is
> *enabled* but user does not exist. This behaviour was discovered when running
> Drill on MapR.
> {noformat}
> Error: VALIDATION ERROR: Schema [[dfs]] is not valid with respect to either
> root schema or current default schema.
> Current default schema: No default schema selected
> {noformat}
> The above error is very confusing and made it very hard to relate to proxy
> user does not exist + impersonation issue.
> The {{fs.access(wsPath, FsAction.READ)}} in
> {{WorkspaceSchemaFactory.accessible fails with IOException,}} which is not
> handled in {{accessible}} but in {{DynamicRootSchema.loadSchemaFactory}}. At
> this point none of the schemas are registered and hence the root schema will
> be registered as default schema.
> The query execution continues and fails much ahead at
> {{DrillSqlWorker.getQueryPlan}} where the {{SqlConverter.validate}}
> eventually throws {{SchemaUtilites.throwSchemaNotFoundException}}.
> One possible fix could be to handle {{IOException}} similar to
> {{FileNotFoundException}} in {{WorkspaceSchemaFactory.accessible}}.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6855) Query from non-existent proxy user fails with "No default schema selected" when impersonation is enabled
[
https://issues.apache.org/jira/browse/DRILL-6855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sorabh Hamirwasia updated DRILL-6855:
-
Labels: ready-to-commit (was: )
> Query from non-existent proxy user fails with "No default schema selected"
> when impersonation is enabled
>
>
> Key: DRILL-6855
> URL: https://issues.apache.org/jira/browse/DRILL-6855
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.15.0
>Reporter: Abhishek Ravi
>Assignee: Abhishek Ravi
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.16.0
>
>
> Query from a *proxy user* fails with following error when *impersonation* is
> *enabled* but user does not exist. This behaviour was discovered when running
> Drill on MapR.
> {noformat}
> Error: VALIDATION ERROR: Schema [[dfs]] is not valid with respect to either
> root schema or current default schema.
> Current default schema: No default schema selected
> {noformat}
> The above error is very confusing and made it very hard to relate to proxy
> user does not exist + impersonation issue.
> The {{fs.access(wsPath, FsAction.READ)}} in
> {{WorkspaceSchemaFactory.accessible fails with IOException,}} which is not
> handled in {{accessible}} but in {{DynamicRootSchema.loadSchemaFactory}}. At
> this point none of the schemas are registered and hence the root schema will
> be registered as default schema.
> The query execution continues and fails much ahead at
> {{DrillSqlWorker.getQueryPlan}} where the {{SqlConverter.validate}}
> eventually throws {{SchemaUtilites.throwSchemaNotFoundException}}.
> One possible fix could be to handle {{IOException}} similar to
> {{FileNotFoundException}} in {{WorkspaceSchemaFactory.accessible}}.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-7026) Support Resource Management across queries in Drill
[ https://issues.apache.org/jira/browse/DRILL-7026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sorabh Hamirwasia updated DRILL-7026: - Description: Tracks support for Resource Management with dynamic memory requirement across queries using current cluster state. It will also improve multi-tenancy support to configure multiple named queues for a Drill cluster. (was: Tracks support for dynamic memory resource management across queries using current cluster state. It will also improve multi-tenancy support to configure multiple named queues for a Drill cluster.) > Support Resource Management across queries in Drill > --- > > Key: DRILL-7026 > URL: https://issues.apache.org/jira/browse/DRILL-7026 > Project: Apache Drill > Issue Type: New Feature > Components: Execution - Flow, Query Planning & Optimization >Affects Versions: 1.16.0 >Reporter: Sorabh Hamirwasia >Assignee: Sorabh Hamirwasia >Priority: Major > Labels: doc-impacting > Fix For: Future > > > Tracks support for Resource Management with dynamic memory requirement across > queries using current cluster state. It will also improve multi-tenancy > support to configure multiple named queues for a Drill cluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-7026) Support Resource Management across queries in Drill
[ https://issues.apache.org/jira/browse/DRILL-7026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16760267#comment-16760267 ] Sorabh Hamirwasia commented on DRILL-7026: -- [~hmaduri] [~karthikm] > Support Resource Management across queries in Drill > --- > > Key: DRILL-7026 > URL: https://issues.apache.org/jira/browse/DRILL-7026 > Project: Apache Drill > Issue Type: New Feature > Components: Execution - Flow, Query Planning & Optimization >Affects Versions: 1.16.0 >Reporter: Sorabh Hamirwasia >Assignee: Sorabh Hamirwasia >Priority: Major > Labels: doc-impacting > Fix For: Future > > > Tracks support for dynamic memory resource management across queries using > current cluster state. It will also improve multi-tenancy support to > configure multiple named queues for a Drill cluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-7026) Support Resource Management across queries in Drill
[ https://issues.apache.org/jira/browse/DRILL-7026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sorabh Hamirwasia updated DRILL-7026: - Issue Type: New Feature (was: Task) > Support Resource Management across queries in Drill > --- > > Key: DRILL-7026 > URL: https://issues.apache.org/jira/browse/DRILL-7026 > Project: Apache Drill > Issue Type: New Feature > Components: Execution - Flow, Query Planning & Optimization >Affects Versions: 1.16.0 >Reporter: Sorabh Hamirwasia >Assignee: Sorabh Hamirwasia >Priority: Major > Labels: doc-impacting > Fix For: Future > > > Tracks support for dynamic memory resource management across queries using > current cluster state. It will also improve multi-tenancy support to > configure multiple named queues for a Drill cluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-7026) Support Resource Management across queries in Drill
Sorabh Hamirwasia created DRILL-7026: Summary: Support Resource Management across queries in Drill Key: DRILL-7026 URL: https://issues.apache.org/jira/browse/DRILL-7026 Project: Apache Drill Issue Type: Task Components: Execution - Flow, Query Planning & Optimization Affects Versions: 1.16.0 Reporter: Sorabh Hamirwasia Assignee: Sorabh Hamirwasia Fix For: Future Tracks support for dynamic memory resource management across queries using current cluster state. It will also improve multi-tenancy support to configure multiple named queues for a Drill cluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-7016) Wrong query result with RuntimeFilter enabled when order of join and filter condition is swapped
[
https://issues.apache.org/jira/browse/DRILL-7016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16755771#comment-16755771
]
Sorabh Hamirwasia commented on DRILL-7016:
--
Looked more into the issue and it's *not* because of filter on top of Runtime
Filter operator. While generating RuntimeFilter there is a bug in which left
and right side fields in join condition is decided to be used in BloomFilter.
It doesn't uses the ordinals for right keys and instead directly get the field
name starting with index 0 for each left keys. Hence with changing order of
filter and join condition the ordinals of right fields changes and bloom filter
is generated for wrong right side field. For example: In case 1 below
bloomFilter is generated on right side column c_mktsegment instead of
c_custkey. Whereas in case 2 bloomFilter is generated on right side column
c_custkey.
*Case 1:*
{code:java}
01-04 HashJoin(condition=[=($2, $0)], joinType=[inner], semi-join: =[false]) :
rowType = RecordType(ANY o_custkey, ANY c_mktsegment, ANY c_custkey): rowcount
= 1.5E7, cumulative cost = {6.3675E7 rows, 2.38725E8 cpu, 1.8E7 io, 3.87072E9
network, 396.05 memory}, id = 65202{code}
1 row selected (3.654 seconds)
{code:java}
0: jdbc:drill:drillbits=10.10.100.188> select count(*)
. . . . . . . . . . . . . . semicolon> from
. . . . . . . . . . . . . . semicolon> customer c,
. . . . . . . . . . . . . . semicolon> orders o
. . . . . . . . . . . . . . semicolon> where c.c_mktsegment = 'HOUSEHOLD'
. . . . . . . . . . . . . . semicolon> and c.c_custkey = o.o_custkey;{code}
+-+
| EXPR$0 |
+-+
| 19826 |
+-+
*Case 2:*
{code:java}
01-04 HashJoin(condition=[=($1, $0)], joinType=[inner], semi-join: =[false]) :
rowType = RecordType(ANY o_custkey, ANY c_custkey, ANY c_mktsegment): rowcount
= 1.5E7, cumulative cost = {6.3675E7 rows, 2.38725E8 cpu, 1.8E7 io, 3.87072E9
network, 396.05 memory}, id = 66134{code}
1 row selected (1.328 seconds)
{code:java}
0: jdbc:drill:drillbits=10.10.100.188> select count(*)
. . . . . . . . . . . . . . semicolon> from
. . . . . . . . . . . . . . semicolon> customer c,
. . . . . . . . . . . . . . semicolon> orders o
. . . . . . . . . . . . . . semicolon> where c.c_custkey = o.o_custkey and
. . . . . . . . . . . . . . semicolon> c.c_mktsegment = 'HOUSEHOLD'
. . . . . . . . . . . . . . semicolon> ;{code}
+--+
| EXPR$0 |
+--+
| 2990828 |
+--+
> Wrong query result with RuntimeFilter enabled when order of join and filter
> condition is swapped
>
>
> Key: DRILL-7016
> URL: https://issues.apache.org/jira/browse/DRILL-7016
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.16.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sorabh Hamirwasia
>Priority: Major
> Fix For: 1.16.0
>
>
> Below 2 queries generate different results:
> *Query1: Result: 19826*
> {code:java}
> select count(*)
> from
> customer c,
> orders o
> where
> c.c_mktsegment = 'HOUSEHOLD'
> and c.c_custkey = o.o_custkey
> {code}
> *Query2: Result: 2990828*
> {code:java}
> select count(*)
> from
> customer c,
> orders o
> where
> c.c_custkey = o.o_custkey and
> c.c_mktsegment = 'HOUSEHOLD'
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-7016) Wrong query result with RuntimeFilter enabled when order of join and filter condition is swapped
[
https://issues.apache.org/jira/browse/DRILL-7016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sorabh Hamirwasia updated DRILL-7016:
-
Reviewer: weijie.tong
> Wrong query result with RuntimeFilter enabled when order of join and filter
> condition is swapped
>
>
> Key: DRILL-7016
> URL: https://issues.apache.org/jira/browse/DRILL-7016
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.16.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sorabh Hamirwasia
>Priority: Major
> Fix For: 1.16.0
>
>
> Below 2 queries generate different results:
> *Query1: Result: 19826*
> {code:java}
> select count(*)
> from
> customer c,
> orders o
> where
> c.c_mktsegment = 'HOUSEHOLD'
> and c.c_custkey = o.o_custkey
> {code}
> *Query2: Result: 2990828*
> {code:java}
> select count(*)
> from
> customer c,
> orders o
> where
> c.c_custkey = o.o_custkey and
> c.c_mktsegment = 'HOUSEHOLD'
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-7016) Wrong query result with RuntimeFilter enabled when order of join and filter condition is swapped
Sorabh Hamirwasia created DRILL-7016:
Summary: Wrong query result with RuntimeFilter enabled when order
of join and filter condition is swapped
Key: DRILL-7016
URL: https://issues.apache.org/jira/browse/DRILL-7016
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.16.0
Reporter: Sorabh Hamirwasia
Assignee: Sorabh Hamirwasia
Fix For: 1.16.0
Below 2 queries generate different results:
*Query1: Result: 19826*
{code:java}
select count(*)
from
customer c,
orders o
where
c.c_mktsegment = 'HOUSEHOLD'
and c.c_custkey = o.o_custkey
{code}
*Query2: Result: 2990828*
{code:java}
select count(*)
from
customer c,
orders o
where
c.c_custkey = o.o_custkey and
c.c_mktsegment = 'HOUSEHOLD'
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-7002) RuntimeFilter produce wrong results while setting exec.hashjoin.num_partitions=1
[ https://issues.apache.org/jira/browse/DRILL-7002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sorabh Hamirwasia updated DRILL-7002: - Component/s: Execution - Relational Operators > RuntimeFilter produce wrong results while setting > exec.hashjoin.num_partitions=1 > > > Key: DRILL-7002 > URL: https://issues.apache.org/jira/browse/DRILL-7002 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Relational Operators >Affects Versions: 1.16.0 >Reporter: weijie.tong >Assignee: weijie.tong >Priority: Major > Labels: ready-to-commit > Fix For: 1.16.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (DRILL-6998) Queries failing with "Failed to aggregate or route the RFW" due to "java.lang.ArrayIndexOutOfBoundsException" do not complete
[
https://issues.apache.org/jira/browse/DRILL-6998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sorabh Hamirwasia resolved DRILL-6998.
--
Resolution: Fixed
> Queries failing with "Failed to aggregate or route the RFW" due to
> "java.lang.ArrayIndexOutOfBoundsException" do not complete
> -
>
> Key: DRILL-6998
> URL: https://issues.apache.org/jira/browse/DRILL-6998
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.16.0
>Reporter: Abhishek Ravi
>Assignee: weijie.tong
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.16.0
>
>
> Following query joins 2 tables on *two* (>1) fields.
> {noformat}
> select count(*) from lineitem l inner join partsupp p on l.l_partkey =
> p.ps_partkey AND l.l_suppkey = p.ps_suppkey
> {noformat}
> The query does not return even though Fragment 0:0 reports a state change
> from {{RUNNING}} -> {{FINISHED}}
> Following is the jstack output of the {{Frag0:0}}.
> {noformat}
> "23b85137-b102-39a9-70d9-72381c5fb93b:frag:0:0" #16037 daemon prio=10
> os_prio=0 tid=0x7f5f48d415d0 nid=0x1a61 waiting on condition
> [0x7f61b32b2000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.drill.exec.work.filter.RuntimeFilterSink.close(RuntimeFilterSink.java:116)
> at
> org.apache.drill.exec.work.filter.RuntimeFilterRouter.waitForComplete(RuntimeFilterRouter.java:113)
> at
> org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:738)
> at
> org.apache.drill.exec.work.foreman.QueryStateProcessor.wrapUpCompletion(QueryStateProcessor.java:315)
> at
> org.apache.drill.exec.work.foreman.QueryStateProcessor.running(QueryStateProcessor.java:276)
> at
> org.apache.drill.exec.work.foreman.QueryStateProcessor.moveToState(QueryStateProcessor.java:92)
> - locked <0x00055f9a7468> (a
> org.apache.drill.exec.work.foreman.QueryStateProcessor)
> at
> org.apache.drill.exec.work.foreman.QueryStateProcessor$StateSwitch.processEvent(QueryStateProcessor.java:349)
> at
> org.apache.drill.exec.work.foreman.QueryStateProcessor$StateSwitch.processEvent(QueryStateProcessor.java:342)
> at
> org.apache.drill.common.EventProcessor.processEvents(EventProcessor.java:107)
> at
> org.apache.drill.common.EventProcessor.sendEvent(EventProcessor.java:65)
> at
> org.apache.drill.exec.work.foreman.QueryStateProcessor$StateSwitch.addEvent(QueryStateProcessor.java:344)
> at
> org.apache.drill.exec.work.foreman.QueryStateProcessor.addToEventQueue(QueryStateProcessor.java:155)
> at
> org.apache.drill.exec.work.foreman.Foreman.addToEventQueue(Foreman.java:213)
> at
> org.apache.drill.exec.work.foreman.QueryManager.nodeComplete(QueryManager.java:519)
> at
> org.apache.drill.exec.work.foreman.QueryManager.access$100(QueryManager.java:65)
> at
> org.apache.drill.exec.work.foreman.QueryManager$NodeTracker.fragmentComplete(QueryManager.java:483)
> at
> org.apache.drill.exec.work.foreman.QueryManager.fragmentDone(QueryManager.java:155)
> at
> org.apache.drill.exec.work.foreman.QueryManager.access$400(QueryManager.java:65)
> at
> org.apache.drill.exec.work.foreman.QueryManager$1.statusUpdate(QueryManager.java:546)
> at
> org.apache.drill.exec.rpc.control.WorkEventBus.statusUpdate(WorkEventBus.java:63)
> at
> org.apache.drill.exec.work.batch.ControlMessageHandler.requestFragmentStatus(ControlMessageHandler.java:253)
> at
> org.apache.drill.exec.rpc.control.LocalControlConnectionManager.runCommand(LocalControlConnectionManager.java:130)
> at
> org.apache.drill.exec.rpc.control.ControlTunnel.sendFragmentStatus(ControlTunnel.java:89)
> at
> org.apache.drill.exec.work.fragment.FragmentStatusReporter.sendStatus(FragmentStatusReporter.java:122)
> at
> org.apache.drill.exec.work.fragment.FragmentStatusReporter.stateChanged(FragmentStatusReporter.java:91)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:367)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:219)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:330)
> at
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> a
[jira] [Updated] (DRILL-7002) RuntimeFilter produce wrong results while setting exec.hashjoin.num_partitions=1
[ https://issues.apache.org/jira/browse/DRILL-7002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sorabh Hamirwasia updated DRILL-7002: - Reviewer: Sorabh Hamirwasia > RuntimeFilter produce wrong results while setting > exec.hashjoin.num_partitions=1 > > > Key: DRILL-7002 > URL: https://issues.apache.org/jira/browse/DRILL-7002 > Project: Apache Drill > Issue Type: Bug >Reporter: weijie.tong >Assignee: weijie.tong >Priority: Major > Fix For: 1.16.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-7002) RuntimeFilter produce wrong results while setting exec.hashjoin.num_partitions=1
[ https://issues.apache.org/jira/browse/DRILL-7002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sorabh Hamirwasia updated DRILL-7002: - Affects Version/s: 1.16.0 > RuntimeFilter produce wrong results while setting > exec.hashjoin.num_partitions=1 > > > Key: DRILL-7002 > URL: https://issues.apache.org/jira/browse/DRILL-7002 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.16.0 >Reporter: weijie.tong >Assignee: weijie.tong >Priority: Major > Labels: ready-to-commit > Fix For: 1.16.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-7002) RuntimeFilter produce wrong results while setting exec.hashjoin.num_partitions=1
[ https://issues.apache.org/jira/browse/DRILL-7002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sorabh Hamirwasia updated DRILL-7002: - Labels: ready-to-commit (was: ) > RuntimeFilter produce wrong results while setting > exec.hashjoin.num_partitions=1 > > > Key: DRILL-7002 > URL: https://issues.apache.org/jira/browse/DRILL-7002 > Project: Apache Drill > Issue Type: Bug >Reporter: weijie.tong >Assignee: weijie.tong >Priority: Major > Labels: ready-to-commit > Fix For: 1.16.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6983) PAM Auth Enabled on Drill-On-YARN only works on YARN user
[
https://issues.apache.org/jira/browse/DRILL-6983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16752010#comment-16752010
]
Sorabh Hamirwasia commented on DRILL-6983:
--
Can you please check if you have followed all the steps from here [1] like
making Drill process user part of shadow group, etc
Also the recommended pam module to use will be libpam4j [2] instead of jpam. So
please try using it and see if it makes any difference.
[1]: https://drill.apache.org/docs/configuring-plain-security/
[2]: https://drill.apache.org/docs/using-libpam4j-as-the-pam-authenticator/
> PAM Auth Enabled on Drill-On-YARN only works on YARN user
> -
>
> Key: DRILL-6983
> URL: https://issues.apache.org/jira/browse/DRILL-6983
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - HTTP
>Affects Versions: 1.14.0, 1.15.0
>Reporter: Michael Dennis Uanang
>Priority: Major
> Attachments: Selection_999(203).png, Selection_999(204).png,
> Selection_999(205).png
>
>
> Hi,
> I'm having problem running Drill-on-YARN with PAM authentication enabled. PAM
> auth is working, BUT only accepting login via WEBUI for YARN user.
> _drill-override.conf_
>
> {code:java}
> drill.exec: {
> cluster-id: "drillbits2",
> zk.connect: "app40:2181,app41:2181,app42:2181",
> impersonation: {
> enabled: true
> },
> security: {
> auth.mechanisms: [ "PLAIN" ],
> user.auth.enabled: true,
> user.auth.packages += "org.apache.drill.exec.rpc.user.security",
> user.auth.impl: "pam",
> user.auth.pam_profiles: [ "login", "sshd" ]
> }
> }
> {code}
>
>
> SEE errors below:
> !Selection_999(204).png!
>
> !Selection_999(203).png!
> As you can see from the screenshot, when trying to login via WEBUI using
> infra or drill user, I'm having error 'password check failed for user
> (USER)`. But you'll also notice that it's giving me authentication failure
> for UID=1018 which is YARN
> !Selection_999(205).png!
>
> Please help me to right direction or if I'm missing something.
> Thank you.
>
> MD
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6947) RuntimeFilter memory leak due to BF ByteBuf ownership transferring
[ https://issues.apache.org/jira/browse/DRILL-6947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sorabh Hamirwasia updated DRILL-6947: - Component/s: Execution - Flow > RuntimeFilter memory leak due to BF ByteBuf ownership transferring > --- > > Key: DRILL-6947 > URL: https://issues.apache.org/jira/browse/DRILL-6947 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Flow >Affects Versions: 1.16.0 >Reporter: weijie.tong >Assignee: weijie.tong >Priority: Major > Labels: ready-to-commit > Fix For: 1.16.0 > > > RuntimeFilter's BF ByteBuf ownership should be transferred right at broadcast > and random hash cases. Currently due to we not treat this transferring > reasonable, it caused the memory leak. > To broadcast case,the HashJoin operator's allocator allocated the BF, the > allocated BF's ownership should be transferred to its receiver : the > FragmentContextImpl or the final RuntimeFilter operator. Otherwise, the > OperatorContextImpl's close method will complain about the memory leak when > closing the corresponding allocator. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6998) Queries failing with "Failed to aggregate or route the RFW" due to "java.lang.ArrayIndexOutOfBoundsException" do not complete
[
https://issues.apache.org/jira/browse/DRILL-6998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sorabh Hamirwasia updated DRILL-6998:
-
Reviewer: Sorabh Hamirwasia
> Queries failing with "Failed to aggregate or route the RFW" due to
> "java.lang.ArrayIndexOutOfBoundsException" do not complete
> -
>
> Key: DRILL-6998
> URL: https://issues.apache.org/jira/browse/DRILL-6998
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.16.0
>Reporter: Abhishek Ravi
>Assignee: weijie.tong
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.16.0
>
>
> Following query joins 2 tables on *two* (>1) fields.
> {noformat}
> select count(*) from lineitem l inner join partsupp p on l.l_partkey =
> p.ps_partkey AND l.l_suppkey = p.ps_suppkey
> {noformat}
> The query does not return even though Fragment 0:0 reports a state change
> from {{RUNNING}} -> {{FINISHED}}
> Following is the jstack output of the {{Frag0:0}}.
> {noformat}
> "23b85137-b102-39a9-70d9-72381c5fb93b:frag:0:0" #16037 daemon prio=10
> os_prio=0 tid=0x7f5f48d415d0 nid=0x1a61 waiting on condition
> [0x7f61b32b2000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.drill.exec.work.filter.RuntimeFilterSink.close(RuntimeFilterSink.java:116)
> at
> org.apache.drill.exec.work.filter.RuntimeFilterRouter.waitForComplete(RuntimeFilterRouter.java:113)
> at
> org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:738)
> at
> org.apache.drill.exec.work.foreman.QueryStateProcessor.wrapUpCompletion(QueryStateProcessor.java:315)
> at
> org.apache.drill.exec.work.foreman.QueryStateProcessor.running(QueryStateProcessor.java:276)
> at
> org.apache.drill.exec.work.foreman.QueryStateProcessor.moveToState(QueryStateProcessor.java:92)
> - locked <0x00055f9a7468> (a
> org.apache.drill.exec.work.foreman.QueryStateProcessor)
> at
> org.apache.drill.exec.work.foreman.QueryStateProcessor$StateSwitch.processEvent(QueryStateProcessor.java:349)
> at
> org.apache.drill.exec.work.foreman.QueryStateProcessor$StateSwitch.processEvent(QueryStateProcessor.java:342)
> at
> org.apache.drill.common.EventProcessor.processEvents(EventProcessor.java:107)
> at
> org.apache.drill.common.EventProcessor.sendEvent(EventProcessor.java:65)
> at
> org.apache.drill.exec.work.foreman.QueryStateProcessor$StateSwitch.addEvent(QueryStateProcessor.java:344)
> at
> org.apache.drill.exec.work.foreman.QueryStateProcessor.addToEventQueue(QueryStateProcessor.java:155)
> at
> org.apache.drill.exec.work.foreman.Foreman.addToEventQueue(Foreman.java:213)
> at
> org.apache.drill.exec.work.foreman.QueryManager.nodeComplete(QueryManager.java:519)
> at
> org.apache.drill.exec.work.foreman.QueryManager.access$100(QueryManager.java:65)
> at
> org.apache.drill.exec.work.foreman.QueryManager$NodeTracker.fragmentComplete(QueryManager.java:483)
> at
> org.apache.drill.exec.work.foreman.QueryManager.fragmentDone(QueryManager.java:155)
> at
> org.apache.drill.exec.work.foreman.QueryManager.access$400(QueryManager.java:65)
> at
> org.apache.drill.exec.work.foreman.QueryManager$1.statusUpdate(QueryManager.java:546)
> at
> org.apache.drill.exec.rpc.control.WorkEventBus.statusUpdate(WorkEventBus.java:63)
> at
> org.apache.drill.exec.work.batch.ControlMessageHandler.requestFragmentStatus(ControlMessageHandler.java:253)
> at
> org.apache.drill.exec.rpc.control.LocalControlConnectionManager.runCommand(LocalControlConnectionManager.java:130)
> at
> org.apache.drill.exec.rpc.control.ControlTunnel.sendFragmentStatus(ControlTunnel.java:89)
> at
> org.apache.drill.exec.work.fragment.FragmentStatusReporter.sendStatus(FragmentStatusReporter.java:122)
> at
> org.apache.drill.exec.work.fragment.FragmentStatusReporter.stateChanged(FragmentStatusReporter.java:91)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:367)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:219)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:330)
> at
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
[jira] [Updated] (DRILL-6947) RuntimeFilter memory leak due to BF ByteBuf ownership transferring
[ https://issues.apache.org/jira/browse/DRILL-6947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sorabh Hamirwasia updated DRILL-6947: - Affects Version/s: 1.16.0 > RuntimeFilter memory leak due to BF ByteBuf ownership transferring > --- > > Key: DRILL-6947 > URL: https://issues.apache.org/jira/browse/DRILL-6947 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.16.0 >Reporter: weijie.tong >Assignee: weijie.tong >Priority: Major > Labels: ready-to-commit > Fix For: 1.16.0 > > > RuntimeFilter's BF ByteBuf ownership should be transferred right at broadcast > and random hash cases. Currently due to we not treat this transferring > reasonable, it caused the memory leak. > To broadcast case,the HashJoin operator's allocator allocated the BF, the > allocated BF's ownership should be transferred to its receiver : the > FragmentContextImpl or the final RuntimeFilter operator. Otherwise, the > OperatorContextImpl's close method will complain about the memory leak when > closing the corresponding allocator. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6947) RuntimeFilter memory leak due to BF ByteBuf ownership transferring
[ https://issues.apache.org/jira/browse/DRILL-6947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sorabh Hamirwasia updated DRILL-6947: - Labels: ready-to-commit (was: ) > RuntimeFilter memory leak due to BF ByteBuf ownership transferring > --- > > Key: DRILL-6947 > URL: https://issues.apache.org/jira/browse/DRILL-6947 > Project: Apache Drill > Issue Type: Bug >Reporter: weijie.tong >Assignee: weijie.tong >Priority: Major > Labels: ready-to-commit > Fix For: 1.16.0 > > > RuntimeFilter's BF ByteBuf ownership should be transferred right at broadcast > and random hash cases. Currently due to we not treat this transferring > reasonable, it caused the memory leak. > To broadcast case,the HashJoin operator's allocator allocated the BF, the > allocated BF's ownership should be transferred to its receiver : the > FragmentContextImpl or the final RuntimeFilter operator. Otherwise, the > OperatorContextImpl's close method will complain about the memory leak when > closing the corresponding allocator. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6999) Queries failed with "Failed to aggregate or route the RFW" due to "java.lang.ArrayIndexOutOfBoundsException"
[
https://issues.apache.org/jira/browse/DRILL-6999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sorabh Hamirwasia updated DRILL-6999:
-
Reviewer: Sorabh Hamirwasia
> Queries failed with "Failed to aggregate or route the RFW" due to
> "java.lang.ArrayIndexOutOfBoundsException"
>
>
> Key: DRILL-6999
> URL: https://issues.apache.org/jira/browse/DRILL-6999
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.16.0
>Reporter: Sorabh Hamirwasia
>Assignee: weijie.tong
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.16.0
>
>
> Following query joins 2 tables on *two* (>1) fields.
> {noformat}
> select count(*) from lineitem l inner join partsupp p on l.l_partkey =
> p.ps_partkey AND l.l_suppkey = p.ps_suppkey
> {noformat}
> This is because AsyncAggregateWorker exits due to the following exception
> when multiple fields are used in the join condition
> {code:java}
> 2019-01-22 16:01:18,773 [drill-executor-1301] ERROR
> o.a.d.e.w.filter.RuntimeFilterSink - Failed to aggregate or route the RFW
> java.lang.ArrayIndexOutOfBoundsException: 1
> at
> org.apache.drill.exec.work.filter.RuntimeFilterWritable.unwrap(RuntimeFilterWritable.java:67)
> ~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.filter.RuntimeFilterWritable.aggregate(RuntimeFilterWritable.java:78)
> ~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.filter.RuntimeFilterSink.aggregate(RuntimeFilterSink.java:140)
> ~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.filter.RuntimeFilterSink.access$600(RuntimeFilterSink.java:52)
> ~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.filter.RuntimeFilterSink$AsyncAggregateWorker.run(RuntimeFilterSink.java:246)
> ~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_151]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [na:1.8.0_151]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [na:1.8.0_151]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [na:1.8.0_151]
> at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151]
> {code}
> Hit the issue with latest changes in the PR ->
> https://github.com/apache/drill/pull/1600
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6998) Queries failing with "Failed to aggregate or route the RFW" due to "java.lang.ArrayIndexOutOfBoundsException" do not complete
[
https://issues.apache.org/jira/browse/DRILL-6998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sorabh Hamirwasia updated DRILL-6998:
-
Labels: ready-to-commit (was: )
> Queries failing with "Failed to aggregate or route the RFW" due to
> "java.lang.ArrayIndexOutOfBoundsException" do not complete
> -
>
> Key: DRILL-6998
> URL: https://issues.apache.org/jira/browse/DRILL-6998
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.16.0
>Reporter: Abhishek Ravi
>Assignee: weijie.tong
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.16.0
>
>
> Following query joins 2 tables on *two* (>1) fields.
> {noformat}
> select count(*) from lineitem l inner join partsupp p on l.l_partkey =
> p.ps_partkey AND l.l_suppkey = p.ps_suppkey
> {noformat}
> The query does not return even though Fragment 0:0 reports a state change
> from {{RUNNING}} -> {{FINISHED}}
> Following is the jstack output of the {{Frag0:0}}.
> {noformat}
> "23b85137-b102-39a9-70d9-72381c5fb93b:frag:0:0" #16037 daemon prio=10
> os_prio=0 tid=0x7f5f48d415d0 nid=0x1a61 waiting on condition
> [0x7f61b32b2000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.drill.exec.work.filter.RuntimeFilterSink.close(RuntimeFilterSink.java:116)
> at
> org.apache.drill.exec.work.filter.RuntimeFilterRouter.waitForComplete(RuntimeFilterRouter.java:113)
> at
> org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:738)
> at
> org.apache.drill.exec.work.foreman.QueryStateProcessor.wrapUpCompletion(QueryStateProcessor.java:315)
> at
> org.apache.drill.exec.work.foreman.QueryStateProcessor.running(QueryStateProcessor.java:276)
> at
> org.apache.drill.exec.work.foreman.QueryStateProcessor.moveToState(QueryStateProcessor.java:92)
> - locked <0x00055f9a7468> (a
> org.apache.drill.exec.work.foreman.QueryStateProcessor)
> at
> org.apache.drill.exec.work.foreman.QueryStateProcessor$StateSwitch.processEvent(QueryStateProcessor.java:349)
> at
> org.apache.drill.exec.work.foreman.QueryStateProcessor$StateSwitch.processEvent(QueryStateProcessor.java:342)
> at
> org.apache.drill.common.EventProcessor.processEvents(EventProcessor.java:107)
> at
> org.apache.drill.common.EventProcessor.sendEvent(EventProcessor.java:65)
> at
> org.apache.drill.exec.work.foreman.QueryStateProcessor$StateSwitch.addEvent(QueryStateProcessor.java:344)
> at
> org.apache.drill.exec.work.foreman.QueryStateProcessor.addToEventQueue(QueryStateProcessor.java:155)
> at
> org.apache.drill.exec.work.foreman.Foreman.addToEventQueue(Foreman.java:213)
> at
> org.apache.drill.exec.work.foreman.QueryManager.nodeComplete(QueryManager.java:519)
> at
> org.apache.drill.exec.work.foreman.QueryManager.access$100(QueryManager.java:65)
> at
> org.apache.drill.exec.work.foreman.QueryManager$NodeTracker.fragmentComplete(QueryManager.java:483)
> at
> org.apache.drill.exec.work.foreman.QueryManager.fragmentDone(QueryManager.java:155)
> at
> org.apache.drill.exec.work.foreman.QueryManager.access$400(QueryManager.java:65)
> at
> org.apache.drill.exec.work.foreman.QueryManager$1.statusUpdate(QueryManager.java:546)
> at
> org.apache.drill.exec.rpc.control.WorkEventBus.statusUpdate(WorkEventBus.java:63)
> at
> org.apache.drill.exec.work.batch.ControlMessageHandler.requestFragmentStatus(ControlMessageHandler.java:253)
> at
> org.apache.drill.exec.rpc.control.LocalControlConnectionManager.runCommand(LocalControlConnectionManager.java:130)
> at
> org.apache.drill.exec.rpc.control.ControlTunnel.sendFragmentStatus(ControlTunnel.java:89)
> at
> org.apache.drill.exec.work.fragment.FragmentStatusReporter.sendStatus(FragmentStatusReporter.java:122)
> at
> org.apache.drill.exec.work.fragment.FragmentStatusReporter.stateChanged(FragmentStatusReporter.java:91)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:367)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:219)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:330)
> at
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624
[jira] [Updated] (DRILL-6999) Queries failed with "Failed to aggregate or route the RFW" due to "java.lang.ArrayIndexOutOfBoundsException"
[
https://issues.apache.org/jira/browse/DRILL-6999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sorabh Hamirwasia updated DRILL-6999:
-
Labels: ready-to-commit (was: )
> Queries failed with "Failed to aggregate or route the RFW" due to
> "java.lang.ArrayIndexOutOfBoundsException"
>
>
> Key: DRILL-6999
> URL: https://issues.apache.org/jira/browse/DRILL-6999
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.16.0
>Reporter: Sorabh Hamirwasia
>Assignee: weijie.tong
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.16.0
>
>
> Following query joins 2 tables on *two* (>1) fields.
> {noformat}
> select count(*) from lineitem l inner join partsupp p on l.l_partkey =
> p.ps_partkey AND l.l_suppkey = p.ps_suppkey
> {noformat}
> This is because AsyncAggregateWorker exits due to the following exception
> when multiple fields are used in the join condition
> {code:java}
> 2019-01-22 16:01:18,773 [drill-executor-1301] ERROR
> o.a.d.e.w.filter.RuntimeFilterSink - Failed to aggregate or route the RFW
> java.lang.ArrayIndexOutOfBoundsException: 1
> at
> org.apache.drill.exec.work.filter.RuntimeFilterWritable.unwrap(RuntimeFilterWritable.java:67)
> ~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.filter.RuntimeFilterWritable.aggregate(RuntimeFilterWritable.java:78)
> ~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.filter.RuntimeFilterSink.aggregate(RuntimeFilterSink.java:140)
> ~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.filter.RuntimeFilterSink.access$600(RuntimeFilterSink.java:52)
> ~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.filter.RuntimeFilterSink$AsyncAggregateWorker.run(RuntimeFilterSink.java:246)
> ~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_151]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [na:1.8.0_151]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [na:1.8.0_151]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [na:1.8.0_151]
> at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151]
> {code}
> Hit the issue with latest changes in the PR ->
> https://github.com/apache/drill/pull/1600
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-7000) Queries failing with "Failed to aggregate or route the RFW" do not complete
[
https://issues.apache.org/jira/browse/DRILL-7000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sorabh Hamirwasia updated DRILL-7000:
-
Description:
The query does not return even though Fragment 0:0 reports a state change from
RUNNING -> FINISHED
{code:java}
Following is the jstack output of the Frag0:0.
"23b85137-b102-39a9-70d9-72381c5fb93b:frag:0:0" #16037 daemon prio=10 os_prio=0
tid=0x7f5f48d415d0 nid=0x1a61 waiting on condition [0x7f61b32b2000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.drill.exec.work.filter.RuntimeFilterSink.close(RuntimeFilterSink.java:116)
at
org.apache.drill.exec.work.filter.RuntimeFilterRouter.waitForComplete(RuntimeFilterRouter.java:113)
at
org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:738)
at
org.apache.drill.exec.work.foreman.QueryStateProcessor.wrapUpCompletion(QueryStateProcessor.java:315)
at
org.apache.drill.exec.work.foreman.QueryStateProcessor.running(QueryStateProcessor.java:276)
at
org.apache.drill.exec.work.foreman.QueryStateProcessor.moveToState(QueryStateProcessor.java:92)
locked <0x00055f9a7468> (a
org.apache.drill.exec.work.foreman.QueryStateProcessor)
at
org.apache.drill.exec.work.foreman.QueryStateProcessor$StateSwitch.processEvent(QueryStateProcessor.java:349)
at
org.apache.drill.exec.work.foreman.QueryStateProcessor$StateSwitch.processEvent(QueryStateProcessor.java:342)
at
org.apache.drill.common.EventProcessor.processEvents(EventProcessor.java:107)
at org.apache.drill.common.EventProcessor.sendEvent(EventProcessor.java:65)
at
org.apache.drill.exec.work.foreman.QueryStateProcessor$StateSwitch.addEvent(QueryStateProcessor.java:344)
at
org.apache.drill.exec.work.foreman.QueryStateProcessor.addToEventQueue(QueryStateProcessor.java:155)
at org.apache.drill.exec.work.foreman.Foreman.addToEventQueue(Foreman.java:213)
at
org.apache.drill.exec.work.foreman.QueryManager.nodeComplete(QueryManager.java:519)
at
org.apache.drill.exec.work.foreman.QueryManager.access$100(QueryManager.java:65)
at
org.apache.drill.exec.work.foreman.QueryManager$NodeTracker.fragmentComplete(QueryManager.java:483)
at
org.apache.drill.exec.work.foreman.QueryManager.fragmentDone(QueryManager.java:155)
at
org.apache.drill.exec.work.foreman.QueryManager.access$400(QueryManager.java:65)
at
org.apache.drill.exec.work.foreman.QueryManager$1.statusUpdate(QueryManager.java:546)
at
org.apache.drill.exec.rpc.control.WorkEventBus.statusUpdate(WorkEventBus.java:63)
at
org.apache.drill.exec.work.batch.ControlMessageHandler.requestFragmentStatus(ControlMessageHandler.java:253)
at
org.apache.drill.exec.rpc.control.LocalControlConnectionManager.runCommand(LocalControlConnectionManager.java:130)
at
org.apache.drill.exec.rpc.control.ControlTunnel.sendFragmentStatus(ControlTunnel.java:89)
at
org.apache.drill.exec.work.fragment.FragmentStatusReporter.sendStatus(FragmentStatusReporter.java:122)
at
org.apache.drill.exec.work.fragment.FragmentStatusReporter.stateChanged(FragmentStatusReporter.java:91)
at
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:367)
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:219)
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:330)
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748) {code}
- From the code, it seems that RuntimeFilterSink.close is stuck at
{code:java}
while (!asyncAggregateWorker.over.get()) {
try
{ Thread.sleep(100); }
catch (InterruptedException e)
{ logger.error("interrupted while sleeping to wait for the aggregating worker
thread to exit", e); }
}
{code}
This is because AsyncAggregateWorker exits due to the following exception,
before it could set asyncAggregateWorker.over is set to false.
{code:java}
2019-01-22 16:01:18,773 [drill-executor-1301] ERROR
o.a.d.e.w.filter.RuntimeFilterSink - Failed to aggregate or route the RFW
java.lang.ArrayIndexOutOfBoundsException: 1
at
org.apache.drill.exec.work.filter.RuntimeFilterWritable.unwrap(RuntimeFilterWritable.java:67)
~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
at
org.apache.drill.exec.work.filter.RuntimeFilterWritable.aggregate(RuntimeFilterWritable.java:78)
~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
at
org.apache.drill.exec.work.filter.RuntimeFilterSink.aggregate(RuntimeFilterSink.java:140)
~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
at
org.apache.drill.exec.work.filter.RuntimeFilterSink.access$600(RuntimeFilterSink.java:52)
~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
[jira] [Updated] (DRILL-7000) Queries failing with "Failed to aggregate or route the RFW" do not complete
[
https://issues.apache.org/jira/browse/DRILL-7000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sorabh Hamirwasia updated DRILL-7000:
-
Labels: ready-to-commit (was: )
> Queries failing with "Failed to aggregate or route the RFW" do not complete
> ---
>
> Key: DRILL-7000
> URL: https://issues.apache.org/jira/browse/DRILL-7000
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.16.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sorabh Hamirwasia
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.16.0
>
>
> The query does not return even though Fragment 0:0 reports a state change
> from RUNNING -> FINISHED
>
> {code:java}
> Following is the jstack output of the Frag0:0.
> "23b85137-b102-39a9-70d9-72381c5fb93b:frag:0:0" #16037 daemon prio=10
> os_prio=0 tid=0x7f5f48d415d0 nid=0x1a61 waiting on condition
> [0x7f61b32b2000]
> java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.drill.exec.work.filter.RuntimeFilterSink.close(RuntimeFilterSink.java:116)
> at
> org.apache.drill.exec.work.filter.RuntimeFilterRouter.waitForComplete(RuntimeFilterRouter.java:113)
> at
> org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:738)
> at
> org.apache.drill.exec.work.foreman.QueryStateProcessor.wrapUpCompletion(QueryStateProcessor.java:315)
> at
> org.apache.drill.exec.work.foreman.QueryStateProcessor.running(QueryStateProcessor.java:276)
> at
> org.apache.drill.exec.work.foreman.QueryStateProcessor.moveToState(QueryStateProcessor.java:92)
> locked <0x00055f9a7468> (a
> org.apache.drill.exec.work.foreman.QueryStateProcessor)
> at
> org.apache.drill.exec.work.foreman.QueryStateProcessor$StateSwitch.processEvent(QueryStateProcessor.java:349)
> at
> org.apache.drill.exec.work.foreman.QueryStateProcessor$StateSwitch.processEvent(QueryStateProcessor.java:342)
> at
> org.apache.drill.common.EventProcessor.processEvents(EventProcessor.java:107)
> at org.apache.drill.common.EventProcessor.sendEvent(EventProcessor.java:65)
> at
> org.apache.drill.exec.work.foreman.QueryStateProcessor$StateSwitch.addEvent(QueryStateProcessor.java:344)
> at
> org.apache.drill.exec.work.foreman.QueryStateProcessor.addToEventQueue(QueryStateProcessor.java:155)
> at
> org.apache.drill.exec.work.foreman.Foreman.addToEventQueue(Foreman.java:213)
> at
> org.apache.drill.exec.work.foreman.QueryManager.nodeComplete(QueryManager.java:519)
> at
> org.apache.drill.exec.work.foreman.QueryManager.access$100(QueryManager.java:65)
> at
> org.apache.drill.exec.work.foreman.QueryManager$NodeTracker.fragmentComplete(QueryManager.java:483)
> at
> org.apache.drill.exec.work.foreman.QueryManager.fragmentDone(QueryManager.java:155)
> at
> org.apache.drill.exec.work.foreman.QueryManager.access$400(QueryManager.java:65)
> at
> org.apache.drill.exec.work.foreman.QueryManager$1.statusUpdate(QueryManager.java:546)
> at
> org.apache.drill.exec.rpc.control.WorkEventBus.statusUpdate(WorkEventBus.java:63)
> at
> org.apache.drill.exec.work.batch.ControlMessageHandler.requestFragmentStatus(ControlMessageHandler.java:253)
> at
> org.apache.drill.exec.rpc.control.LocalControlConnectionManager.runCommand(LocalControlConnectionManager.java:130)
> at
> org.apache.drill.exec.rpc.control.ControlTunnel.sendFragmentStatus(ControlTunnel.java:89)
> at
> org.apache.drill.exec.work.fragment.FragmentStatusReporter.sendStatus(FragmentStatusReporter.java:122)
> at
> org.apache.drill.exec.work.fragment.FragmentStatusReporter.stateChanged(FragmentStatusReporter.java:91)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:367)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:219)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:330)
> at
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748) {code}
> - From the code, it seems that RuntimeFilterSink.close is stuck at
>
> {code:java}
> while (!asyncAggregateWorker.over.get()) {
> try
> { Thread.sleep(100); }
> catch (InterruptedException e)
> { logger.error("interrupted while sleeping to wait for the aggregating worker
> thread to exit", e); }
> }
> {code}
> This is because AsyncAggregateWorker exits due to the following exception,
> before it could set asyncAggregateWorker.over is set to false.
> {code:java}
> 2019-01-22
[jira] [Created] (DRILL-6999) Queries failed with "Failed to aggregate or route the RFW" due to "java.lang.ArrayIndexOutOfBoundsException"
Sorabh Hamirwasia created DRILL-6999:
Summary: Queries failed with "Failed to aggregate or route the
RFW" due to "java.lang.ArrayIndexOutOfBoundsException"
Key: DRILL-6999
URL: https://issues.apache.org/jira/browse/DRILL-6999
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.16.0
Reporter: Sorabh Hamirwasia
Assignee: weijie.tong
Fix For: 1.16.0
Following query joins 2 tables on *two* (>1) fields.
{noformat}
select count(*) from lineitem l inner join partsupp p on l.l_partkey =
p.ps_partkey AND l.l_suppkey = p.ps_suppkey
{noformat}
This is because AsyncAggregateWorker exits due to the following exception when
multiple fields are used in the join condition
{code:java}
2019-01-22 16:01:18,773 [drill-executor-1301] ERROR
o.a.d.e.w.filter.RuntimeFilterSink - Failed to aggregate or route the RFW
java.lang.ArrayIndexOutOfBoundsException: 1
at
org.apache.drill.exec.work.filter.RuntimeFilterWritable.unwrap(RuntimeFilterWritable.java:67)
~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
at
org.apache.drill.exec.work.filter.RuntimeFilterWritable.aggregate(RuntimeFilterWritable.java:78)
~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
at
org.apache.drill.exec.work.filter.RuntimeFilterSink.aggregate(RuntimeFilterSink.java:140)
~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
at
org.apache.drill.exec.work.filter.RuntimeFilterSink.access$600(RuntimeFilterSink.java:52)
~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
at
org.apache.drill.exec.work.filter.RuntimeFilterSink$AsyncAggregateWorker.run(RuntimeFilterSink.java:246)
~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_151]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[na:1.8.0_151]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[na:1.8.0_151]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[na:1.8.0_151]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151]
{code}
Hit the issue with latest changes in the PR ->
https://github.com/apache/drill/pull/1600
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6998) Queries failing with "Failed to aggregate or route the RFW" due to "java.lang.ArrayIndexOutOfBoundsException" do not complete
[
https://issues.apache.org/jira/browse/DRILL-6998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16750598#comment-16750598
]
Sorabh Hamirwasia commented on DRILL-6998:
--
There are 2 issues here:
1) IOB when multiple fields is used in join condition.
2) Foreman thread deadlocks when aggregator thread hits an exception while
aggregation. For this I looked into it more and just setting over.set(true) in
catch block will not be sufficient. Since with that the producer thread from
Netty layer might have already produced a RFW in the queue which will be leaked.
I will open 2 separate child JIRA's for these issues and post a PR for the
second case.
> Queries failing with "Failed to aggregate or route the RFW" due to
> "java.lang.ArrayIndexOutOfBoundsException" do not complete
> -
>
> Key: DRILL-6998
> URL: https://issues.apache.org/jira/browse/DRILL-6998
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.16.0
>Reporter: Abhishek Ravi
>Assignee: weijie.tong
>Priority: Major
> Fix For: 1.16.0
>
>
> Following query joins 2 tables on *two* (>1) fields.
> {noformat}
> select count(*) from lineitem l inner join partsupp p on l.l_partkey =
> p.ps_partkey AND l.l_suppkey = p.ps_suppkey
> {noformat}
> The query does not return even though Fragment 0:0 reports a state change
> from {{RUNNING}} -> {{FINISHED}}
> Following is the jstack output of the {{Frag0:0}}.
> {noformat}
> "23b85137-b102-39a9-70d9-72381c5fb93b:frag:0:0" #16037 daemon prio=10
> os_prio=0 tid=0x7f5f48d415d0 nid=0x1a61 waiting on condition
> [0x7f61b32b2000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.drill.exec.work.filter.RuntimeFilterSink.close(RuntimeFilterSink.java:116)
> at
> org.apache.drill.exec.work.filter.RuntimeFilterRouter.waitForComplete(RuntimeFilterRouter.java:113)
> at
> org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:738)
> at
> org.apache.drill.exec.work.foreman.QueryStateProcessor.wrapUpCompletion(QueryStateProcessor.java:315)
> at
> org.apache.drill.exec.work.foreman.QueryStateProcessor.running(QueryStateProcessor.java:276)
> at
> org.apache.drill.exec.work.foreman.QueryStateProcessor.moveToState(QueryStateProcessor.java:92)
> - locked <0x00055f9a7468> (a
> org.apache.drill.exec.work.foreman.QueryStateProcessor)
> at
> org.apache.drill.exec.work.foreman.QueryStateProcessor$StateSwitch.processEvent(QueryStateProcessor.java:349)
> at
> org.apache.drill.exec.work.foreman.QueryStateProcessor$StateSwitch.processEvent(QueryStateProcessor.java:342)
> at
> org.apache.drill.common.EventProcessor.processEvents(EventProcessor.java:107)
> at
> org.apache.drill.common.EventProcessor.sendEvent(EventProcessor.java:65)
> at
> org.apache.drill.exec.work.foreman.QueryStateProcessor$StateSwitch.addEvent(QueryStateProcessor.java:344)
> at
> org.apache.drill.exec.work.foreman.QueryStateProcessor.addToEventQueue(QueryStateProcessor.java:155)
> at
> org.apache.drill.exec.work.foreman.Foreman.addToEventQueue(Foreman.java:213)
> at
> org.apache.drill.exec.work.foreman.QueryManager.nodeComplete(QueryManager.java:519)
> at
> org.apache.drill.exec.work.foreman.QueryManager.access$100(QueryManager.java:65)
> at
> org.apache.drill.exec.work.foreman.QueryManager$NodeTracker.fragmentComplete(QueryManager.java:483)
> at
> org.apache.drill.exec.work.foreman.QueryManager.fragmentDone(QueryManager.java:155)
> at
> org.apache.drill.exec.work.foreman.QueryManager.access$400(QueryManager.java:65)
> at
> org.apache.drill.exec.work.foreman.QueryManager$1.statusUpdate(QueryManager.java:546)
> at
> org.apache.drill.exec.rpc.control.WorkEventBus.statusUpdate(WorkEventBus.java:63)
> at
> org.apache.drill.exec.work.batch.ControlMessageHandler.requestFragmentStatus(ControlMessageHandler.java:253)
> at
> org.apache.drill.exec.rpc.control.LocalControlConnectionManager.runCommand(LocalControlConnectionManager.java:130)
> at
> org.apache.drill.exec.rpc.control.ControlTunnel.sendFragmentStatus(ControlTunnel.java:89)
> at
> org.apache.drill.exec.work.fragment.FragmentStatusReporter.sendStatus(FragmentStatusReporter.java:122)
> at
> org.apache.drill.exec.work.fragment.FragmentStatusReporter.stateChanged(FragmentStatusReporter.java:91)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:367)
> at
> org.
[jira] [Created] (DRILL-7000) Queries failing with "Failed to aggregate or route the RFW" do not complete
Sorabh Hamirwasia created DRILL-7000:
Summary: Queries failing with "Failed to aggregate or route the
RFW" do not complete
Key: DRILL-7000
URL: https://issues.apache.org/jira/browse/DRILL-7000
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.16.0
Reporter: Sorabh Hamirwasia
Assignee: Sorabh Hamirwasia
Fix For: 1.16.0
The query does not return even though Fragment 0:0 reports a state change from
RUNNING -> FINISHED
{code:java}
Following is the jstack output of the Frag0:0.
"23b85137-b102-39a9-70d9-72381c5fb93b:frag:0:0" #16037 daemon prio=10 os_prio=0
tid=0x7f5f48d415d0 nid=0x1a61 waiting on condition [0x7f61b32b2000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.drill.exec.work.filter.RuntimeFilterSink.close(RuntimeFilterSink.java:116)
at
org.apache.drill.exec.work.filter.RuntimeFilterRouter.waitForComplete(RuntimeFilterRouter.java:113)
at
org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:738)
at
org.apache.drill.exec.work.foreman.QueryStateProcessor.wrapUpCompletion(QueryStateProcessor.java:315)
at
org.apache.drill.exec.work.foreman.QueryStateProcessor.running(QueryStateProcessor.java:276)
at
org.apache.drill.exec.work.foreman.QueryStateProcessor.moveToState(QueryStateProcessor.java:92)
locked <0x00055f9a7468> (a
org.apache.drill.exec.work.foreman.QueryStateProcessor)
at
org.apache.drill.exec.work.foreman.QueryStateProcessor$StateSwitch.processEvent(QueryStateProcessor.java:349)
at
org.apache.drill.exec.work.foreman.QueryStateProcessor$StateSwitch.processEvent(QueryStateProcessor.java:342)
at
org.apache.drill.common.EventProcessor.processEvents(EventProcessor.java:107)
at org.apache.drill.common.EventProcessor.sendEvent(EventProcessor.java:65)
at
org.apache.drill.exec.work.foreman.QueryStateProcessor$StateSwitch.addEvent(QueryStateProcessor.java:344)
at
org.apache.drill.exec.work.foreman.QueryStateProcessor.addToEventQueue(QueryStateProcessor.java:155)
at org.apache.drill.exec.work.foreman.Foreman.addToEventQueue(Foreman.java:213)
at
org.apache.drill.exec.work.foreman.QueryManager.nodeComplete(QueryManager.java:519)
at
org.apache.drill.exec.work.foreman.QueryManager.access$100(QueryManager.java:65)
at
org.apache.drill.exec.work.foreman.QueryManager$NodeTracker.fragmentComplete(QueryManager.java:483)
at
org.apache.drill.exec.work.foreman.QueryManager.fragmentDone(QueryManager.java:155)
at
org.apache.drill.exec.work.foreman.QueryManager.access$400(QueryManager.java:65)
at
org.apache.drill.exec.work.foreman.QueryManager$1.statusUpdate(QueryManager.java:546)
at
org.apache.drill.exec.rpc.control.WorkEventBus.statusUpdate(WorkEventBus.java:63)
at
org.apache.drill.exec.work.batch.ControlMessageHandler.requestFragmentStatus(ControlMessageHandler.java:253)
at
org.apache.drill.exec.rpc.control.LocalControlConnectionManager.runCommand(LocalControlConnectionManager.java:130)
at
org.apache.drill.exec.rpc.control.ControlTunnel.sendFragmentStatus(ControlTunnel.java:89)
at
org.apache.drill.exec.work.fragment.FragmentStatusReporter.sendStatus(FragmentStatusReporter.java:122)
at
org.apache.drill.exec.work.fragment.FragmentStatusReporter.stateChanged(FragmentStatusReporter.java:91)
at
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:367)
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:219)
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:330)
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748) {code}
- From the code, it seems that RuntimeFilterSink.close is stuck at
{code:java}
while (!asyncAggregateWorker.over.get()) {
try
{ Thread.sleep(100); }
catch (InterruptedException e)
{ logger.error("interrupted while sleeping to wait for the aggregating worker
thread to exit", e); }
}
{code}
This is because AsyncAggregateWorker exits due to the following exception,
before it could set asyncAggregateWorker.over is set to false.
{code:java}
2019-01-22 16:01:18,773 [drill-executor-1301] ERROR
o.a.d.e.w.filter.RuntimeFilterSink - Failed to aggregate or route the RFW
java.lang.ArrayIndexOutOfBoundsException: 1
at
org.apache.drill.exec.work.filter.RuntimeFilterWritable.unwrap(RuntimeFilterWritable.java:67)
~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
at
org.apache.drill.exec.work.filter.RuntimeFilterWritable.aggregate(RuntimeFilterWritable.java:78)
~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
a
[jira] [Commented] (DRILL-6827) Apache Drill 1.14 on a kerberized Cloudera cluster (CDH 5.14).
[
https://issues.apache.org/jira/browse/DRILL-6827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16749564#comment-16749564
]
Sorabh Hamirwasia commented on DRILL-6827:
--
Your configurations is incorrect. Please refer to documentation here for
WebServer configuration:
https://drill.apache.org/docs/configuring-drill-to-use-spnego-for-http-authentication/
Also I am seeing that you have both ssl and sasl encryption enabled which is an
overkill.
{code:java}
user.encryption.sasl.enabled: true, user.encryption.sasl.max_wrapped_size:
65536 }
,
security.user.encryption.ssl:
{ enabled: true, keyPassword: "X", handshakeTimeout: 1,
provider: "JDK" }
,
ssl:
{ keyStorePath: "X", keyStorePassword: "X", trustStorePath:
"X", trustStorePassword: "X" }
{code}
> Apache Drill 1.14 on a kerberized Cloudera cluster (CDH 5.14).
> --
>
> Key: DRILL-6827
> URL: https://issues.apache.org/jira/browse/DRILL-6827
> Project: Apache Drill
> Issue Type: Bug
> Components: Security
>Affects Versions: 1.14.0
> Environment: * Apache Drill 1.14
> * Cloudera CDH 5.14
>Reporter: Ibrahim Safieddine
>Priority: Critical
>
> Hello,
>
> I'am using apache Drill 1.14 on a kerberized Cloudera cluster (CDH 5.14).
>
> When I activate kerberos authentification, drill server refuse to start with
> error:
> {color:#ff}_org.apache.drill.exec.exception.DrillbitStartupException:
> Authentication is enabled for WebServer but none of the security mechanism
> was configured properly. Please verify the configurations and try
> again._{color}
>
> I can see in the logs that the kerberos authentification is ok:
> [main] INFO o.a.d.exec.server.BootStrapContext - Process user name: 'root'
> and logged in successfully as 'tata/xx.yy...@xx.yy'
>
> Can you help me please?
>
> Based on the Apache Drill documentation, there is my conf/drill-override.conf:
>
> drill.exec: {
> cluster-id: "drillbits1",
> zk.connect: "xx.yy.zz:2181",
> service_name: "service1",
> impersonation: {
> enabled: true,
> max_chained_user_hops: 3
> },
> security: {
> user.auth.enabled:true,
> auth.mechanisms:["KERBEROS"],
> auth.principal:"tata/xx.yy...@xx.yy",
> auth.keytab:"keytab1.keytab",
> drill.exec.security.auth.auth_to_local:hive,
> auth.realm: "XX.YY",
> user.encryption.sasl.enabled: true,
> user.encryption.sasl.max_wrapped_size: 65536
> },
> security.user.encryption.ssl: {
> enabled: true,
> keyPassword: "X",
> handshakeTimeout: 1,
> provider: "JDK"
> },
> ssl: {
> keyStorePath: "X",
> keyStorePassword: "X",
> trustStorePath: "X",
> trustStorePassword: "X"
> },
> http: {
> enabled: true,
> auth.enabled: false,
> auth.mechanisms: ["KERBEROS"],
> ssl_enabled: true,
> port: 8047
> session_max_idle_secs: 3600, # Default value 1hr
> cors: {
> enabled: false,
> allowedOrigins: ["null"],
> allowedMethods: ["GET", "POST", "HEAD", "OPTIONS"],
> allowedHeaders: ["X-Requested-With", "Content-Type", "Accept",
> "Origin"],
> credentials: true
> }
> }
> }
> Thank you
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6991) Kerberos ticket is being dumped in the log if log level is "debug" for stdout
[
https://issues.apache.org/jira/browse/DRILL-6991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16749039#comment-16749039
]
Sorabh Hamirwasia commented on DRILL-6991:
--
Why is this a problem ? TGT is encrypted with TGS secret key so only TGS can
decrypt it.
> Kerberos ticket is being dumped in the log if log level is "debug" for stdout
> --
>
> Key: DRILL-6991
> URL: https://issues.apache.org/jira/browse/DRILL-6991
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.15.0
>Reporter: Anton Gozhiy
>Priority: Major
>
> *Prerequisites:*
> # Drill is installed on cluster with Kerberos security
> # Into conf/logback.xml, set the following log level:
> {code:xml}
>
>
>
>
> {code}
> *Steps:*
> # Start Drill
> # Connect using sqlline using the following string:
> {noformat}
> bin/sqlline -u "jdbc:drill:zk=;principal="
> {noformat}
> *Expected result:*
> No sensitive information should be displayed
> *Actual result:*
> Kerberos ticket and session key are being dumped into console output:
> {noformat}
> 14:35:38.806 [TGT Renewer for mapr/node1.cluster.com@NODE1] DEBUG
> o.a.h.security.UserGroupInformation - Found tgt Ticket (hex) =
> : 61 82 01 3D 30 82 01 39 A0 03 02 01 05 A1 07 1B a..=0..9
> 0010: 05 4E 4F 44 45 31 A2 1A 30 18 A0 03 02 01 02 A1 .NODE1..0...
> 0020: 11 30 0F 1B 06 6B 72 62 74 67 74 1B 05 4E 4F 44 .0...krbtgt..NOD
> 0030: 45 31 A3 82 01 0B 30 82 01 07 A0 03 02 01 12 A1 E10.
> 0040: 03 02 01 01 A2 81 FA 04 81 F7 03 8D A9 FA 7D 89
> 0050: 1B DF 37 B7 4D E6 6C 99 3E 8F FA 48 D9 9A 79 F3 ..7.M.l.>..H..y.
> 0060: 92 34 7F BF 67 1E 77 4A 2F C9 AF 82 93 4E 46 1D .4..g.wJ/NF.
> 0070: 41 74 B0 AF 41 A8 8B 02 71 83 CC 14 51 72 60 EE At..A...q...Qr`.
> 0080: 29 67 14 F0 A6 33 63 07 41 AA 8D DC 7B 5B 41 F3 )g...3c.A[A.
> 0090: 83 48 8B 2A 0B 4D 6D 57 9A 6E CF 6B DC 0B C0 D1 .H.*.MmW.n.k
> 00A0: 83 BB 27 40 88 7E 9F 2B D1 FD A8 6A E1 BF F6 CC ..'@...+...j
> 00B0: 0E 0C FB 93 5D 69 9A 8B 11 88 0C F2 7C E1 FD 04 ]i..
> 00C0: F5 AB 66 0C A4 A4 7B 30 D1 7F F1 2D D6 A1 52 D1 ..f0...-..R.
> 00D0: 79 59 F2 06 CB 65 FB 73 63 1D 5B E9 4F 28 73 EB yY...e.sc.[.O(s.
> 00E0: 72 7F 04 46 34 56 F4 40 6C C0 2C 39 C0 5B C6 25 r..F4V.@l.,9.[.%
> 00F0: ED EF 64 07 CE ED 35 9D D7 91 6C 8F C9 CE 16 F5 ..d...5...l.
> 0100: CA 5E 6F DE 08 D2 68 30 C7 03 97 E7 C0 FF D9 52 .^o...h0...R
> 0110: F8 1D 2F DB 63 6D 12 4A CD 60 AD D0 BA FA 4B CF ../.cm.J.`K.
> 0120: 2C B9 8C CA 5A E6 EC 10 5A 0A 1F 84 B0 80 BD 39 ,...Z...Z..9
> 0130: 42 2C 33 EB C0 AA 0D 44 F0 F4 E9 87 24 43 BB 9A B,3D$C..
> 0140: 52 R
> Client Principal = mapr/node1.cluster.com@NODE1
> Server Principal = krbtgt/NODE1@NODE1
> Session Key = EncryptionKey: keyType=18 keyBytes (hex dump)=
> : 50 DA D1 D7 91 D3 64 BE 45 7B D8 02 25 81 18 25 P.d.E...%..%
> 0010: DA 59 4F BA 76 67 BB 39 9C F7 17 46 A7 C5 00 E2 .YO.vg.9...F
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6932) Drill can't submit Physical plan returned by submitting Logical plan
[
https://issues.apache.org/jira/browse/DRILL-6932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16742537#comment-16742537
]
Sorabh Hamirwasia commented on DRILL-6932:
--
The problem is that the id for operators after getting physical plan in step 3
above is all 0. Whereas if we directly get a physical plan for query then id
for each operator is unique. Also I see the physical plan generated using above
steps and directly from sql query is different, as there is no Project operator
in former.
*Physical Operator from original query:*
{code:java}
{
"head" : {
"version" : 1,
"generator" : {
"type" : "ExplainHandler",
"info" : ""
},
"type" : "APACHE_DRILL_PHYSICAL",
"options" : [ {
"kind" : "BOOLEAN",
"accessibleScopes" : "ALL",
"name" : "exec.errors.verbose",
"bool_val" : true,
"scope" : "SESSION"
}, {
"kind" : "LONG",
"accessibleScopes" : "ALL",
"name" : "planner.width.max_per_node",
"num_val" : 2,
"scope" : "SESSION"
} ],
"queue" : 0,
"hasResourcePlan" : false,
"resultMode" : "EXEC"
},
"graph" : [ {
"pop" : "fs-scan",
"@id" : 4,
"userName" : "sorabhhamirwasia",
"files" : [ "classpath:/employee.json" ],
"storage" : {
"type" : "file",
"connection" : "classpath:///",
"config" : null,
"workspaces" : { },
"formats" : {
"image" : {
"type" : "image",
"extensions" : [ "jpg", "jpeg", "jpe", "tif", "tiff", "dng", "psd", "png",
"bmp", "gif", "ico", "pcx", "wav", "wave", "avi", "webp", "mov", "mp4", "m4a",
"m4p", "m4b", "m4r", "m4v", "3gp", "3g2", "eps", "epsf", "epsi", "ai", "arw",
"crw", "cr2", "nef", "orf", "raf", "rw2", "rwl", "srw", "x3f" ]
},
"txt" : {
"type" : "text",
"extensions" : [ "txt" ],
"delimiter" : "\u"
},
"sequencefile" : {
"type" : "sequencefile",
"extensions" : [ "seq" ]
},
"psv" : {
"type" : "text",
"extensions" : [ "tbl" ],
"delimiter" : "|"
},
"tsv" : {
"type" : "text",
"extensions" : [ "tsv" ],
"delimiter" : "\t"
},
"csv" : {
"type" : "text",
"extensions" : [ "csv" ],
"delimiter" : ","
},
"ssv" : {
"type" : "text",
"extensions" : [ "ssv" ],
"delimiter" : " "
},
"csvh-test" : {
"type" : "text",
"extensions" : [ "csvh-test" ],
"skipFirstLine" : true,
"extractHeader" : true,
"delimiter" : ","
},
"json" : {
"type" : "json",
"extensions" : [ "json" ]
},
"csvh" : {
"type" : "text",
"extensions" : [ "csvh" ],
"extractHeader" : true,
"delimiter" : ","
},
"parquet" : {
"type" : "parquet"
},
"avro" : {
"type" : "avro"
}
},
"enabled" : true
},
"format" : {
"type" : "json",
"extensions" : [ "json" ]
},
"columns" : [ "`**`" ],
"selectionRoot" : "classpath:/employee.json",
"cost" : 463.0
}, {
"pop" : "limit",
"@id" : 3,
"child" : 4,
"first" : 0,
"last" : 2,
"initialAllocation" : 100,
"maxAllocation" : 100,
"cost" : 2.0
}, {
"pop" : "selection-vector-remover",
"@id" : 2,
"child" : 3,
"initialAllocation" : 100,
"maxAllocation" : 100,
"cost" : 2.0
}, {
"pop" : "project",
"@id" : 1,
"exprs" : [ {
"ref" : "`**`",
"expr" : "`**`"
} ],
"child" : 2,
"outputProj" : true,
"initialAllocation" : 100,
"maxAllocation" : 100,
"cost" : 2.0
}, {
"pop" : "screen",
"@id" : 0,
"child" : 1,
"initialAllocation" : 100,
"maxAllocation" : 100,
"cost" : 2.0
} ]
}{code}
*Physical Operator using above steps:*
{code:java}
{
"head" : {
"version" : 1,
"generator" : {
"type" : "org.apache.drill.exec.planner.logical.DrillImplementor",
"info" : ""
},
"type" : "APACHE_DRILL_PHYSICAL",
"options" : [ {
"kind" : "BOOLEAN",
"accessibleScopes" : "ALL",
"name" : "exec.errors.verbose",
"bool_val" : true,
"scope" : "SESSION"
}, {
"kind" : "LONG",
"accessibleScopes" : "ALL",
"name" : "planner.width.max_per_node",
"num_val" : 2,
"scope" : "SESSION"
} ],
"queue" : 0,
"hasResourcePlan" : false,
"resultMode" : "EXEC"
},
"graph" : [ {
"pop" : "fs-scan",
"@id" : 0,
"userName" : "anonymous",
"files" : [ "classpath:/employee.json" ],
"storage" : {
"type" : "file",
"connection" : "classpath:///",
"config" : null,
"workspaces" : { },
"formats" : {
"image" : {
"type" : "image",
"extensions" : [ "jpg", "jpeg", "jpe", "tif", "tiff", "dng", "psd", "png",
"bmp", "gif", "ico", "pcx", "wav", "wave", "avi", "webp", "mov", "mp4", "m4a",
"m4p", "m4b", "m4r", "m4v", "3gp", "3g2", "eps", "epsf", "epsi", "ai", "arw",
"crw", "cr2", "nef", "orf", "raf", "rw2", "rwl", "srw", "x3f" ]
},
"txt" : {
"type" : "text",
"extensions" : [ "txt" ],
"delimiter" : "\u"
},
"sequencefile" : {
"type" : "sequencefile",
"extensions" : [ "seq" ]
},
"psv" : {
"type" : "text",
"extensions" : [ "tbl" ],
"delimiter" : "|"
},
"tsv" : {
"type" : "text",
"extensions" : [ "tsv" ],
"delimiter" : "\t"
},
"csv" : {
"type" : "text",
"extensions" : [ "csv" ],
"delimiter" : ","
},
"ssv" : {
"type" : "text",
"extensions" : [ "ssv" ],
"delimiter" : " "
[jira] [Updated] (DRILL-6971) Display query state in query result page
[ https://issues.apache.org/jira/browse/DRILL-6971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sorabh Hamirwasia updated DRILL-6971: - Summary: Display query state in query result page (was: Display QueryState in query result page) > Display query state in query result page > > > Key: DRILL-6971 > URL: https://issues.apache.org/jira/browse/DRILL-6971 > Project: Apache Drill > Issue Type: Improvement > Components: Web Server >Affects Versions: 1.15.0 >Reporter: Sorabh Hamirwasia >Assignee: Sorabh Hamirwasia >Priority: Major > Fix For: 1.16.0 > > Original Estimate: 24h > Remaining Estimate: 24h > > Currently Query Result page doesn't show any state of the query. For > cancellation scenarios when there is no results fetched then it just shows No > Result Found and when partial results are returned then it will show the > returned data. Though user can look into the query state in profile page they > have to navigate away from result. It will be useful to know about the query > state with the result in same page. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6971) Display QueryState in query result page
Sorabh Hamirwasia created DRILL-6971: Summary: Display QueryState in query result page Key: DRILL-6971 URL: https://issues.apache.org/jira/browse/DRILL-6971 Project: Apache Drill Issue Type: Improvement Components: Web Server Affects Versions: 1.15.0 Reporter: Sorabh Hamirwasia Assignee: Sorabh Hamirwasia Fix For: 1.16.0 Currently Query Result page doesn't show any state of the query. For cancellation scenarios when there is no results fetched then it just shows No Result Found and when partial results are returned then it will show the returned data. Though user can look into the query state in profile page they have to navigate away from result. It will be useful to know about the query state with the result in same page. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6918) Querying empty topics fails with "NumberFormatException"
[
https://issues.apache.org/jira/browse/DRILL-6918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sorabh Hamirwasia updated DRILL-6918:
-
Reviewer: Sorabh Hamirwasia
> Querying empty topics fails with "NumberFormatException"
>
>
> Key: DRILL-6918
> URL: https://issues.apache.org/jira/browse/DRILL-6918
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Kafka
>Affects Versions: 1.14.0
>Reporter: Abhishek Ravi
>Assignee: Abhishek Ravi
>Priority: Minor
> Labels: ready-to-commit
> Fix For: 1.16.0
>
>
> Queries with filter conditions fail with {{NumberFormatException}} when
> querying empty topics.
> {noformat}
> 0: jdbc:drill:drillbit=10.10.100.189> select * from `topic2` where Field1 =
> 'abc';
> Error: SYSTEM ERROR: NumberFormatException: abc
> Fragment 0:0
> Please, refer to logs for more information.
> [Error Id: a0718456-c053-4820-9bd8-69c683598344 on qa-node189.qa.lab:31010]
> (state=,code=0)
> {noformat}
>
> *Logs:*
> {noformat}
> 2018-12-20 22:36:34,576 [23e3760d-7d23-5489-e2fb-6daf383053ee:foreman] INFO
> o.a.drill.exec.work.foreman.Foreman - Query text for query with id
> 23e3760d-7d23-5489-e2fb-6daf383053ee issued by root: select * from `topic2`
> where Field1 = 'abc'
> 2018-12-20 22:36:35,134 [23e3760d-7d23-5489-e2fb-6daf383053ee:foreman] INFO
> o.a.d.e.s.k.KafkaPushDownFilterIntoScan - Partitions ScanSpec before
> pushdown: [KafkaPartitionScanSpec [topicName=topic2, partitionId=2,
> startOffset=0, endOffset=0], KafkaPartitionScanSpec [topicName=topic2,
> partitionId=1, startOffset=0, endOffset=0], KafkaPartitionScanSpec
> [topicName=topic2, partitionId=0, startOffset=0, endOffset=0]]
> 2018-12-20 22:36:35,170 [23e3760d-7d23-5489-e2fb-6daf383053ee:frag:0:0] INFO
> o.a.d.e.s.k.KafkaScanBatchCreator - Number of record readers initialized : 3
> 2018-12-20 22:36:35,171 [23e3760d-7d23-5489-e2fb-6daf383053ee:frag:0:0] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 23e3760d-7d23-5489-e2fb-6daf383053ee:0:0: State change requested
> AWAITING_ALLOCATION --> RUNNING
> 2018-12-20 22:36:35,172 [23e3760d-7d23-5489-e2fb-6daf383053ee:frag:0:0] INFO
> o.a.d.e.w.f.FragmentStatusReporter -
> 23e3760d-7d23-5489-e2fb-6daf383053ee:0:0: State to report: RUNNING
> 2018-12-20 22:36:35,173 [23e3760d-7d23-5489-e2fb-6daf383053ee:frag:0:0] INFO
> o.a.d.e.s.k.d.MessageReaderFactory - Initialized Message Reader :
> JsonMessageReader[jsonReader=null]
> 2018-12-20 22:36:35,177 [23e3760d-7d23-5489-e2fb-6daf383053ee:frag:0:0] INFO
> o.a.d.e.store.kafka.MessageIterator - Start offset of topic2:2 is - 0
> 2018-12-20 22:36:35,177 [23e3760d-7d23-5489-e2fb-6daf383053ee:frag:0:0] INFO
> o.a.d.e.s.kafka.KafkaRecordReader - Last offset processed for topic2:2 is - 0
> 2018-12-20 22:36:35,177 [23e3760d-7d23-5489-e2fb-6daf383053ee:frag:0:0] INFO
> o.a.d.e.s.kafka.KafkaRecordReader - Total time to fetch messages from
> topic2:2 is - 0 milliseconds
> 2018-12-20 22:36:35,178 [23e3760d-7d23-5489-e2fb-6daf383053ee:frag:0:0] WARN
> o.a.d.e.e.ExpressionTreeMaterializer - Unable to find value vector of path
> `Field1`, returning null instance.
> 2018-12-20 22:36:35,191 [23e3760d-7d23-5489-e2fb-6daf383053ee:frag:0:0] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 23e3760d-7d23-5489-e2fb-6daf383053ee:0:0: State change requested RUNNING -->
> FAILED
> 2018-12-20 22:36:35,191 [23e3760d-7d23-5489-e2fb-6daf383053ee:frag:0:0] ERROR
> o.a.d.e.physical.impl.BaseRootExec - Batch dump started: dumping last 2
> failed batches
> 2018-12-20 22:36:35,191 [23e3760d-7d23-5489-e2fb-6daf383053ee:frag:0:0] ERROR
> o.a.d.e.p.i.s.RemovingRecordBatch -
> RemovingRecordBatch[container=org.apache.drill.exec.record.VectorContainer@3ce6a91e[recordCount
> = 0, schemaChanged = true, schema = null, wrappers = [], ...], state=FIRST,
> copier=null]
> 2018-12-20 22:36:35,191 [23e3760d-7d23-5489-e2fb-6daf383053ee:frag:0:0] ERROR
> o.a.d.e.p.i.filter.FilterRecordBatch -
> FilterRecordBatch[container=org.apache.drill.exec.record.VectorContainer@2057ff66[recordCount
> = 0, schemaChanged = true, schema = null, wrappers =
> [org.apache.drill.exec.vector.NullableIntVector@32edcdf2[field = [`T4¦¦**`
> (INT:OPTIONAL)], ...],
> org.apache.drill.exec.vector.NullableIntVector@3a5bf582[field = [`Field1`
> (INT:OPTIONAL)], ...]], ...], selectionVector2=[SV2: recs=0 - ], filter=null,
> popConfig=org.apache.drill.exec.physical.config.Filter@1d69df75]
> 2018-12-20 22:36:35,191 [23e3760d-7d23-5489-e2fb-6daf383053ee:frag:0:0] ERROR
> o.a.d.e.physical.impl.BaseRootExec - Batch dump completed.
> 2018-12-20 22:36:35,192 [23e3760d-7d23-5489-e2fb-6daf383053ee:frag:0:0] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 23e3760d-7d23-5489-e2fb-6daf383053ee:0:0: State change requested FAILED -->
> FIN
[jira] [Updated] (DRILL-6934) Update the option documentation for planner.enable_unnest_lateral
[ https://issues.apache.org/jira/browse/DRILL-6934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sorabh Hamirwasia updated DRILL-6934: - Labels: ready-to-commit (was: ) > Update the option documentation for planner.enable_unnest_lateral > - > > Key: DRILL-6934 > URL: https://issues.apache.org/jira/browse/DRILL-6934 > Project: Apache Drill > Issue Type: Bug > Components: Documentation >Affects Versions: 1.15.0 >Reporter: Sorabh Hamirwasia >Assignee: Sorabh Hamirwasia >Priority: Major > Labels: ready-to-commit > Fix For: 1.16.0 > > > The planner option documentation on WebUI shows that default value of option > planner.enable_unnest_lateral is false which is wrong. It is true by default > post from 1.15 onwards and hence needs to be changed to reflect that. > > |planner.enable_unnest_lateral|Enables lateral join functionality. Default is > false. (Drill 1.14+)| -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6934) Update the option documentation for planner.enable_unnest_lateral
[ https://issues.apache.org/jira/browse/DRILL-6934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sorabh Hamirwasia updated DRILL-6934: - Description: The planner option documentation on WebUI shows that default value of option planner.enable_unnest_lateral is false which is wrong. It is true by default post from 1.15 onwards and hence needs to be changed to reflect that. |planner.enable_unnest_lateral|Enables lateral join functionality. Default is false. (Drill 1.14+)| was:The planner option documentation on WebUI shows that default value of option planner.enable_unnest_lateral is false which is wrong. It is true by default post from 1.15 onwards and hence needs to be changed to reflect that. > Update the option documentation for planner.enable_unnest_lateral > - > > Key: DRILL-6934 > URL: https://issues.apache.org/jira/browse/DRILL-6934 > Project: Apache Drill > Issue Type: Bug > Components: Documentation >Affects Versions: 1.15.0 >Reporter: Sorabh Hamirwasia >Assignee: Sorabh Hamirwasia >Priority: Major > Fix For: 1.16.0 > > > The planner option documentation on WebUI shows that default value of option > planner.enable_unnest_lateral is false which is wrong. It is true by default > post from 1.15 onwards and hence needs to be changed to reflect that. > > |planner.enable_unnest_lateral|Enables lateral join functionality. Default is > false. (Drill 1.14+)| -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6934) Update the option documentation for planner.enable_unnest_lateral
[ https://issues.apache.org/jira/browse/DRILL-6934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16729879#comment-16729879 ] Sorabh Hamirwasia commented on DRILL-6934: -- [~bbevens] - Its documentation within the code not on the website. I have opened a PR for it > Update the option documentation for planner.enable_unnest_lateral > - > > Key: DRILL-6934 > URL: https://issues.apache.org/jira/browse/DRILL-6934 > Project: Apache Drill > Issue Type: Bug > Components: Documentation >Affects Versions: 1.15.0 >Reporter: Sorabh Hamirwasia >Assignee: Sorabh Hamirwasia >Priority: Major > Fix For: 1.16.0 > > > The planner option documentation on WebUI shows that default value of option > planner.enable_unnest_lateral is false which is wrong. It is true by default > post from 1.15 onwards and hence needs to be changed to reflect that. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6934) Update the option documentation for planner.enable_unnest_lateral
Sorabh Hamirwasia created DRILL-6934: Summary: Update the option documentation for planner.enable_unnest_lateral Key: DRILL-6934 URL: https://issues.apache.org/jira/browse/DRILL-6934 Project: Apache Drill Issue Type: Bug Components: Documentation Affects Versions: 1.15.0 Reporter: Sorabh Hamirwasia Assignee: Sorabh Hamirwasia Fix For: 1.16.0 The planner option documentation on WebUI shows that default value of option planner.enable_unnest_lateral is false which is wrong. It is true by default post from 1.15 onwards and hence needs to be changed to reflect that. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6906) File permissions are not being honored
[
https://issues.apache.org/jira/browse/DRILL-6906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16723635#comment-16723635
]
Sorabh Hamirwasia commented on DRILL-6906:
--
This was found to be the config issue on MapR cluster where below variable was
expected to be set for impersonation to work:
{code:java}
export MAPR_IMPERSONATION_ENABLED=${MAPR_IMPERSONATION_ENABLED:-"true"}
{code}
> File permissions are not being honored
> --
>
> Key: DRILL-6906
> URL: https://issues.apache.org/jira/browse/DRILL-6906
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC, Client - ODBC
>Affects Versions: 1.15.0
>Reporter: Robert Hou
>Assignee: Pritesh Maker
>Priority: Major
> Fix For: 1.15.0
>
>
> I ran sqlline with user "kuser1".
> {noformat}
> /opt/mapr/drill/drill-1.15.0.apache/bin/sqlline -u
> "jdbc:drill:drillbit=10.10.30.206" -n kuser1 -p mapr
> {noformat}
> I tried to access a file that is only accessible by root:
> {noformat}
> [root@perfnode206 drill-test-framework_krystal]# hf -ls
> /drill/testdata/impersonation/neg_tc5/student
> -rwx-- 3 root root 64612 2018-06-19 10:30
> /drill/testdata/impersonation/neg_tc5/student
> {noformat}
> I am able to read the table, which should not be possible. I used this
> commit for Drill 1.15.
> {noformat}
> git.commit.id=bf2b414ac62cfc515fdd77f2688bb110073d764d
> git.commit.message.full=DRILL-6866\: Upgrade to SqlLine 1.6.0\n\n1. Changed
> SqlLine version to 1.6.0.\n2. Overridden new getVersion method in
> DrillSqlLineApplication.\n3. Set maxColumnWidth to 80 to avoid issue
> described in DRILL-6769.\n4. Changed colorScheme to obsidian.\n5. Output null
> value for varchar / char / boolean types as null instead of empty string.\n6.
> Changed access modifier from package default to public for JDBC classes that
> implement external interfaces to avoid issues when calling methods from these
> classes using reflection.\n\ncloses \#1556
> {noformat}
> This is from drillbit.log. It shows that user is kuser1.
> {noformat}
> 2018-12-15 05:00:52,516 [23eb04fb-1701-bea7-dd97-ecda58795b3b:foreman] DEBUG
> o.a.d.e.w.f.QueryStateProcessor - 23eb04fb-1701-bea7-dd97-ecda58795b3b: State
> change requested PREPARING --> PLANNING
> 2018-12-15 05:00:52,531 [23eb04fb-1701-bea7-dd97-ecda58795b3b:foreman] INFO
> o.a.drill.exec.work.foreman.Foreman - Query text for query with id
> 23eb04fb-1701-bea7-dd97-ecda58795b3b issued by kuser1: select * from
> dfs.`/drill/testdata/impersonation/neg_tc5/student`
> {noformat}
> It is not clear to me if this is a Drill problem or a file system problem. I
> tested MFS by logging in as kuser1 and trying to copy the file using "hadoop
> fs -copyToLocal /drill/testdata/impersonation/neg_tc5/student" and got an
> error, and was not able to copy the file. So I think MFS permissions are
> working.
> I also tried with Drill 1.14, and I get the expected error:
> {noformat}
> 0: jdbc:drill:drillbit=10.10.30.206> select * from
> dfs.`/drill/testdata/impersonation/neg_tc5/student` limit 1;
> Error: VALIDATION ERROR: From line 1, column 15 to line 1, column 17: Object
> '/drill/testdata/impersonation/neg_tc5/student' not found within 'dfs'
> [Error Id: cdf18c2a-b005-4f92-b819-d4324e8807d9 on
> perfnode206.perf.lab:31010] (state=,code=0)
> {noformat}
> The commit for Drill 1.14 is:
> {noformat}
> git.commit.message.full=[maven-release-plugin] prepare release drill-1.14.0\n
> git.commit.id=0508a128853ce796ca7e99e13008e49442f83147
> {noformat}
> This problem exists with both Apache JDBC and Simba ODBC.
> Here is drill-distrib.conf. drill-override.conf is empty. It is the same
> for both 1.14 and 1.15.
> {noformat}
> drill.exec: {
> cluster-id: "secure206-drillbits",
> zk.connect:
> "perfnode206.perf.lab:5181,perfnode207.perf.lab:5181,perfnode208.perf.lab:5181",
> rpc.user.client.threads: "4",
> options.store.parquet.block-size: "268435456",
> sys.store.provider.zk.blobroot: "maprfs:///apps/drill",
> spill.directories: [ "/tmp/drill/spill" ],
> spill.fs: "maprfs:///",
> storage.action_on_plugins_override_file: "rename"
> zk.apply_secure_acl: true,
> impersonation.enabled: true,
> impersonation.max_chained_user_hops: 3,
> options.exec.impersonation.inbound_policies:
> "[{proxy_principals:{users:[\"mapr\"]},target_principals:{users:[\"*\"]}}]",
> security.auth.mechanisms: ["PLAIN", "KERBEROS"],
> security.auth.principal : "mapr/maprs...@qa.lab",
> security.auth.keytab : "/etc/drill/mapr_maprsasl.keytab",
> security.user.auth.enabled: true,
> security.user.auth.packages += "org.apache.drill.exec.rpc.user.security",
> security.user.auth.impl: "pam4j",
> security.user.auth.pam_profiles: ["sudo", "login"],
> http.ssl_enabled: true,
> ssl.useHad
[jira] [Comment Edited] (DRILL-6906) File permissions are not being honored
[
https://issues.apache.org/jira/browse/DRILL-6906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16723635#comment-16723635
]
Sorabh Hamirwasia edited comment on DRILL-6906 at 12/18/18 3:44 AM:
This was found to be the config issue on MapR cluster where below variable was
expected to be set in drill-env.sh for impersonation to work:
{code:java}
export MAPR_IMPERSONATION_ENABLED=${MAPR_IMPERSONATION_ENABLED:-"true"}
{code}
was (Author: shamirwasia):
This was found to be the config issue on MapR cluster where below variable was
expected to be set for impersonation to work:
{code:java}
export MAPR_IMPERSONATION_ENABLED=${MAPR_IMPERSONATION_ENABLED:-"true"}
{code}
> File permissions are not being honored
> --
>
> Key: DRILL-6906
> URL: https://issues.apache.org/jira/browse/DRILL-6906
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC, Client - ODBC
>Affects Versions: 1.15.0
>Reporter: Robert Hou
>Assignee: Kunal Khatua
>Priority: Major
> Fix For: 1.15.0
>
>
> I ran sqlline with user "kuser1".
> {noformat}
> /opt/mapr/drill/drill-1.15.0.apache/bin/sqlline -u
> "jdbc:drill:drillbit=10.10.30.206" -n kuser1 -p mapr
> {noformat}
> I tried to access a file that is only accessible by root:
> {noformat}
> [root@perfnode206 drill-test-framework_krystal]# hf -ls
> /drill/testdata/impersonation/neg_tc5/student
> -rwx-- 3 root root 64612 2018-06-19 10:30
> /drill/testdata/impersonation/neg_tc5/student
> {noformat}
> I am able to read the table, which should not be possible. I used this
> commit for Drill 1.15.
> {noformat}
> git.commit.id=bf2b414ac62cfc515fdd77f2688bb110073d764d
> git.commit.message.full=DRILL-6866\: Upgrade to SqlLine 1.6.0\n\n1. Changed
> SqlLine version to 1.6.0.\n2. Overridden new getVersion method in
> DrillSqlLineApplication.\n3. Set maxColumnWidth to 80 to avoid issue
> described in DRILL-6769.\n4. Changed colorScheme to obsidian.\n5. Output null
> value for varchar / char / boolean types as null instead of empty string.\n6.
> Changed access modifier from package default to public for JDBC classes that
> implement external interfaces to avoid issues when calling methods from these
> classes using reflection.\n\ncloses \#1556
> {noformat}
> This is from drillbit.log. It shows that user is kuser1.
> {noformat}
> 2018-12-15 05:00:52,516 [23eb04fb-1701-bea7-dd97-ecda58795b3b:foreman] DEBUG
> o.a.d.e.w.f.QueryStateProcessor - 23eb04fb-1701-bea7-dd97-ecda58795b3b: State
> change requested PREPARING --> PLANNING
> 2018-12-15 05:00:52,531 [23eb04fb-1701-bea7-dd97-ecda58795b3b:foreman] INFO
> o.a.drill.exec.work.foreman.Foreman - Query text for query with id
> 23eb04fb-1701-bea7-dd97-ecda58795b3b issued by kuser1: select * from
> dfs.`/drill/testdata/impersonation/neg_tc5/student`
> {noformat}
> It is not clear to me if this is a Drill problem or a file system problem. I
> tested MFS by logging in as kuser1 and trying to copy the file using "hadoop
> fs -copyToLocal /drill/testdata/impersonation/neg_tc5/student" and got an
> error, and was not able to copy the file. So I think MFS permissions are
> working.
> I also tried with Drill 1.14, and I get the expected error:
> {noformat}
> 0: jdbc:drill:drillbit=10.10.30.206> select * from
> dfs.`/drill/testdata/impersonation/neg_tc5/student` limit 1;
> Error: VALIDATION ERROR: From line 1, column 15 to line 1, column 17: Object
> '/drill/testdata/impersonation/neg_tc5/student' not found within 'dfs'
> [Error Id: cdf18c2a-b005-4f92-b819-d4324e8807d9 on
> perfnode206.perf.lab:31010] (state=,code=0)
> {noformat}
> The commit for Drill 1.14 is:
> {noformat}
> git.commit.message.full=[maven-release-plugin] prepare release drill-1.14.0\n
> git.commit.id=0508a128853ce796ca7e99e13008e49442f83147
> {noformat}
> This problem exists with both Apache JDBC and Simba ODBC.
> Here is drill-distrib.conf. drill-override.conf is empty. It is the same
> for both 1.14 and 1.15.
> {noformat}
> drill.exec: {
> cluster-id: "secure206-drillbits",
> zk.connect:
> "perfnode206.perf.lab:5181,perfnode207.perf.lab:5181,perfnode208.perf.lab:5181",
> rpc.user.client.threads: "4",
> options.store.parquet.block-size: "268435456",
> sys.store.provider.zk.blobroot: "maprfs:///apps/drill",
> spill.directories: [ "/tmp/drill/spill" ],
> spill.fs: "maprfs:///",
> storage.action_on_plugins_override_file: "rename"
> zk.apply_secure_acl: true,
> impersonation.enabled: true,
> impersonation.max_chained_user_hops: 3,
> options.exec.impersonation.inbound_policies:
> "[{proxy_principals:{users:[\"mapr\"]},target_principals:{users:[\"*\"]}}]",
> security.auth.mechanisms: ["PLAIN", "KERBEROS"],
> security.auth.principal : "mapr/maprs...@qa.lab",
>
[jira] [Assigned] (DRILL-6906) File permissions are not being honored
[
https://issues.apache.org/jira/browse/DRILL-6906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sorabh Hamirwasia reassigned DRILL-6906:
Assignee: Kunal Khatua (was: Pritesh Maker)
> File permissions are not being honored
> --
>
> Key: DRILL-6906
> URL: https://issues.apache.org/jira/browse/DRILL-6906
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC, Client - ODBC
>Affects Versions: 1.15.0
>Reporter: Robert Hou
>Assignee: Kunal Khatua
>Priority: Major
> Fix For: 1.15.0
>
>
> I ran sqlline with user "kuser1".
> {noformat}
> /opt/mapr/drill/drill-1.15.0.apache/bin/sqlline -u
> "jdbc:drill:drillbit=10.10.30.206" -n kuser1 -p mapr
> {noformat}
> I tried to access a file that is only accessible by root:
> {noformat}
> [root@perfnode206 drill-test-framework_krystal]# hf -ls
> /drill/testdata/impersonation/neg_tc5/student
> -rwx-- 3 root root 64612 2018-06-19 10:30
> /drill/testdata/impersonation/neg_tc5/student
> {noformat}
> I am able to read the table, which should not be possible. I used this
> commit for Drill 1.15.
> {noformat}
> git.commit.id=bf2b414ac62cfc515fdd77f2688bb110073d764d
> git.commit.message.full=DRILL-6866\: Upgrade to SqlLine 1.6.0\n\n1. Changed
> SqlLine version to 1.6.0.\n2. Overridden new getVersion method in
> DrillSqlLineApplication.\n3. Set maxColumnWidth to 80 to avoid issue
> described in DRILL-6769.\n4. Changed colorScheme to obsidian.\n5. Output null
> value for varchar / char / boolean types as null instead of empty string.\n6.
> Changed access modifier from package default to public for JDBC classes that
> implement external interfaces to avoid issues when calling methods from these
> classes using reflection.\n\ncloses \#1556
> {noformat}
> This is from drillbit.log. It shows that user is kuser1.
> {noformat}
> 2018-12-15 05:00:52,516 [23eb04fb-1701-bea7-dd97-ecda58795b3b:foreman] DEBUG
> o.a.d.e.w.f.QueryStateProcessor - 23eb04fb-1701-bea7-dd97-ecda58795b3b: State
> change requested PREPARING --> PLANNING
> 2018-12-15 05:00:52,531 [23eb04fb-1701-bea7-dd97-ecda58795b3b:foreman] INFO
> o.a.drill.exec.work.foreman.Foreman - Query text for query with id
> 23eb04fb-1701-bea7-dd97-ecda58795b3b issued by kuser1: select * from
> dfs.`/drill/testdata/impersonation/neg_tc5/student`
> {noformat}
> It is not clear to me if this is a Drill problem or a file system problem. I
> tested MFS by logging in as kuser1 and trying to copy the file using "hadoop
> fs -copyToLocal /drill/testdata/impersonation/neg_tc5/student" and got an
> error, and was not able to copy the file. So I think MFS permissions are
> working.
> I also tried with Drill 1.14, and I get the expected error:
> {noformat}
> 0: jdbc:drill:drillbit=10.10.30.206> select * from
> dfs.`/drill/testdata/impersonation/neg_tc5/student` limit 1;
> Error: VALIDATION ERROR: From line 1, column 15 to line 1, column 17: Object
> '/drill/testdata/impersonation/neg_tc5/student' not found within 'dfs'
> [Error Id: cdf18c2a-b005-4f92-b819-d4324e8807d9 on
> perfnode206.perf.lab:31010] (state=,code=0)
> {noformat}
> The commit for Drill 1.14 is:
> {noformat}
> git.commit.message.full=[maven-release-plugin] prepare release drill-1.14.0\n
> git.commit.id=0508a128853ce796ca7e99e13008e49442f83147
> {noformat}
> This problem exists with both Apache JDBC and Simba ODBC.
> Here is drill-distrib.conf. drill-override.conf is empty. It is the same
> for both 1.14 and 1.15.
> {noformat}
> drill.exec: {
> cluster-id: "secure206-drillbits",
> zk.connect:
> "perfnode206.perf.lab:5181,perfnode207.perf.lab:5181,perfnode208.perf.lab:5181",
> rpc.user.client.threads: "4",
> options.store.parquet.block-size: "268435456",
> sys.store.provider.zk.blobroot: "maprfs:///apps/drill",
> spill.directories: [ "/tmp/drill/spill" ],
> spill.fs: "maprfs:///",
> storage.action_on_plugins_override_file: "rename"
> zk.apply_secure_acl: true,
> impersonation.enabled: true,
> impersonation.max_chained_user_hops: 3,
> options.exec.impersonation.inbound_policies:
> "[{proxy_principals:{users:[\"mapr\"]},target_principals:{users:[\"*\"]}}]",
> security.auth.mechanisms: ["PLAIN", "KERBEROS"],
> security.auth.principal : "mapr/maprs...@qa.lab",
> security.auth.keytab : "/etc/drill/mapr_maprsasl.keytab",
> security.user.auth.enabled: true,
> security.user.auth.packages += "org.apache.drill.exec.rpc.user.security",
> security.user.auth.impl: "pam4j",
> security.user.auth.pam_profiles: ["sudo", "login"],
> http.ssl_enabled: true,
> ssl.useHadoopConfig: true,
> http.auth.mechanisms: ["FORM", "SPNEGO"],
> http.auth.spnego.principal: "HTTP/perfnode206.perf@qa.lab",
> http.auth.spnego.keytab: "/etc/drill_spnego/perfnode206.keytab"
> }
> {noformat}
--
[jira] [Updated] (DRILL-6906) File permissions are not being honored
[
https://issues.apache.org/jira/browse/DRILL-6906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sorabh Hamirwasia updated DRILL-6906:
-
Priority: Major (was: Blocker)
> File permissions are not being honored
> --
>
> Key: DRILL-6906
> URL: https://issues.apache.org/jira/browse/DRILL-6906
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC, Client - ODBC
>Affects Versions: 1.15.0
>Reporter: Robert Hou
>Assignee: Pritesh Maker
>Priority: Major
> Fix For: 1.15.0
>
>
> I ran sqlline with user "kuser1".
> {noformat}
> /opt/mapr/drill/drill-1.15.0.apache/bin/sqlline -u
> "jdbc:drill:drillbit=10.10.30.206" -n kuser1 -p mapr
> {noformat}
> I tried to access a file that is only accessible by root:
> {noformat}
> [root@perfnode206 drill-test-framework_krystal]# hf -ls
> /drill/testdata/impersonation/neg_tc5/student
> -rwx-- 3 root root 64612 2018-06-19 10:30
> /drill/testdata/impersonation/neg_tc5/student
> {noformat}
> I am able to read the table, which should not be possible. I used this
> commit for Drill 1.15.
> {noformat}
> git.commit.id=bf2b414ac62cfc515fdd77f2688bb110073d764d
> git.commit.message.full=DRILL-6866\: Upgrade to SqlLine 1.6.0\n\n1. Changed
> SqlLine version to 1.6.0.\n2. Overridden new getVersion method in
> DrillSqlLineApplication.\n3. Set maxColumnWidth to 80 to avoid issue
> described in DRILL-6769.\n4. Changed colorScheme to obsidian.\n5. Output null
> value for varchar / char / boolean types as null instead of empty string.\n6.
> Changed access modifier from package default to public for JDBC classes that
> implement external interfaces to avoid issues when calling methods from these
> classes using reflection.\n\ncloses \#1556
> {noformat}
> This is from drillbit.log. It shows that user is kuser1.
> {noformat}
> 2018-12-15 05:00:52,516 [23eb04fb-1701-bea7-dd97-ecda58795b3b:foreman] DEBUG
> o.a.d.e.w.f.QueryStateProcessor - 23eb04fb-1701-bea7-dd97-ecda58795b3b: State
> change requested PREPARING --> PLANNING
> 2018-12-15 05:00:52,531 [23eb04fb-1701-bea7-dd97-ecda58795b3b:foreman] INFO
> o.a.drill.exec.work.foreman.Foreman - Query text for query with id
> 23eb04fb-1701-bea7-dd97-ecda58795b3b issued by kuser1: select * from
> dfs.`/drill/testdata/impersonation/neg_tc5/student`
> {noformat}
> It is not clear to me if this is a Drill problem or a file system problem. I
> tested MFS by logging in as kuser1 and trying to copy the file using "hadoop
> fs -copyToLocal /drill/testdata/impersonation/neg_tc5/student" and got an
> error, and was not able to copy the file. So I think MFS permissions are
> working.
> I also tried with Drill 1.14, and I get the expected error:
> {noformat}
> 0: jdbc:drill:drillbit=10.10.30.206> select * from
> dfs.`/drill/testdata/impersonation/neg_tc5/student` limit 1;
> Error: VALIDATION ERROR: From line 1, column 15 to line 1, column 17: Object
> '/drill/testdata/impersonation/neg_tc5/student' not found within 'dfs'
> [Error Id: cdf18c2a-b005-4f92-b819-d4324e8807d9 on
> perfnode206.perf.lab:31010] (state=,code=0)
> {noformat}
> The commit for Drill 1.14 is:
> {noformat}
> git.commit.message.full=[maven-release-plugin] prepare release drill-1.14.0\n
> git.commit.id=0508a128853ce796ca7e99e13008e49442f83147
> {noformat}
> This problem exists with both Apache JDBC and Simba ODBC.
> Here is drill-distrib.conf. drill-override.conf is empty. It is the same
> for both 1.14 and 1.15.
> {noformat}
> drill.exec: {
> cluster-id: "secure206-drillbits",
> zk.connect:
> "perfnode206.perf.lab:5181,perfnode207.perf.lab:5181,perfnode208.perf.lab:5181",
> rpc.user.client.threads: "4",
> options.store.parquet.block-size: "268435456",
> sys.store.provider.zk.blobroot: "maprfs:///apps/drill",
> spill.directories: [ "/tmp/drill/spill" ],
> spill.fs: "maprfs:///",
> storage.action_on_plugins_override_file: "rename"
> zk.apply_secure_acl: true,
> impersonation.enabled: true,
> impersonation.max_chained_user_hops: 3,
> options.exec.impersonation.inbound_policies:
> "[{proxy_principals:{users:[\"mapr\"]},target_principals:{users:[\"*\"]}}]",
> security.auth.mechanisms: ["PLAIN", "KERBEROS"],
> security.auth.principal : "mapr/maprs...@qa.lab",
> security.auth.keytab : "/etc/drill/mapr_maprsasl.keytab",
> security.user.auth.enabled: true,
> security.user.auth.packages += "org.apache.drill.exec.rpc.user.security",
> security.user.auth.impl: "pam4j",
> security.user.auth.pam_profiles: ["sudo", "login"],
> http.ssl_enabled: true,
> ssl.useHadoopConfig: true,
> http.auth.mechanisms: ["FORM", "SPNEGO"],
> http.auth.spnego.principal: "HTTP/perfnode206.perf@qa.lab",
> http.auth.spnego.keytab: "/etc/drill_spnego/perfnode206.keytab"
> }
> {noformat}
--
This message was se
[jira] [Commented] (DRILL-6906) File permissions are not being honored
[
https://issues.apache.org/jira/browse/DRILL-6906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16722408#comment-16722408
]
Sorabh Hamirwasia commented on DRILL-6906:
--
Can you share Drillbit config for both the scenarios ?
> File permissions are not being honored
> --
>
> Key: DRILL-6906
> URL: https://issues.apache.org/jira/browse/DRILL-6906
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC, Client - ODBC
>Affects Versions: 1.15.0
>Reporter: Robert Hou
>Assignee: Pritesh Maker
>Priority: Blocker
> Fix For: 1.15.0
>
>
> I ran sqlline with user "kuser1".
> {noformat}
> /opt/mapr/drill/drill-1.15.0.apache/bin/sqlline -u
> "jdbc:drill:drillbit=10.10.30.206" -n kuser1 -p mapr
> {noformat}
> I tried to access a file that is only accessible by root:
> {noformat}
> [root@perfnode206 drill-test-framework_krystal]# hf -ls
> /drill/testdata/impersonation/neg_tc5/student
> -rwx-- 3 root root 64612 2018-06-19 10:30
> /drill/testdata/impersonation/neg_tc5/student
> {noformat}
> I am able to read the table, which should not be possible. I used this
> commit for Drill 1.15.
> {noformat}
> git.commit.id=bf2b414ac62cfc515fdd77f2688bb110073d764d
> git.commit.message.full=DRILL-6866\: Upgrade to SqlLine 1.6.0\n\n1. Changed
> SqlLine version to 1.6.0.\n2. Overridden new getVersion method in
> DrillSqlLineApplication.\n3. Set maxColumnWidth to 80 to avoid issue
> described in DRILL-6769.\n4. Changed colorScheme to obsidian.\n5. Output null
> value for varchar / char / boolean types as null instead of empty string.\n6.
> Changed access modifier from package default to public for JDBC classes that
> implement external interfaces to avoid issues when calling methods from these
> classes using reflection.\n\ncloses \#1556
> {noformat}
> This is from drillbit.log. It shows that user is kuser1.
> {noformat}
> 2018-12-15 05:00:52,516 [23eb04fb-1701-bea7-dd97-ecda58795b3b:foreman] DEBUG
> o.a.d.e.w.f.QueryStateProcessor - 23eb04fb-1701-bea7-dd97-ecda58795b3b: State
> change requested PREPARING --> PLANNING
> 2018-12-15 05:00:52,531 [23eb04fb-1701-bea7-dd97-ecda58795b3b:foreman] INFO
> o.a.drill.exec.work.foreman.Foreman - Query text for query with id
> 23eb04fb-1701-bea7-dd97-ecda58795b3b issued by kuser1: select * from
> dfs.`/drill/testdata/impersonation/neg_tc5/student`
> {noformat}
> It is not clear to me if this is a Drill problem or a file system problem. I
> tested MFS by logging in as kuser1 and trying to copy the file using "hadoop
> fs -copyToLocal /drill/testdata/impersonation/neg_tc5/student" and got an
> error, and was not able to copy the file. So I think MFS permissions are
> working.
> I also tried with Drill 1.14, and I get the expected error:
> {noformat}
> 0: jdbc:drill:drillbit=10.10.30.206> select * from
> dfs.`/drill/testdata/impersonation/neg_tc5/student` limit 1;
> Error: VALIDATION ERROR: From line 1, column 15 to line 1, column 17: Object
> '/drill/testdata/impersonation/neg_tc5/student' not found within 'dfs'
> [Error Id: cdf18c2a-b005-4f92-b819-d4324e8807d9 on
> perfnode206.perf.lab:31010] (state=,code=0)
> {noformat}
> The commit for Drill 1.14 is:
> {noformat}
> git.commit.message.full=[maven-release-plugin] prepare release drill-1.14.0\n
> git.commit.id=0508a128853ce796ca7e99e13008e49442f83147
> {noformat}
> This problem exists with both Apache JDBC and Simba ODBC.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6751) Upgrade Apache parent POM to version 21
[ https://issues.apache.org/jira/browse/DRILL-6751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sorabh Hamirwasia updated DRILL-6751: - Labels: ready-to-commit (was: ) > Upgrade Apache parent POM to version 21 > --- > > Key: DRILL-6751 > URL: https://issues.apache.org/jira/browse/DRILL-6751 > Project: Apache Drill > Issue Type: Task > Components: Tools, Build & Test >Affects Versions: 1.14.0 >Reporter: Arina Ielchiieva >Assignee: Vitalii Diravka >Priority: Major > Labels: ready-to-commit > Fix For: 1.15.0 > > > Upgrade Apache parent POM to version 21 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
