[GitHub] drill issue #1029: DRILL-5867: List profiles in pages rather than a long ver...
Github user arina-ielchiieva commented on the issue: https://github.com/apache/drill/pull/1029 +1, LGTM. ---
[GitHub] drill issue #1023: DRILL-5922 Fixed Child Allocator Leak. DRILL-5926 Stabali...
Github user arina-ielchiieva commented on the issue: https://github.com/apache/drill/pull/1023 @ilooner please resolve conflicts. ---
[GitHub] drill pull request #921: DRILL-4286 Graceful shutdown of drillbit
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/921#discussion_r151110657
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/DrillRoot.java
---
@@ -58,13 +63,170 @@
@Inject UserAuthEnabled authEnabled;
@Inject WorkManager work;
@Inject SecurityContext sc;
+ @Inject Drillbit drillbit;
@GET
@Produces(MediaType.TEXT_HTML)
public Viewable getClusterInfo() {
return ViewableWithPermissions.create(authEnabled.get(),
"/rest/index.ftl", sc, getClusterInfoJSON());
}
+
+ @SuppressWarnings("resource")
+ @GET
+ @Path("/state")
+ @Produces(MediaType.APPLICATION_JSON)
+ public Response getDrillbitStatus(){
+Collection drillbits =
getClusterInfoJSON().getDrillbits();
+Map drillStatusMap = new HashMap();
+for (DrillbitInfo drillbit : drillbits) {
+
drillStatusMap.put(drillbit.getAddress()+"-"+drillbit.getUserPort(),drillbit.getState());
+}
+return Response.ok()
+.entity(drillStatusMap)
+.header("Access-Control-Allow-Origin", "*")
+.header("Access-Control-Allow-Methods", "GET, POST, DELETE,
PUT")
+ .header("Access-Control-Allow-Credentials","true")
+.allow("OPTIONS").build();
+ }
+
+ @SuppressWarnings("resource")
+ @GET
+ @Path("/graceperiod")
--- End diff --
/gracePeriod
---
[GitHub] drill pull request #921: DRILL-4286 Graceful shutdown of drillbit
Github user arina-ielchiieva commented on a diff in the pull request: https://github.com/apache/drill/pull/921#discussion_r15410 --- Diff: exec/java-exec/src/main/resources/rest/index.ftl --- @@ -179,6 +184,98 @@ +
[GitHub] drill pull request #921: DRILL-4286 Graceful shutdown of drillbit
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/921#discussion_r151110780
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/DrillRoot.java
---
@@ -58,13 +63,170 @@
@Inject UserAuthEnabled authEnabled;
@Inject WorkManager work;
@Inject SecurityContext sc;
+ @Inject Drillbit drillbit;
@GET
@Produces(MediaType.TEXT_HTML)
public Viewable getClusterInfo() {
return ViewableWithPermissions.create(authEnabled.get(),
"/rest/index.ftl", sc, getClusterInfoJSON());
}
+
+ @SuppressWarnings("resource")
+ @GET
+ @Path("/state")
+ @Produces(MediaType.APPLICATION_JSON)
+ public Response getDrillbitStatus(){
+Collection drillbits =
getClusterInfoJSON().getDrillbits();
+Map drillStatusMap = new HashMap();
+for (DrillbitInfo drillbit : drillbits) {
+
drillStatusMap.put(drillbit.getAddress()+"-"+drillbit.getUserPort(),drillbit.getState());
+}
+return Response.ok()
+.entity(drillStatusMap)
+.header("Access-Control-Allow-Origin", "*")
+.header("Access-Control-Allow-Methods", "GET, POST, DELETE,
PUT")
+ .header("Access-Control-Allow-Credentials","true")
+.allow("OPTIONS").build();
+ }
+
+ @SuppressWarnings("resource")
+ @GET
+ @Path("/graceperiod")
+ @Produces(MediaType.APPLICATION_JSON)
+ public Map getGracePeriod(){
+
+final DrillConfig config = work.getContext().getConfig();
+final int gracePeriod = config.getInt(ExecConstants.GRACE_PERIOD);
+Map gracePeriodMap = new HashMap();
+gracePeriodMap.put("graceperiod",gracePeriod);
+return gracePeriodMap;
+ }
+
+ @SuppressWarnings("resource")
+ @GET
+ @Path("/queriesCount")
+ @Produces(MediaType.APPLICATION_JSON)
+ public Response getRemainingQueries() {
+Map queriesInfo = new HashMap();
+queriesInfo = work.getRemainingQueries();
+return Response.ok()
+.entity(queriesInfo)
+.header("Access-Control-Allow-Origin", "*")
+.header("Access-Control-Allow-Methods", "GET, POST, DELETE,
PUT")
+ .header("Access-Control-Allow-Credentials","true")
+.allow("OPTIONS").build();
+ }
+
+ @SuppressWarnings("resource")
+ @POST
+ @Path("/graceful_shutdown")
+ @Produces(MediaType.APPLICATION_JSON)
+ public Response shutdownDrillbit() throws Exception {
+Map shutdownInfo = new HashMap();
+try {
+ new Thread(new Runnable() {
+public void run() {
+ try {
+drillbit.close();
+ } catch (Exception e) {
+e.printStackTrace();
--- End diff --
There are at least 7 usages in code all should be removed.
---
[GitHub] drill pull request #921: DRILL-4286 Graceful shutdown of drillbit
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/921#discussion_r15078
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/DrillRoot.java
---
@@ -315,7 +481,12 @@ public int compareTo(DrillbitInfo drillbitToCompare) {
if (this.isVersionMatch() == drillbitToCompare.isVersionMatch()) {
if (this.version.equals(drillbitToCompare.getVersion())) {
- return this.address.compareTo(drillbitToCompare.getAddress());
+ {
+if(this.address.equals(drillbitToCompare.getAddress())) {
--- End diff --
`if (`.
---
Re: [DISCUSS] Drill 1.12.0 release
Current status, we are close to the code freeze which will happen not later then the end of the next week. Blocker: DRILL-5917: Ban org.json:json library in Drill (developer - Vlad R., code reviewer - Arina) - in progress. Targeted for 1.12 release: DRILL-4779: Kafka storage plugin support (developer - Anil & Kamesh, code reviewer - Paul) - needs next round of code review. DRILL-5943: Avoid the strong check introduced by DRILL-5582 for PLAIN mechanism (developer - Sorabh, code reviewer - Parth & Laurent) - waiting for Parth code review. DRILL-5771: Fix serDe errors for format plugins (developer - Arina, code reviewer - Tim) - code review is done, waiting for the merge. Kind regards On Fri, Nov 10, 2017 at 9:32 AM, Chunhui Shi wrote: > Hi Arina, > > > Could we consider to include DRILL-5089 in 1.12.0? It is about lazy > loading schema for storage plugins. Could you or Paul take a look at the > pull request for this JIRA https://github.com/apache/drill/pull/1032? I > think both of you are familiar with this part. > > > Thanks, > > > Chunhui > > > From: Arina Yelchiyeva > Sent: Thursday, November 9, 2017 8:11:35 AM > To: dev@drill.apache.org > Subject: Re: [DISCUSS] Drill 1.12.0 release > > Yes, they are already in master. > > On Thu, Nov 9, 2017 at 6:05 PM, Charles Givre wrote: > > > We’re including the Networking functions in this release right? > > > > > On Nov 9, 2017, at 11:04, Arina Yelchiyeva > > > wrote: > > > > > > If changes will be done before cut off date, targeting mid November > that > > it > > > will be possible to include this Jira. > > > > > > On Thu, Nov 9, 2017 at 6:03 PM, Charles Givre > wrote: > > > > > >> Hi Arina, > > >> Can we include DRILL-4091 Support for additional GIS operations in > > version > > >> 1.12? In general the code looked pretty good. There was a unit test > > >> missing which the developer submitted and some minor formatting issues > > >> which I’m still waiting on. > > >> Thanks, > > >> —C > > >> > > >> > > >> > > >>> On Nov 9, 2017, at 10:58, Arina Yelchiyeva < > arina.yelchiy...@gmail.com > > > > > >> wrote: > > >>> > > >>> Current status: > > >>> > > >>> Blocker: > > >>> DRILL-5917: Ban org.json:json library in Drill (developer - Vlad R., > > code > > >>> reviewer - ?) - in progress. > > >>> > > >>> Targeted for 1.12 release: > > >>> DRILL-5337: OpenTSDB plugin (developer - Dmitriy & Vlad S., code > > >> reviewer - > > >>> Arina) - code review in final stage. > > >>> DRILL-4779: Kafka storage plugin support (developer - Anil & Kamesh, > > code > > >>> reviewer - Paul) - in review. > > >>> DRILL-5943: Avoid the strong check introduced by DRILL-5582 for PLAIN > > >>> mechanism (developer - Sorabh, code reviewer - Parth & Laurent) - > > waiting > > >>> for the code review. > > >>> DRILL-5771: Fix serDe errors for format plugins (developer - Arina, > > code > > >>> reviewer - Tim) - waiting for the code review. > > >>> > > >>> Kind regards > > >>> Arina > > >>> > > >>> On Fri, Oct 20, 2017 at 1:49 PM, Arina Yelchiyeva < > > >>> arina.yelchiy...@gmail.com> wrote: > > >>> > > Current status: > > > > Targeted for 1.12 release: > > DRILL-5832: Migrate OperatorFixture to use SystemOptionManager > rather > > >> than > > mock (developer - Paul, code reviewer - ?) - waiting for the code > > review > > DRILL-5842: Refactor and simplify the fragment, operator contexts > for > > testing (developer - Paul, code reviewer - ?) - waiting for the code > > review > > DRILL-5834: Adding network functions (developer - Charles, code > > reviewer > > - Arina) - waiting changes after code review > > DRILL-5337: OpenTSDB plugin (developer - Dmitriy, code reviewer - > > >> Arina) - waiting > > for the code review > > DRILL-5772: Enable UTF-8 support in query string by default > > (developer - > > Arina, code reviewer - Paul) - finalizing approach > > DRILL-4779: Kafka storage plugin support (developer - Anil, code > > >> reviewer > > - ?) - finishing implementation > > > > Under question: > > DRILL-4286: Graceful shutdown of drillbit (developer - Jyothsna, > code > > reviewer - ?) - waiting for the status update from the developer > > > > Please free to suggest other items that are targeted for 1.12 > release. > > There are many Jiras that have fix version marked as 1.12, it would > be > > >> good > > if developers revisit them and update fix version to the actual one. > > Link to the dashboard - https://issues.apache.org/ > > jira/secure/RapidBoard.jspa?rapidView=185&projectKey= > > DRILL&view=detail > > > > Kind regards > > Arina > > > > > > On Wed, Oct 11, 2017 at 2:42 AM, Parth Chandra > > >> wrote: > > > > > I'm waiting to merge the SSL changes in. Waiting a couple of days > > >> more to > > > see if there are any more comments before I merge the changes in. > > > > >
[GitHub] drill pull request #1036: DRILL-5962: Adding ST_AsJSON functionality
Github user ChrisSandison commented on a diff in the pull request:
https://github.com/apache/drill/pull/1036#discussion_r151142574
--- Diff:
contrib/gis/src/main/java/org/apache/drill/exec/expr/fn/impl/gis/STAsJSON.java
---
@@ -0,0 +1,58 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.expr.fn.impl.gis;
+
+import javax.inject.Inject;
+
+import org.apache.drill.exec.expr.DrillSimpleFunc;
+import org.apache.drill.exec.expr.annotations.FunctionTemplate;
+import org.apache.drill.exec.expr.annotations.Output;
+import org.apache.drill.exec.expr.annotations.Param;
+import org.apache.drill.exec.expr.holders.VarBinaryHolder;
+import org.apache.drill.exec.expr.holders.VarCharHolder;
+
+import io.netty.buffer.DrillBuf;
+
+@FunctionTemplate(name = "st_asjson", scope =
FunctionTemplate.FunctionScope.SIMPLE,
+ nulls = FunctionTemplate.NullHandling.NULL_IF_NULL)
+public class STAsJSON implements DrillSimpleFunc {
+ @Param
+ VarBinaryHolder geom1Param;
+
+ @Output
+ VarCharHolder out;
+
+ @Inject
+ DrillBuf buffer;
+
+ public void setup() {
+ }
+
+ public void eval() {
+com.esri.core.geometry.ogc.OGCGeometry geom1 =
com.esri.core.geometry.ogc.OGCGeometry
+ .fromBinary(geom1Param.buffer.nioBuffer(geom1Param.start,
geom1Param.end - geom1Param.start));
+
+String json = geom1.asJson();
+
+int outputSize = json.getBytes().length;
--- End diff --
Good call, I'm going to make that same change in #1034
---
[GitHub] drill pull request #1036: DRILL-5962: Adding ST_AsJSON functionality
Github user arina-ielchiieva commented on a diff in the pull request: https://github.com/apache/drill/pull/1036#discussion_r151143439 --- Diff: contrib/gis/src/main/java/org/apache/drill/exec/expr/fn/impl/gis/STAsJSON.java --- @@ -0,0 +1,58 @@ +/** --- End diff -- @ChrisSandison Apache header should be in a from of comment. Please here and in other PR. ---
[GitHub] drill pull request #1036: DRILL-5962: Adding ST_AsJSON functionality
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/1036#discussion_r151184263
--- Diff:
contrib/gis/src/main/java/org/apache/drill/exec/expr/fn/impl/gis/STAsJSON.java
---
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.expr.fn.impl.gis;
+
+import javax.inject.Inject;
+
+import org.apache.drill.exec.expr.DrillSimpleFunc;
+import org.apache.drill.exec.expr.annotations.FunctionTemplate;
+import org.apache.drill.exec.expr.annotations.Output;
+import org.apache.drill.exec.expr.annotations.Param;
+import org.apache.drill.exec.expr.holders.VarBinaryHolder;
+import org.apache.drill.exec.expr.holders.VarCharHolder;
+
+import io.netty.buffer.DrillBuf;
+
+@FunctionTemplate(name = "st_asjson", scope =
FunctionTemplate.FunctionScope.SIMPLE,
+ nulls = FunctionTemplate.NullHandling.NULL_IF_NULL)
+public class STAsJSON implements DrillSimpleFunc {
--- End diff --
`StAsJson`
---
[GitHub] drill pull request #1036: DRILL-5962: Adding ST_AsJSON functionality
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/1036#discussion_r151184326
--- Diff:
contrib/gis/src/main/java/org/apache/drill/exec/expr/fn/impl/gis/STAsJSON.java
---
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.expr.fn.impl.gis;
+
+import javax.inject.Inject;
+
+import org.apache.drill.exec.expr.DrillSimpleFunc;
+import org.apache.drill.exec.expr.annotations.FunctionTemplate;
+import org.apache.drill.exec.expr.annotations.Output;
+import org.apache.drill.exec.expr.annotations.Param;
+import org.apache.drill.exec.expr.holders.VarBinaryHolder;
+import org.apache.drill.exec.expr.holders.VarCharHolder;
+
+import io.netty.buffer.DrillBuf;
+
+@FunctionTemplate(name = "st_asjson", scope =
FunctionTemplate.FunctionScope.SIMPLE,
+ nulls = FunctionTemplate.NullHandling.NULL_IF_NULL)
+public class STAsJSON implements DrillSimpleFunc {
+ @Param
+ VarBinaryHolder geom1Param;
+
+ @Output
+ VarCharHolder out;
+
+ @Inject
+ DrillBuf buffer;
+
+ public void setup() {
+ }
+
+ public void eval() {
+com.esri.core.geometry.ogc.OGCGeometry geom1 =
com.esri.core.geometry.ogc.OGCGeometry
--- End diff --
`geom`
---
[GitHub] drill pull request #1036: DRILL-5962: Adding ST_AsJSON functionality
Github user arina-ielchiieva commented on a diff in the pull request: https://github.com/apache/drill/pull/1036#discussion_r151184182 --- Diff: contrib/gis/src/main/java/org/apache/drill/exec/expr/fn/impl/gis/STAsJSON.java --- @@ -0,0 +1,59 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.drill.exec.expr.fn.impl.gis; + +import javax.inject.Inject; + +import org.apache.drill.exec.expr.DrillSimpleFunc; +import org.apache.drill.exec.expr.annotations.FunctionTemplate; +import org.apache.drill.exec.expr.annotations.Output; +import org.apache.drill.exec.expr.annotations.Param; +import org.apache.drill.exec.expr.holders.VarBinaryHolder; +import org.apache.drill.exec.expr.holders.VarCharHolder; + +import io.netty.buffer.DrillBuf; + +@FunctionTemplate(name = "st_asjson", scope = FunctionTemplate.FunctionScope.SIMPLE, --- End diff -- 1. `st_as_json` (more sql like way) 2. Please add description of the function in the java doc? ---
[GitHub] drill pull request #1036: DRILL-5962: Adding ST_AsJSON functionality
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/1036#discussion_r151184679
--- Diff:
contrib/gis/src/test/java/org/apache/drill/exec/expr/fn/impl/gis/TestGeometryFunctions.java
---
@@ -49,6 +50,28 @@ public void testGeometryPointCreation() throws Exception
{
}
@Test
+ public void testJSONFromPointCreation() throws Exception {
+testBuilder()
+ .sqlQuery("select ST_AsJSON(ST_Point(-121.895, 37.339)) "
++ "from cp.`/sample-data/CA-cities.csv` limit 1")
+ .ordered().baselineColumns("EXPR$0")
--- End diff --
Please put `baselineColumns("EXPR$0")` to the new line.
---
[GitHub] drill pull request #1036: DRILL-5962: Adding ST_AsJSON functionality
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/1036#discussion_r151184397
--- Diff:
contrib/gis/src/main/java/org/apache/drill/exec/expr/fn/impl/gis/STAsJSON.java
---
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.expr.fn.impl.gis;
+
+import javax.inject.Inject;
+
+import org.apache.drill.exec.expr.DrillSimpleFunc;
+import org.apache.drill.exec.expr.annotations.FunctionTemplate;
+import org.apache.drill.exec.expr.annotations.Output;
+import org.apache.drill.exec.expr.annotations.Param;
+import org.apache.drill.exec.expr.holders.VarBinaryHolder;
+import org.apache.drill.exec.expr.holders.VarCharHolder;
+
+import io.netty.buffer.DrillBuf;
+
+@FunctionTemplate(name = "st_asjson", scope =
FunctionTemplate.FunctionScope.SIMPLE,
+ nulls = FunctionTemplate.NullHandling.NULL_IF_NULL)
+public class STAsJSON implements DrillSimpleFunc {
+ @Param
+ VarBinaryHolder geom1Param;
--- End diff --
`geomParam`
---
[jira] [Created] (DRILL-5969) unit test should continue to tests of storage plugins even there are some failures in exec
Chunhui Shi created DRILL-5969: -- Summary: unit test should continue to tests of storage plugins even there are some failures in exec Key: DRILL-5969 URL: https://issues.apache.org/jira/browse/DRILL-5969 Project: Apache Drill Issue Type: Bug Reporter: Chunhui Shi We are seeing some random issues in unit tests such as https://issues.apache.org/jira/browse/DRILL-5925. While we should fix these issues, we may want to have different ways to handle such situation: 1, we may want to continue running the unit tests esp. those in storage plugins regardless non-fundamental random failures happened in exec module. 2, we may want to re-run these failed tests individually. If we saw these failed tests failed the first time but passed the second run when it was run individually, we could mark these as 'random failures' and decide to continue the whole set of unit tests. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (DRILL-5970) DrillParquetReader always builds the schema with "OPTIONAL" dataMode columns instead of "REQUIRED" ones
Vitalii Diravka created DRILL-5970:
--
Summary: DrillParquetReader always builds the schema with
"OPTIONAL" dataMode columns instead of "REQUIRED" ones
Key: DRILL-5970
URL: https://issues.apache.org/jira/browse/DRILL-5970
Project: Apache Drill
Issue Type: Bug
Components: Execution - Codegen, Execution - Data Types, Storage -
Parquet
Affects Versions: 1.11.0
Reporter: Vitalii Diravka
Assignee: Vitalii Diravka
The root cause of the issue is that adding REQUIRED (not-nullable) data types
to the container in the all MapWriters is not implemented.
It can lead to get invalid schema.
{code}
0: jdbc:drill:zk=local> CREATE TABLE dfs.tmp.bof_repro_1 as select * from
(select CONVERT_FROM('["hello","hai"]','JSON') AS MYCOL, 'Bucket1' AS Bucket
FROM (VALUES(1)));
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
details.
+---++
| Fragment | Number of records written |
+---++
| 0_0 | 1 |
+---++
1 row selected (2.376 seconds)
{code}
Run from Drill unit test framework (to see "data mode"):
{code}
@Test
public void test() throws Exception {
setColumnWidths(new int[] {25, 25});
List queryDataBatches = testSqlWithResults("select * from
dfs.tmp.bof_repro_1");
printResult(queryDataBatches);
}
1 row(s):
---
| MYCOL | Bucket|
---
| ["hello","hai"] | Bucket1 |
---
Total record count: 1
{code}
{code}
vitalii@vitalii-pc:~/parquet-tools/parquet-mr/parquet-tools/target$ java -jar
parquet-tools-1.6.0rc3-SNAPSHOT.jar schema /tmp/bof_repro_1/0_0_0.parquet
message root {
repeated binary MYCOL (UTF8);
required binary Bucket (UTF8);
}
{code}
To simulate of obtaining the wrong result you can try the query with
aggregation by using a new parquet reader (used by default for complex data
types) and old parquet reader. False "Hash aggregate does not support schema
changes" error will happen.
1) Create two parquet files.
{code}
0: jdbc:drill:schema=dfs> CREATE TABLE dfs.tmp.bof_repro_1 as select * from
(select CONVERT_FROM('["hello","hai"]','JSON') AS MYCOL, 'Bucket1' AS Bucket
FROM (VALUES(1)));
+---++
| Fragment | Number of records written |
+---++
| 0_0 | 1 |
+---++
1 row selected (1.122 seconds)
0: jdbc:drill:schema=dfs> CREATE TABLE dfs.tmp.bof_repro_2 as select * from
(select CONVERT_FROM('[]','JSON') AS MYCOL, 'Bucket1' AS Bucket FROM
(VALUES(1)));
+---++
| Fragment | Number of records written |
+---++
| 0_0 | 1 |
+---++
1 row selected (0.552 seconds)
0: jdbc:drill:schema=dfs> select * from dfs.tmp.bof_repro_2;
{code}
2) Copy the parquet files from bof_repro_1 to bof_repro_2.
{code}
[root@naravm1 ~]# hadoop fs -ls /tmp/bof_repro_1
Found 1 items
-rw-r--r-- 3 mapr mapr415 2017-07-25 11:46
/tmp/bof_repro_1/0_0_0.parquet
[root@naravm1 ~]# hadoop fs -ls /tmp/bof_repro_2
Found 1 items
-rw-r--r-- 3 mapr mapr368 2017-07-25 11:46
/tmp/bof_repro_2/0_0_0.parquet
[root@naravm1 ~]# hadoop fs -cp /tmp/bof_repro_1/0_0_0.parquet
/tmp/bof_repro_2/0_0_1.parquet
[root@naravm1 ~]#
{code}
3) Query the table.
{code}
0: jdbc:drill:schema=dfs> ALTER SESSION SET `planner.enable_streamagg`=false;
+---++
| ok | summary |
+---++
| true | planner.enable_streamagg updated. |
+---++
1 row selected (0.124 seconds)
0: jdbc:drill:schema=dfs> select * from dfs.tmp.bof_repro_2;
+--+--+
| MYCOL | Bucket |
+--+--+
| ["hello","hai"] | Bucket1 |
| null | Bucket1 |
+--+--+
2 rows selected (0.247 seconds)
0: jdbc:drill:schema=dfs> select bucket, count(*) from dfs.tmp.bof_repro_2
group by bucket;
Error: UNSUPPORTED_OPERATION ERROR: Hash aggregate does not support schema
changes
Fragment 0:0
[Error Id: 60f8ada3-5f00-4413-a676-4881fc8cb409 on naravm3:31010]
(state=,code=0)
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] drill pull request #921: DRILL-4286 Graceful shutdown of drillbit
Github user dvjyothsna commented on a diff in the pull request:
https://github.com/apache/drill/pull/921#discussion_r151201163
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/work/WorkManager.java ---
@@ -165,32 +169,59 @@ public DrillbitContext getContext() {
*
* This is intended to be used by {@link
org.apache.drill.exec.server.Drillbit#close()}.
*/
- public void waitToExit() {
+ public void waitToExit(Drillbit bit, boolean forcefulShutdown) {
synchronized(this) {
- if (queries.isEmpty() && runningFragments.isEmpty()) {
+ numOfRunningQueries = queries.size();
+ numOfRunningFragments = runningFragments.size();
--- End diff --
We compare if the remaining queries are going down or if new queries are
added during shutdown.
---
[GitHub] drill pull request #921: DRILL-4286 Graceful shutdown of drillbit
Github user dvjyothsna commented on a diff in the pull request:
https://github.com/apache/drill/pull/921#discussion_r151201233
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/work/WorkManager.java ---
@@ -165,32 +169,59 @@ public DrillbitContext getContext() {
*
* This is intended to be used by {@link
org.apache.drill.exec.server.Drillbit#close()}.
*/
- public void waitToExit() {
+ public void waitToExit(Drillbit bit, boolean forcefulShutdown) {
synchronized(this) {
- if (queries.isEmpty() && runningFragments.isEmpty()) {
+ numOfRunningQueries = queries.size();
+ numOfRunningFragments = runningFragments.size();
+ if ( queries.isEmpty() && runningFragments.isEmpty()) {
--- End diff --
We compare if the remaining queries are going down or if new queries are
added during shutdown.
---
[GitHub] drill issue #1023: DRILL-5922 Fixed Child Allocator Leak. DRILL-5926 Stabali...
Github user ilooner commented on the issue: https://github.com/apache/drill/pull/1023 @arina-ielchiieva Spoke with @paul-rogers . This will likely introduce some conflicts for https://github.com/apache/drill/pull/921 . In an effort to lighten the workload for that contributor we'll delay merging this change. I'll remove the ready-to-commit label on the jira until the other PR is merged. After https://github.com/apache/drill/pull/921 is merged I'll rebase and mark this as read-to-commit once more. ---
[GitHub] drill pull request #921: DRILL-4286 Graceful shutdown of drillbit
Github user dvjyothsna commented on a diff in the pull request: https://github.com/apache/drill/pull/921#discussion_r151246632 --- Diff: protocol/src/main/java/org/apache/drill/exec/proto/beans/RpcType.java --- @@ -25,28 +25,8 @@ HANDSHAKE(0), ACK(1), GOODBYE(2), -RUN_QUERY(3), -CANCEL_QUERY(4), -REQUEST_RESULTS(5), -RESUME_PAUSED_QUERY(11), -GET_QUERY_PLAN_FRAGMENTS(12), -GET_CATALOGS(14), -GET_SCHEMAS(15), -GET_TABLES(16), -GET_COLUMNS(17), -CREATE_PREPARED_STATEMENT(22), -GET_SERVER_META(8), -QUERY_DATA(6), -QUERY_HANDLE(7), -QUERY_PLAN_FRAGMENTS(13), -CATALOGS(18), -SCHEMAS(19), -TABLES(20), -COLUMNS(21), -PREPARED_STATEMENT(23), -SERVER_META(9), -QUERY_RESULT(10), -SASL_MESSAGE(24); +REQ_RECORD_BATCH(3), +SASL_MESSAGE(4); --- End diff -- This is not my change. This file is generated by protobuf. When I looked into the issue I found out that user.proto and BitData.proto, Bitcontrol.proto have same enum (RpcType) . That is causing the issue. ---
[jira] [Created] (DRILL-5971) Fix INT64, INT32 logical types in complex parquet reader
Parth Chandra created DRILL-5971: Summary: Fix INT64, INT32 logical types in complex parquet reader Key: DRILL-5971 URL: https://issues.apache.org/jira/browse/DRILL-5971 Project: Apache Drill Issue Type: Bug Reporter: Parth Chandra The 'complex' Parquet reader does not recognize the Parquet logical types INT64, and INT32. Should be a simple change to add these logical types. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] drill pull request #1037: DRILL-5968: Add support for empty service_host use...
Github user parthchandra commented on a diff in the pull request:
https://github.com/apache/drill/pull/1037#discussion_r151260752
--- Diff: contrib/native/client/src/include/drill/userProperties.hpp ---
@@ -28,6 +28,20 @@ class DECLSPEC_DRILL_CLIENT DrillUserProperties{
static const std::map USER_PROPERTIES;
DrillUserProperties(){};
+
+/// @brief Insert or update property value associate with the
property key if the value is
+/// empty or key is undefined. This function is useful for setting
default property.
+///
+/// @param in_propName The property name.
+/// @param in_propValue The property value.
+void SetDefaultProperty(const std::string& in_propName, const
std::string& in_propValue)
--- End diff --
Follow naming convention setDefaultProperty instead of SetDefaultProperty
---
[GitHub] drill pull request #1037: DRILL-5968: Add support for empty service_host use...
Github user parthchandra commented on a diff in the pull request:
https://github.com/apache/drill/pull/1037#discussion_r151260565
--- Diff: contrib/native/client/src/include/drill/userProperties.hpp ---
@@ -28,6 +28,20 @@ class DECLSPEC_DRILL_CLIENT DrillUserProperties{
static const std::map USER_PROPERTIES;
DrillUserProperties(){};
+
+/// @brief Insert or update property value associate with the
property key if the value is
+/// empty or key is undefined. This function is useful for setting
default property.
+///
+/// @param in_propName The property name.
+/// @param in_propValue The property value.
+void SetDefaultProperty(const std::string& in_propName, const
std::string& in_propValue)
+{
+// If the element value is empty (newly inserted or user
defined), then update.
+if (m_properties[in_propName].empty())
--- End diff --
I would have preferred to use isPropSet to see if the property value is
set. Operator[] for a map is intended only to add new values. I believe this
might add a new empty element to the map.
---
[GitHub] drill pull request #1036: DRILL-5962: Adding ST_AsJSON functionality
Github user brendanstennett commented on a diff in the pull request:
https://github.com/apache/drill/pull/1036#discussion_r151267136
--- Diff:
contrib/gis/src/main/java/org/apache/drill/exec/expr/fn/impl/gis/STAsJson.java
---
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/*
+ * Wrapper for ESRI ST_AsJson to convert geometry into REST Json.
+ * Emulates functionality from spatial-framework-for-hadoop.
+ */
+package org.apache.drill.exec.expr.fn.impl.gis;
+
+import javax.inject.Inject;
+
+import org.apache.drill.exec.expr.DrillSimpleFunc;
+import org.apache.drill.exec.expr.annotations.FunctionTemplate;
+import org.apache.drill.exec.expr.annotations.Output;
+import org.apache.drill.exec.expr.annotations.Param;
+import org.apache.drill.exec.expr.holders.VarBinaryHolder;
+import org.apache.drill.exec.expr.holders.VarCharHolder;
+
+import io.netty.buffer.DrillBuf;
+
+@FunctionTemplate(name = "st_as_json", scope =
FunctionTemplate.FunctionScope.SIMPLE,
+ nulls = FunctionTemplate.NullHandling.NULL_IF_NULL)
+public class STAsJson implements DrillSimpleFunc {
--- End diff --
@ChrisSandison I would suggest naming this `ST_AsGeoJSON` to stay in line
with the conventions set forth in PostGIS.
http://www.postgis.net/docs/ST_AsGeoJSON.html
---
[GitHub] drill pull request #1037: DRILL-5968: Add support for empty service_host use...
Github user superbstreak commented on a diff in the pull request:
https://github.com/apache/drill/pull/1037#discussion_r151282112
--- Diff: contrib/native/client/src/include/drill/userProperties.hpp ---
@@ -28,6 +28,20 @@ class DECLSPEC_DRILL_CLIENT DrillUserProperties{
static const std::map USER_PROPERTIES;
DrillUserProperties(){};
+
+/// @brief Insert or update property value associate with the
property key if the value is
+/// empty or key is undefined. This function is useful for setting
default property.
+///
+/// @param in_propName The property name.
+/// @param in_propValue The property value.
+void SetDefaultProperty(const std::string& in_propName, const
std::string& in_propValue)
+{
+// If the element value is empty (newly inserted or user
defined), then update.
+if (m_properties[in_propName].empty())
--- End diff --
If the value doesn't exist, the original implementation with the
setProperty(2) would also insert a new key. Not sure if that was intended or
not.
---
[GitHub] drill pull request #1037: DRILL-5968: Add support for empty service_host use...
Github user superbstreak commented on a diff in the pull request:
https://github.com/apache/drill/pull/1037#discussion_r151282197
--- Diff: contrib/native/client/src/include/drill/userProperties.hpp ---
@@ -28,6 +28,20 @@ class DECLSPEC_DRILL_CLIENT DrillUserProperties{
static const std::map USER_PROPERTIES;
DrillUserProperties(){};
+
+/// @brief Insert or update property value associate with the
property key if the value is
+/// empty or key is undefined. This function is useful for setting
default property.
+///
+/// @param in_propName The property name.
+/// @param in_propValue The property value.
+void SetDefaultProperty(const std::string& in_propName, const
std::string& in_propValue)
--- End diff --
gotcha thanks!
---
[GitHub] drill pull request #1024: DRILL-3640: Support JDBC Statement.setQueryTimeout...
Github user kkhatua commented on a diff in the pull request:
https://github.com/apache/drill/pull/1024#discussion_r151286559
--- Diff:
exec/jdbc/src/main/java/org/apache/drill/jdbc/impl/DrillResultSetImpl.java ---
@@ -96,6 +105,14 @@ private void throwIfClosed() throws
AlreadyClosedSqlException,
throw new AlreadyClosedSqlException( "ResultSet is already
closed." );
}
}
+
+//Implicit check for whether timeout is set
+if (elapsedTimer != null) {
--- End diff --
So, I used your approach and made a modification to injecting the pause for
`sending-data` () in the `ScreenCreator` class.
```
org.apache.drill.jdbc.SqlTimeoutException: Query timed out in 2 seconds
at
org.apache.drill.jdbc.impl.DrillCursor$ResultsListener.awaitFirstMessage(DrillCursor.java:156)
at
org.apache.drill.jdbc.impl.DrillCursor.loadInitialSchema(DrillCursor.java:588)
at
org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:1917)
at
org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:65)
at
org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:473)
at
org.apache.drill.jdbc.impl.DrillMetaImpl.prepareAndExecute(DrillMetaImpl.java:1100)
at
org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:477)
at
org.apache.drill.jdbc.impl.DrillConnectionImpl.prepareAndExecuteInternal(DrillConnectionImpl.java:191)
at
org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:110)
at
org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:130)
at
org.apache.drill.jdbc.impl.DrillStatementImpl.executeQuery(DrillStatementImpl.java:112)
at
org.apache.drill.jdbc.StatementTest.testServerTriggeredQueryTimeout(StatementTest.java:206)
```
However, since the executeQuery() call failed, I don't have a queryId to
unpause at the injection.
Closing the (test-specific) connection did not help release back memory.
```
15:43:41.818 [Thread-8] ERROR o.a.d.exec.server.BootStrapContext - Pool did
not terminate
15:43:41.836 [Thread-8] ERROR o.a.d.exec.server.BootStrapContext - Error
while closing
java.lang.IllegalStateException: Allocator[ROOT] closed with outstanding
child allocators.
Allocator(ROOT) 0/30180480/33376320/3221225472 (res/actual/peak/limit)
child allocators: 2
Allocator(query:25f32edb-db10-4f8a-6a9c-58b0b455c0ab)
1048576/0/0/268435456 (res/actual/peak/limit)
at
org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:496)
~[drill-memory-base-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:76)
[drill-common-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:64)
[drill-common-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
org.apache.drill.exec.server.BootStrapContext.close(BootStrapContext.java:256)
~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:76)
[drill-common-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:64)
[drill-common-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:205)
[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
org.apache.drill.jdbc.impl.DrillConnectionImpl.cleanup(DrillConnectionImpl.java:817)
[classes/:na]
at
org.apache.drill.jdbc.impl.DrillHandler.onConnectionClose(DrillHandler.java:36)
[classes/:na]
at
org.apache.calcite.avatica.AvaticaConnection.close(AvaticaConnection.java:156)
[calcite-avatica-1.4.0-drill-r23.jar:1.4.0-drill-r23]
at
org.apache.drill.jdbc.StatementTest.testServerTriggeredQueryTimeout(StatementTest.java:225)
[test-classes/:na]
```
Any idea on how I can _resume_ such a query without the queryId?
---
[GitHub] drill pull request #1032: DRILL-5089: Dynamically load schema of storage plu...
Github user chunhui-shi commented on a diff in the pull request:
https://github.com/apache/drill/pull/1032#discussion_r151298428
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSystemSchemaFactory.java
---
@@ -73,9 +87,10 @@ public void registerSchemas(SchemaConfig schemaConfig,
SchemaPlus parent) throws
public FileSystemSchema(String name, SchemaConfig schemaConfig) throws
IOException {
super(ImmutableList.of(), name);
+ final DrillFileSystem fs =
ImpersonationUtil.createFileSystem(schemaConfig.getUserName(),
plugin.getFsConf());
for(WorkspaceSchemaFactory f : factories){
-if (f.accessible(schemaConfig.getUserName())) {
- WorkspaceSchema s = f.createSchema(getSchemaPath(),
schemaConfig);
+WorkspaceSchema s = f.createSchema(getSchemaPath(), schemaConfig,
fs);
+if ( s != null) {
--- End diff --
'factories' is from storage plugin, so it will be identical when we don't
update this storage plugin. This FileSystemSchema constructor will be called
only once for a query if this FileSystem storage plugin is needed for this
query.
---
[GitHub] drill pull request #1032: DRILL-5089: Dynamically load schema of storage plu...
Github user chunhui-shi commented on a diff in the pull request:
https://github.com/apache/drill/pull/1032#discussion_r151299650
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/store/SchemaTreeProvider.java
---
@@ -105,12 +106,36 @@ public SchemaPlus createRootSchema(final String
userName, final SchemaConfigInfo
* @return
*/
public SchemaPlus createRootSchema(SchemaConfig schemaConfig) {
+ final SchemaPlus rootSchema =
DynamicSchema.createRootSchema(dContext.getStorage(), schemaConfig);
+ schemaTreesToClose.add(rootSchema);
+ return rootSchema;
+ }
+
+ /**
+ * Return full root schema with schema owner as the given user.
+ *
+ * @param userName Name of the user who is accessing the storage sources.
+ * @param provider {@link SchemaConfigInfoProvider} instance
+ * @return Root of the schema tree.
+ */
+ public SchemaPlus createFullRootSchema(final String userName, final
SchemaConfigInfoProvider provider) {
+final String schemaUser = isImpersonationEnabled ? userName :
ImpersonationUtil.getProcessUserName();
--- End diff --
not that many places, and need to pass in isImpersonationEnabled and
userName if this line became a standalone method. will keep it as is for now.
---
[jira] [Created] (DRILL-5972) Slow performance for query on INFORMATION_SCHEMA.TABLE
Padma Penumarthy created DRILL-5972: --- Summary: Slow performance for query on INFORMATION_SCHEMA.TABLE Key: DRILL-5972 URL: https://issues.apache.org/jira/browse/DRILL-5972 Project: Apache Drill Issue Type: Bug Components: Storage - Information Schema Affects Versions: 1.11.0 Reporter: Padma Penumarthy Assignee: Padma Penumarthy Fix For: 1.13.0 A query like the following on INFORMATION_SCHEMA takes a long time to execute. select TABLE_CATALOG, TABLE_SCHEMA, TABLE_NAME, TABLE_TYPE from INFORMATION_SCHEMA.`TABLES` WHERE TABLE_NAME LIKE '%' AND ( TABLE_SCHEMA = 'hive.default' ) ORDER BY TABLE_TYPE, TABLE_CATALOG, TABLE_SCHEMA, TABLE_NAME; Reason being we fetch table information for all schemas instead of just 'hive.default' schema. If we change the predicate like this, it executes very fast. select TABLE_CATALOG, TABLE_SCHEMA, TABLE_NAME, TABLE_TYPE from INFORMATION_SCHEMA.`TABLES` WHERE ( TABLE_SCHEMA = 'hive.default' ) AND TABLE_NAME LIKE '%' ORDER BY TABLE_TYPE, TABLE_CATALOG, TABLE_SCHEMA, TABLE_NAME; The difference is in the order in which we evaluate the expressions in the predicate. In the first case, we first evaluate TABLE_NAME LIKE '%' and decide that it is inconclusive (since we do not know the schema). So, we go get all tables for all the schemas. In the second case, we first evaluate TABLE_SCHEMA = 'hive.default' and decide that we need to fetch only tables for that schema. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] drill pull request #1032: DRILL-5089: Dynamically load schema of storage plu...
Github user chunhui-shi commented on a diff in the pull request:
https://github.com/apache/drill/pull/1032#discussion_r151301607
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/WorkspaceSchemaFactory.java
---
@@ -150,14 +152,30 @@ public WorkspaceSchemaFactory(
* @return True if the user has access. False otherwise.
*/
public boolean accessible(final String userName) throws IOException {
-final FileSystem fs = ImpersonationUtil.createFileSystem(userName,
fsConf);
+final DrillFileSystem fs =
ImpersonationUtil.createFileSystem(userName, fsConf);
+return accessible(fs);
+ }
+
+ /**
+ * Checks whether a FileSystem object has the permission to list/read
workspace directory
+ * @param fs a DrillFileSystem object that was created with certain user
privilege
+ * @return True if the user has access. False otherwise.
+ * @throws IOException
+ */
+ public boolean accessible(DrillFileSystem fs) throws IOException {
try {
- // We have to rely on the listStatus as a FileSystem can have
complicated controls such as regular unix style
- // permissions, Access Control Lists (ACLs) or Access Control
Expressions (ACE). Hadoop 2.7 version of FileSystem
- // has a limited private API (FileSystem.access) to check the
permissions directly
- // (see https://issues.apache.org/jira/browse/HDFS-6570). Drill
currently relies on Hadoop 2.5.0 version of
- // FileClient. TODO: Update this when DRILL-3749 is fixed.
- fs.listStatus(wsPath);
+ /**
+ * For Windows local file system, fs.access ends up using
DeprecatedRawLocalFileStatus which has
+ * TrustedInstaller as owner, and a member of Administrators group
could not satisfy the permission.
+ * In this case, we will still use method listStatus.
+ * In other cases, we use access method since it is cheaper.
+ */
+ if (SystemUtils.IS_OS_WINDOWS &&
fs.getUri().getScheme().equalsIgnoreCase("file")) {
--- End diff --
FileSystem in hdfs has a constant DEFAULT_FS "file:///", for now I will
define our own.
---
[GitHub] drill pull request #1032: DRILL-5089: Dynamically load schema of storage plu...
Github user chunhui-shi commented on a diff in the pull request:
https://github.com/apache/drill/pull/1032#discussion_r151302799
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DynamicRootSchema.java
---
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql;
+
+import com.google.common.collect.ImmutableSortedSet;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+import org.apache.calcite.DataContext;
+import org.apache.calcite.jdbc.CalciteRootSchema;
+import org.apache.calcite.jdbc.CalciteSchema;
+
+import org.apache.calcite.linq4j.tree.Expression;
+import org.apache.calcite.linq4j.tree.Expressions;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.calcite.schema.impl.AbstractSchema;
+import org.apache.calcite.util.BuiltInMethod;
+import org.apache.calcite.util.Compatible;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.StoragePlugin;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.exec.store.SubSchemaWrapper;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.NavigableSet;
+import java.util.Set;
+
+public class DynamicRootSchema extends DynamicSchema
+implements CalciteRootSchema {
+
+ /** Creates a root schema. */
+ DynamicRootSchema(StoragePluginRegistry storages, SchemaConfig
schemaConfig) {
+super(null, new RootSchema(), "");
+this.schemaConfig = schemaConfig;
+this.storages = storages;
+ }
+
+ @Override
+ public CalciteSchema getSubSchema(String schemaName, boolean
caseSensitive) {
+CalciteSchema retSchema = getSubSchemaMap().get(schemaName);
+
+if (retSchema == null) {
+ loadSchemaFactory(schemaName, caseSensitive);
+}
+
+retSchema = getSubSchemaMap().get(schemaName);
+return retSchema;
+ }
+
+ @Override
+ public NavigableSet getTableNames() {
+Set pluginNames = Sets.newHashSet();
--- End diff --
plugin name in drill is case sensitive.
---
[GitHub] drill pull request #1038: DRILL-5972: Slow performance for query on INFORMAT...
GitHub user ppadma opened a pull request: https://github.com/apache/drill/pull/1038 DRILL-5972: Slow performance for query on INFORMATION_SCHEMA.TABLE Please see DRILL-5972. Problem is while evaluating "boolean and", we are returning as soon as expression result is not TRUE. We should go through all the expressions and if any of them is FALSE, return FALSE. If any of them is INCONCLUSIVE, return INCONCLUSIVE. Tested the fix with lot of hive tables and verified that it fixes the performance problem. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ppadma/drill DRILL-5972 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/1038.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1038 commit 70f489098acc94d9e1af1aa05c35e2dfe01198c6 Author: Padma Penumarthy Date: 2017-11-16T04:31:23Z DRILL-5972: Slow performance for query on INFORMATION_SCHEMA.TABLE ---
[GitHub] drill pull request #1039: DRILL-5584: Add branding and versioning informatio...
GitHub user superbstreak opened a pull request: https://github.com/apache/drill/pull/1039 DRILL-5584: Add branding and versioning information for windows C++ C⦠â¦lient. You can merge this pull request into a Git repository by running: $ git pull https://github.com/superbstreak/drill DRILL-5584 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/1039.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1039 commit 7792fbfa11dc9a622217eb9a8f4644802e4e5e44 Author: Rob Wu Date: 2017-11-16T02:21:48Z DRILL-5584: Add branding and versioning information for windows C++ Client. ---
[GitHub] drill pull request #1039: DRILL-5584: Add branding and versioning informatio...
Github user superbstreak commented on a diff in the pull request: https://github.com/apache/drill/pull/1039#discussion_r151329816 --- Diff: contrib/native/client/src/clientlib/env.h.in --- @@ -30,6 +30,11 @@ #define GIT_SHA_PROP @GIT_SHA_PROP@ #define GIT_COMMIT_PROP @GIT_COMMIT_PROP@ +#define DRILL_LEGALCOPYRIGHT_STR"Copyright (c) 2013-2017 The Apache Software Foundation\0" --- End diff -- Reviewer, please make sure this is correct ---
