[GitHub] [drill] jnturton commented on pull request #2547: DRILL-8223: Refactor auth modes dropping DRILL_PROCESS and allowing credential providers everywhere

[GitBox]

jnturton commented on PR #2547:
URL: https://github.com/apache/drill/pull/2547#issuecomment-1129010432

   Now rebased on #2544.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] jnturton merged pull request #2544: DRILL-8220: Add User Translation Support for OAuth Enabled Plugins

[GitBox]

jnturton merged PR #2544:
URL: https://github.com/apache/drill/pull/2544


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] cgivre commented on a diff in pull request #2544: DRILL-8220: Add User Translation Support for OAuth Enabled Plugins

[GitBox]

cgivre commented on code in PR #2544:
URL: https://github.com/apache/drill/pull/2544#discussion_r873608802


##
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/StorageResources.java:
##
@@ -212,61 +195,15 @@ public Response enablePlugin(@PathParam("name") String 
name, @PathParam("val") B
   @Consumes(MediaType.APPLICATION_JSON)
   @Produces(MediaType.APPLICATION_JSON)
   public Response updateRefreshToken(@PathParam("name") String name, 
OAuthTokenContainer tokens) {

Review Comment:
   Done!  



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] jnturton merged pull request #2550: DRILL-8200: Update Hadoop libs to ≥ 3.2.3 for CVE-2022-26612 (set hadoop.version)

[GitBox]

jnturton merged PR #2550:
URL: https://github.com/apache/drill/pull/2550


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] jnturton commented on a diff in pull request #2544: DRILL-8220: Add User Translation Support for OAuth Enabled Plugins

[GitBox]

jnturton commented on code in PR #2544:
URL: https://github.com/apache/drill/pull/2544#discussion_r873581651


##
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/StorageResources.java:
##
@@ -212,61 +195,15 @@ public Response enablePlugin(@PathParam("name") String 
name, @PathParam("val") B
   @Consumes(MediaType.APPLICATION_JSON)
   @Produces(MediaType.APPLICATION_JSON)
   public Response updateRefreshToken(@PathParam("name") String name, 
OAuthTokenContainer tokens) {

Review Comment:
   Thanks, perhaps deprecation is better even though the next version is 2.0



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] cgivre commented on pull request #2544: DRILL-8220: Add User Translation Support for OAuth Enabled Plugins

[GitBox]

cgivre commented on PR #2544:
URL: https://github.com/apache/drill/pull/2544#issuecomment-1127511521

   @jnturton Thanks for the review comments.  I changed the `getActiveUser()` 
function to `getQueryUser()`.   Is there anything else you'd like addressed?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] cgivre commented on a diff in pull request #2544: DRILL-8220: Add User Translation Support for OAuth Enabled Plugins

[GitBox]

cgivre commented on code in PR #2544:
URL: https://github.com/apache/drill/pull/2544#discussion_r873578777


##
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/OAuthRequests.java:
##
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.drill.exec.server.rest;
+
+import okhttp3.OkHttpClient;
+import okhttp3.OkHttpClient.Builder;
+import okhttp3.Request;
+import org.apache.drill.common.logical.CredentialedStoragePluginConfig;
+import org.apache.drill.common.logical.StoragePluginConfig;
+import org.apache.drill.common.logical.StoragePluginConfig.AuthMode;
+import org.apache.drill.common.logical.security.CredentialsProvider;
+import org.apache.drill.exec.oauth.OAuthTokenProvider;
+import org.apache.drill.exec.oauth.PersistentTokenTable;
+import org.apache.drill.exec.oauth.TokenRegistry;
+import org.apache.drill.exec.server.DrillbitContext;
+import org.apache.drill.exec.server.rest.DrillRestServer.UserAuthEnabled;
+import org.apache.drill.exec.server.rest.StorageResources.JsonResult;
+import org.apache.drill.exec.store.AbstractStoragePlugin;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.exec.store.StoragePluginRegistry.PluginException;
+import org.apache.drill.exec.store.http.oauth.OAuthUtils;
+import org.apache.drill.exec.store.security.oauth.OAuthTokenCredentials;
+import org.eclipse.jetty.util.resource.Resource;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.servlet.http.HttpServletRequest;
+import javax.ws.rs.core.Response;
+import javax.ws.rs.core.Response.Status;
+import javax.ws.rs.core.SecurityContext;
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.nio.charset.StandardCharsets;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+public class OAuthRequests {
+
+  private static final Logger logger = 
LoggerFactory.getLogger(OAuthRequests.class);
+  private static final String OAUTH_SUCCESS_PAGE = 
"/rest/storage/success.html";
+
+  public static Response updateAccessToken(String name,
+   OAuthTokenContainer tokens,
+   StoragePluginRegistry storage,
+   UserAuthEnabled authEnabled,
+   SecurityContext sc) {
+try {
+  if (storage.getPlugin(name).getConfig() instanceof 
CredentialedStoragePluginConfig) {
+DrillbitContext context = ((AbstractStoragePlugin) 
storage.getPlugin(name)).getContext();
+OAuthTokenProvider tokenProvider = context.getoAuthTokenProvider();
+PersistentTokenTable tokenTable = 
tokenProvider.getOauthTokenRegistry(getActiveUser(storage.getPlugin(name).getConfig(),
 authEnabled, sc)).getTokenTable(name);
+
+// Set the access token
+tokenTable.setAccessToken(tokens.getAccessToken());
+
+return Response.status(Status.OK)
+  .entity("Access tokens have been updated.")
+  .build();
+  } else {
+logger.error("{} does not support OAuth2.0.  You can only add tokens 
to OAuth enabled plugins.", name);
+return Response.status(Status.INTERNAL_SERVER_ERROR)
+  .entity(message("Unable to add tokens: %s", name))
+  .build();
+  }
+} catch (PluginException e) {
+  logger.error("Error when adding tokens to {}", name);
+  return Response.status(Status.INTERNAL_SERVER_ERROR)
+.entity(message("Unable to add tokens: %s", e.getMessage()))
+.build();
+}
+  }
+
+  public static Response updateRefreshToken(String name, OAuthTokenContainer 
tokens,
+StoragePluginRegistry storage, 
UserAuthEnabled authEnabled,
+SecurityContext sc) {
+try {
+  if (storage.getPlugin(name).getConfig() instanceof 
CredentialedStoragePluginConfig) {
+DrillbitContext context = ((AbstractStoragePlugin) 
storage.getPlugin(name)).getContext();
+OAuthTokenProvider tokenProvider = context.getoAuthTokenProvider();
+PersistentTokenTable t

[GitHub] [drill] cgivre commented on a diff in pull request #2544: DRILL-8220: Add User Translation Support for OAuth Enabled Plugins

[GitBox]

cgivre commented on code in PR #2544:
URL: https://github.com/apache/drill/pull/2544#discussion_r873575838


##
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/PluginConfigWrapper.java:
##
@@ -114,4 +125,75 @@ public boolean isOauth() {
 
 return 
tokenCredentials.map(OAuthTokenCredentials::getClientID).orElse(null) != null;
   }
+
+  @JsonIgnore
+  public String getClientID() {
+CredentialedStoragePluginConfig securedStoragePluginConfig = 
(CredentialedStoragePluginConfig) config;
+CredentialsProvider credentialsProvider = 
securedStoragePluginConfig.getCredentialsProvider();
+
+return credentialsProvider.getCredentials().getOrDefault("clientID", "");
+  }
+
+  /**
+   * This function generates the authorization URI for use when a non-admin 
user is authorizing
+   * OAuth2.0 access for a storage plugin.  This function is necessary as we 
do not wish to expose
+   * any plugin configuration information to the user.
+   *
+   * If the plugin is not OAuth, or is missing components, the function will 
return an empty string.
+   * @return The authorization URI for an OAuth enabled plugin.
+   */
+  @JsonIgnore
+  public String getAuthorizationURIWithParams() {

Review Comment:
   Sounds good.  Just be aware that this code is used in the `list.ftl` for the 
Credentials page.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] cgivre commented on a diff in pull request #2544: DRILL-8220: Add User Translation Support for OAuth Enabled Plugins

[GitBox]

cgivre commented on code in PR #2544:
URL: https://github.com/apache/drill/pull/2544#discussion_r873575333


##
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/StorageResources.java:
##
@@ -212,61 +195,15 @@ public Response enablePlugin(@PathParam("name") String 
name, @PathParam("val") B
   @Consumes(MediaType.APPLICATION_JSON)
   @Produces(MediaType.APPLICATION_JSON)
   public Response updateRefreshToken(@PathParam("name") String name, 
OAuthTokenContainer tokens) {

Review Comment:
   I had debated removing the endpoints in `StorageResources`.  The only reason 
I left them was in case someone was using these endpoints, then we'd have a 
breaking change when they upgraded.   What if I mark them as deprecated?
   
   As a point of clarification, the reason I moved them to 
`CredentialResources` was because the credentials page allows a user to update 
their creds.  The `StorageResources` requires admin access, whereas the 
credentials page only requires an authorized user.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] jnturton commented on a diff in pull request #2544: DRILL-8220: Add User Translation Support for OAuth Enabled Plugins

[GitBox]

jnturton commented on code in PR #2544:
URL: https://github.com/apache/drill/pull/2544#discussion_r873511567


##
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/OAuthRequests.java:
##
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.drill.exec.server.rest;
+
+import okhttp3.OkHttpClient;
+import okhttp3.OkHttpClient.Builder;
+import okhttp3.Request;
+import org.apache.drill.common.logical.CredentialedStoragePluginConfig;
+import org.apache.drill.common.logical.StoragePluginConfig;
+import org.apache.drill.common.logical.StoragePluginConfig.AuthMode;
+import org.apache.drill.common.logical.security.CredentialsProvider;
+import org.apache.drill.exec.oauth.OAuthTokenProvider;
+import org.apache.drill.exec.oauth.PersistentTokenTable;
+import org.apache.drill.exec.oauth.TokenRegistry;
+import org.apache.drill.exec.server.DrillbitContext;
+import org.apache.drill.exec.server.rest.DrillRestServer.UserAuthEnabled;
+import org.apache.drill.exec.server.rest.StorageResources.JsonResult;
+import org.apache.drill.exec.store.AbstractStoragePlugin;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.exec.store.StoragePluginRegistry.PluginException;
+import org.apache.drill.exec.store.http.oauth.OAuthUtils;
+import org.apache.drill.exec.store.security.oauth.OAuthTokenCredentials;
+import org.eclipse.jetty.util.resource.Resource;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.servlet.http.HttpServletRequest;
+import javax.ws.rs.core.Response;
+import javax.ws.rs.core.Response.Status;
+import javax.ws.rs.core.SecurityContext;
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.nio.charset.StandardCharsets;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+public class OAuthRequests {
+
+  private static final Logger logger = 
LoggerFactory.getLogger(OAuthRequests.class);
+  private static final String OAUTH_SUCCESS_PAGE = 
"/rest/storage/success.html";
+
+  public static Response updateAccessToken(String name,
+   OAuthTokenContainer tokens,
+   StoragePluginRegistry storage,
+   UserAuthEnabled authEnabled,
+   SecurityContext sc) {
+try {
+  if (storage.getPlugin(name).getConfig() instanceof 
CredentialedStoragePluginConfig) {
+DrillbitContext context = ((AbstractStoragePlugin) 
storage.getPlugin(name)).getContext();
+OAuthTokenProvider tokenProvider = context.getoAuthTokenProvider();
+PersistentTokenTable tokenTable = 
tokenProvider.getOauthTokenRegistry(getActiveUser(storage.getPlugin(name).getConfig(),
 authEnabled, sc)).getTokenTable(name);
+
+// Set the access token
+tokenTable.setAccessToken(tokens.getAccessToken());
+
+return Response.status(Status.OK)
+  .entity("Access tokens have been updated.")
+  .build();
+  } else {
+logger.error("{} does not support OAuth2.0.  You can only add tokens 
to OAuth enabled plugins.", name);
+return Response.status(Status.INTERNAL_SERVER_ERROR)
+  .entity(message("Unable to add tokens: %s", name))
+  .build();
+  }
+} catch (PluginException e) {
+  logger.error("Error when adding tokens to {}", name);
+  return Response.status(Status.INTERNAL_SERVER_ERROR)
+.entity(message("Unable to add tokens: %s", e.getMessage()))
+.build();
+}
+  }
+
+  public static Response updateRefreshToken(String name, OAuthTokenContainer 
tokens,
+StoragePluginRegistry storage, 
UserAuthEnabled authEnabled,
+SecurityContext sc) {
+try {
+  if (storage.getPlugin(name).getConfig() instanceof 
CredentialedStoragePluginConfig) {
+DrillbitContext context = ((AbstractStoragePlugin) 
storage.getPlugin(name)).getContext();
+OAuthTokenProvider tokenProvider = context.getoAuthTokenProvider();
+PersistentTokenTable

[GitHub] [drill] jnturton opened a new pull request, #2550: DRILL-8200: Update Hadoop libs to ≥ 3.2.3 for CVE-2022-26612

[GitBox]

jnturton opened a new pull request, #2550:
URL: https://github.com/apache/drill/pull/2550

   # [DRILL-8200](https://issues.apache.org/jira/browse/DRILL-8200): Update 
Hadoop libs to ≥ 3.2.3 for CVE-2022-26612
   
   ## Description
   
   The previous PR for this update did everything except set hadoop.version = 
3.23.
   
   ## Documentation
   N/A
   
   ## Testing
   Existing test coverage.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] cgivre commented on pull request #2547: DRILL-8223: Refactor auth modes dropping DRILL_PROCESS and allowing credential providers everywhere

[GitBox]

cgivre commented on PR #2547:
URL: https://github.com/apache/drill/pull/2547#issuecomment-1127089235

   @jnturton Thank you very much for this PR.  Could we merge 
[DRILL-8220](https://github.com/apache/drill/pull/2544) prior to merging this 
PR?   It has some modifications to the HTTP plugin that are relevant. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] luocooong merged pull request #2548: DRILL-8224: Fix TestHttpPlugin#testSlowResponse

[GitBox]

luocooong merged PR #2548:
URL: https://github.com/apache/drill/pull/2548


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] nielsbasjes opened a new pull request, #2549: DRILL-8225: Update LogParser and Yauaa to support User-Agent Client Hints

[GitBox]

nielsbasjes opened a new pull request, #2549:
URL: https://github.com/apache/drill/pull/2549

   # [DRILL-8225](https://issues.apache.org/jira/browse/DRILL-8225): Update 
LogParser and Yauaa to support User-Agent Client Hints
   
   ## Description
   
   In order to support logging, parsing and analyzing the User-Agent Client 
Hints it was needed to update the LogParser to fix a parsing bug and to update 
Yauaa to support actually analyzing them.
   
   See https://yauaa.basjes.nl/using/clienthints/
   
   ## Documentation
   The change includes the updated Readme for the UDFs.
   
   ## Testing
   The change includes the original tests and additional tests for the new 
functionality.
   For some of the existing tests the readability of the test has been improved.
   
   For specific edge cases (like passing a null value and empty string) the 
parse results have changed.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] vvysotskyi merged pull request #2538: DRILL-8214: Replace EnumerableTableScan usage with LogicalTableScan

[GitBox]

vvysotskyi merged PR #2538:
URL: https://github.com/apache/drill/pull/2538


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] vvysotskyi commented on a diff in pull request #2538: DRILL-8214: Replace EnumerableTableScan usage with LogicalTableScan

[GitBox]

vvysotskyi commented on code in PR #2538:
URL: https://github.com/apache/drill/pull/2538#discussion_r872651152


##
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillPushProjectIntoScanRule.java:
##
@@ -46,16 +46,28 @@
 public class DrillPushProjectIntoScanRule extends RelOptRule {
   public static final RelOptRule INSTANCE =
   new DrillPushProjectIntoScanRule(LogicalProject.class,
-  EnumerableTableScan.class,
+  DirPrunedTableScan.class,
   "DrillPushProjectIntoScanRule:enumerable") {
 
 @Override
 protected boolean skipScanConversion(RelDataType projectRelDataType, 
TableScan scan) {
-  // do not allow skipping conversion of EnumerableTableScan to 
DrillScanRel if rule is applicable
+  // do not allow skipping conversion of DirPrunedTableScan to 
DrillScanRel if rule is applicable
   return false;
 }
   };
 
+  public static final RelOptRule LOGICAL_INSTANCE =
+new DrillPushProjectIntoScanRule(LogicalProject.class,
+  LogicalTableScan.class,
+  "DrillPushProjectIntoScanRule:none") {
+
+  @Override
+  protected boolean skipScanConversion(RelDataType projectRelDataType, 
TableScan scan) {
+// do not allow skipping conversion of EnumerableTableScan to 
DrillScanRel if rule is applicable

Review Comment:
   Thanks, fixed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] cgivre commented on a diff in pull request #2547: DRILL-8223: Refactor auth modes dropping DRILL_PROCESS and allowing credential providers everywhere

[GitBox]

cgivre commented on code in PR #2547:
URL: https://github.com/apache/drill/pull/2547#discussion_r872465743


##
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/StorageResources.java:
##
@@ -276,24 +280,24 @@ public Response updateAccessToken(@PathParam("name") 
String name, OAuthTokenCont
   public Response updateOAuthTokens(@PathParam("name") String name,
 OAuthTokenContainer tokenContainer) {
 try {
-  if (storage.getPlugin(name).getConfig() instanceof 
CredentialedStoragePluginConfig) {
-DrillbitContext context = ((AbstractStoragePlugin) 
storage.getPlugin(name)).getContext();
-OAuthTokenProvider tokenProvider = context.getoAuthTokenProvider();
-PersistentTokenTable tokenTable = 
tokenProvider.getOauthTokenRegistry().getTokenTable(name);
-
-// Set the access and refresh token
-tokenTable.setAccessToken(tokenContainer.getAccessToken());
-tokenTable.setRefreshToken(tokenContainer.getRefreshToken());
-
-return Response.status(Status.OK)
-  .entity("Access tokens have been updated.")
-  .build();
-  } else {
+  StoragePluginConfig config = storage.getPlugin(name).getConfig();
+  if (config.getValue("type") != "http") {

Review Comment:
   @jnturton This is actually addressed in 
https://github.com/apache/drill/pull/2544.   I refactored the OAuth stuff so 
that if other plugins needed to use OAuth, they can reuse almost all of what 
Drill already does.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] rakesh8081 commented on issue #2440: IllegalArgumentException: databaseName can not be null - Mongo DB

[GitBox]

rakesh8081 commented on issue #2440:
URL: https://github.com/apache/drill/issues/2440#issuecomment-1125968326

   I faced a exactly same issue. I am running 3 drillbits on a zk cluster and 
configured a mongostorage which is same for all the three drillbits. 
   
   Mongodb is installed on node3 of the cluster. I am using mongo private IP in 
the connection
   I can run a query from node1 and node3  but not node2. However from a linux 
command line I can connect the mongodb from node. 
   I get 
   ``` org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: 
IllegalArgumentException: databaseName can not be null
   ``` 
   
   from node2.
   
   all the three nodes are identical and ports are open among the three.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] jnturton commented on a diff in pull request #2538: DRILL-8214: Replace EnumerableTableScan usage with LogicalTableScan

[GitBox]

jnturton commented on code in PR #2538:
URL: https://github.com/apache/drill/pull/2538#discussion_r872306477


##
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillPushProjectIntoScanRule.java:
##
@@ -46,16 +46,28 @@
 public class DrillPushProjectIntoScanRule extends RelOptRule {
   public static final RelOptRule INSTANCE =
   new DrillPushProjectIntoScanRule(LogicalProject.class,
-  EnumerableTableScan.class,
+  DirPrunedTableScan.class,
   "DrillPushProjectIntoScanRule:enumerable") {
 
 @Override
 protected boolean skipScanConversion(RelDataType projectRelDataType, 
TableScan scan) {
-  // do not allow skipping conversion of EnumerableTableScan to 
DrillScanRel if rule is applicable
+  // do not allow skipping conversion of DirPrunedTableScan to 
DrillScanRel if rule is applicable
   return false;
 }
   };
 
+  public static final RelOptRule LOGICAL_INSTANCE =
+new DrillPushProjectIntoScanRule(LogicalProject.class,
+  LogicalTableScan.class,
+  "DrillPushProjectIntoScanRule:none") {
+
+  @Override
+  protected boolean skipScanConversion(RelDataType projectRelDataType, 
TableScan scan) {
+// do not allow skipping conversion of EnumerableTableScan to 
DrillScanRel if rule is applicable

Review Comment:
   ```suggestion
   // do not allow skipping conversion of LogicalTableScan to 
DrillScanRel if rule is applicable
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] vdiravka commented on a diff in pull request #2547: DRILL-8223: Refactor auth modes dropping DRILL_PROCESS and allowing credential providers everywhere

[GitBox]

vdiravka commented on code in PR #2547:
URL: https://github.com/apache/drill/pull/2547#discussion_r872117809


##
contrib/storage-http/src/test/java/org/apache/drill/exec/store/http/TestHttpPlugin.java:
##
@@ -1112,6 +1112,7 @@ public void testLimitPushdownWithFilter() throws 
Exception {
   }
 
   @Test
+  @Ignore("This test is flaky when run in CI")

Review Comment:
   #2548 should fix this



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] vvysotskyi commented on a diff in pull request #2547: DRILL-8223: Refactor auth modes dropping DRILL_PROCESS and allowing credential providers everywhere

[GitBox]

vvysotskyi commented on code in PR #2547:
URL: https://github.com/apache/drill/pull/2547#discussion_r872035895


##
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/StorageResources.java:
##
@@ -276,24 +280,24 @@ public Response updateAccessToken(@PathParam("name") 
String name, OAuthTokenCont
   public Response updateOAuthTokens(@PathParam("name") String name,
 OAuthTokenContainer tokenContainer) {
 try {
-  if (storage.getPlugin(name).getConfig() instanceof 
CredentialedStoragePluginConfig) {
-DrillbitContext context = ((AbstractStoragePlugin) 
storage.getPlugin(name)).getContext();
-OAuthTokenProvider tokenProvider = context.getoAuthTokenProvider();
-PersistentTokenTable tokenTable = 
tokenProvider.getOauthTokenRegistry().getTokenTable(name);
-
-// Set the access and refresh token
-tokenTable.setAccessToken(tokenContainer.getAccessToken());
-tokenTable.setRefreshToken(tokenContainer.getRefreshToken());
-
-return Response.status(Status.OK)
-  .entity("Access tokens have been updated.")
-  .build();
-  } else {
+  StoragePluginConfig config = storage.getPlugin(name).getConfig();
+  if (config.getValue("type") != "http") {

Review Comment:
   But looks like error message below should be fixed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] vvysotskyi commented on a diff in pull request #2547: DRILL-8223: Refactor auth modes dropping DRILL_PROCESS and allowing credential providers everywhere

[GitBox]

vvysotskyi commented on code in PR #2547:
URL: https://github.com/apache/drill/pull/2547#discussion_r872035428


##
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/StorageResources.java:
##
@@ -276,24 +280,24 @@ public Response updateAccessToken(@PathParam("name") 
String name, OAuthTokenCont
   public Response updateOAuthTokens(@PathParam("name") String name,
 OAuthTokenContainer tokenContainer) {
 try {
-  if (storage.getPlugin(name).getConfig() instanceof 
CredentialedStoragePluginConfig) {
-DrillbitContext context = ((AbstractStoragePlugin) 
storage.getPlugin(name)).getContext();
-OAuthTokenProvider tokenProvider = context.getoAuthTokenProvider();
-PersistentTokenTable tokenTable = 
tokenProvider.getOauthTokenRegistry().getTokenTable(name);
-
-// Set the access and refresh token
-tokenTable.setAccessToken(tokenContainer.getAccessToken());
-tokenTable.setRefreshToken(tokenContainer.getRefreshToken());
-
-return Response.status(Status.OK)
-  .entity("Access tokens have been updated.")
-  .build();
-  } else {
+  StoragePluginConfig config = storage.getPlugin(name).getConfig();
+  if (config.getValue("type") != "http") {

Review Comment:
   The intention was to allow it not only for HTTP but for all other plugins 
that have `CredentialedStoragePluginConfig`, so they theoretically can also use 
OAuth and update tokens.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] vdiravka commented on a diff in pull request #2500: DRILL-8172: Use the specified memory usage for Travis CI

[GitBox]

vdiravka commented on code in PR #2500:
URL: https://github.com/apache/drill/pull/2500#discussion_r871739585


##
.travis.yml:
##
@@ -45,6 +45,9 @@ cache:
 before_install:
   - export JAVA_HOME="/usr/lib/jvm/java-8-openjdk-arm64"
   - export PATH="$JAVA_HOME/bin:$PATH"
+  - export MEMORYMB=2048

Review Comment:
   #2548 
   @luocooong pls check



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] vdiravka opened a new pull request, #2548: DRILL-8224: Fix TestHttpPlugin#testSlowResponse

[GitBox]

vdiravka opened a new pull request, #2548:
URL: https://github.com/apache/drill/pull/2548

   # [DRILL-8224](https://issues.apache.org/jira/browse/DRILL-8224): Fix 
TestHttpPlugin#testSlowResponse
   
   ## Description
   
   Fix testSlowResponse and increase heap memory for Travis build
   
   ## Documentation
   NA
   
   ## Testing
   This chaneg about fixing tests
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] jnturton merged pull request #2535: DRILL-8211: Replace deprecated RelNode.getChildExps with Project.getProjects

[GitBox]

jnturton merged PR #2535:
URL: https://github.com/apache/drill/pull/2535


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] jnturton commented on a diff in pull request #2535: DRILL-8211: Replace deprecated RelNode.getChildExps with Project.getProjects

[GitBox]

jnturton commented on code in PR #2535:
URL: https://github.com/apache/drill/pull/2535#discussion_r871526487


##
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillPushRowKeyJoinToScanRule.java:
##
@@ -463,8 +463,9 @@ private static boolean isRowKeyColumn(int index, RelNode 
rel) {
   }
   // If no exprs present in projection the column index remains the same 
in the child.
   // Otherwise, the column index is the `RexInputRef` index.
-  if (curRel != null && curRel instanceof DrillProjectRel) {
-List childExprs = curRel.getChildExps();
+  if (curRel instanceof DrillProjectRel) {

Review Comment:
   I didn't realise that an `instanceof` already provides a null check, nice.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] vvysotskyi merged pull request #2539: DRILL-8216: Use EVF-based JSON reader for Values operator

[GitBox]

vvysotskyi merged PR #2539:
URL: https://github.com/apache/drill/pull/2539


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] jnturton merged pull request #2545: DRILL-8221: Collect bug fixes from master for Drill 1.20.1

[GitBox]

jnturton merged PR #2545:
URL: https://github.com/apache/drill/pull/2545


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] jnturton merged pull request #2541: DRILL-8218: Add unit tests for StorageResource REST endpoints

[GitBox]

jnturton merged PR #2541:
URL: https://github.com/apache/drill/pull/2541


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] vvysotskyi commented on a diff in pull request #2539: DRILL-8216: Use EVF-based JSON reader for Values operator

[GitBox]

vvysotskyi commented on code in PR #2539:
URL: https://github.com/apache/drill/pull/2539#discussion_r870903809


##
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/scan/v3/schema/SchemaUtils.java:
##
@@ -234,4 +241,80 @@ static void copyProperties(TupleMetadata source,
   dest.setProperty(ScanProjectionParser.PROJECTION_TYPE_PROP, value);
 }
   }
+
+  /**
+   * Converts specified {@code RelDataType relDataType} into {@link 
ColumnMetadata}.
+   * For the case when specified relDataType is struct, map with recursively 
converted children
+   * will be created.
+   *
+   * @param namefiled name
+   * @param relDataType filed type
+   * @return {@link ColumnMetadata} which corresponds to specified {@code 
RelDataType relDataType}
+   */
+  public static ColumnMetadata getColumnMetadata(String name, RelDataType 
relDataType) {
+switch (relDataType.getSqlTypeName()) {
+  case ARRAY:
+return getArrayMetadata(name, relDataType);
+  case MAP:
+  case OTHER:
+throw new UnsupportedOperationException(String.format("Unsupported 
data type: %s", relDataType.getSqlTypeName()));
+  default:
+if (relDataType.isStruct()) {
+  return getStructMetadata(name, relDataType);
+} else {
+  return new PrimitiveColumnMetadata(
+MaterializedField.create(name,
+  
TypeInferenceUtils.getDrillMajorTypeFromCalciteType(relDataType)));
+}
+}
+  }
+
+  /**
+   * Returns {@link ColumnMetadata} instance which corresponds to specified 
array {@code RelDataType relDataType}.
+   *
+   * @param namename of the filed
+   * @param relDataType the source of type information to construct the schema
+   * @return {@link ColumnMetadata} instance
+   */
+  private static ColumnMetadata getArrayMetadata(String name, RelDataType 
relDataType) {
+RelDataType componentType = relDataType.getComponentType();
+ColumnMetadata childColumnMetadata = getColumnMetadata(name, 
componentType);
+switch (componentType.getSqlTypeName()) {
+  case ARRAY:
+// for the case when nested type is array, it should be placed into 
repeated list
+return MetadataUtils.newRepeatedList(name, childColumnMetadata);
+  case MAP:
+  case OTHER:
+throw new UnsupportedOperationException(String.format("Unsupported 
data type: %s", relDataType.getSqlTypeName()));
+  default:
+if (componentType.isStruct()) {
+  // for the case when nested type is struct, it should be placed into 
repeated map
+  return MetadataUtils.newMapArray(name, 
childColumnMetadata.tupleSchema());
+} else {
+  // otherwise creates column metadata with repeated data mode
+  return new PrimitiveColumnMetadata(
+MaterializedField.create(name,
+  Types.overrideMode(
+
TypeInferenceUtils.getDrillMajorTypeFromCalciteType(componentType),
+TypeProtos.DataMode.REPEATED)));
+}
+}
+  }
+
+  /**
+   * Returns {@link MapColumnMetadata} column metadata created based on 
specified {@code RelDataType relDataType} with
+   * converted to {@link ColumnMetadata} {@code relDataType}'s children.
+   *
+   * @param namename of the filed
+   * @param relDataType {@link RelDataType} the source of the children for 
resulting schema
+   * @return {@link MapColumnMetadata} column metadata
+   */
+  private static MapColumnMetadata getStructMetadata(String name, RelDataType 
relDataType) {
+TupleMetadata mapSchema = new TupleSchema();
+relDataType.getFieldList().stream()
+  .map(field -> getColumnMetadata(field.getName(), field.getType()))

Review Comment:
   In the code it is called struct. Real map types as they are aren't supported 
for values operator.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] cgivre commented on a diff in pull request #2539: DRILL-8216: Use EVF-based JSON reader for Values operator

[GitBox]

cgivre commented on code in PR #2539:
URL: https://github.com/apache/drill/pull/2539#discussion_r870823130


##
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/scan/v3/schema/SchemaUtils.java:
##
@@ -234,4 +241,80 @@ static void copyProperties(TupleMetadata source,
   dest.setProperty(ScanProjectionParser.PROJECTION_TYPE_PROP, value);
 }
   }
+
+  /**
+   * Converts specified {@code RelDataType relDataType} into {@link 
ColumnMetadata}.
+   * For the case when specified relDataType is struct, map with recursively 
converted children
+   * will be created.
+   *
+   * @param namefiled name
+   * @param relDataType filed type
+   * @return {@link ColumnMetadata} which corresponds to specified {@code 
RelDataType relDataType}
+   */
+  public static ColumnMetadata getColumnMetadata(String name, RelDataType 
relDataType) {
+switch (relDataType.getSqlTypeName()) {
+  case ARRAY:
+return getArrayMetadata(name, relDataType);
+  case MAP:
+  case OTHER:
+throw new UnsupportedOperationException(String.format("Unsupported 
data type: %s", relDataType.getSqlTypeName()));
+  default:
+if (relDataType.isStruct()) {
+  return getStructMetadata(name, relDataType);
+} else {
+  return new PrimitiveColumnMetadata(
+MaterializedField.create(name,
+  
TypeInferenceUtils.getDrillMajorTypeFromCalciteType(relDataType)));
+}
+}
+  }
+
+  /**
+   * Returns {@link ColumnMetadata} instance which corresponds to specified 
array {@code RelDataType relDataType}.
+   *
+   * @param namename of the filed
+   * @param relDataType the source of type information to construct the schema
+   * @return {@link ColumnMetadata} instance
+   */
+  private static ColumnMetadata getArrayMetadata(String name, RelDataType 
relDataType) {
+RelDataType componentType = relDataType.getComponentType();
+ColumnMetadata childColumnMetadata = getColumnMetadata(name, 
componentType);
+switch (componentType.getSqlTypeName()) {
+  case ARRAY:
+// for the case when nested type is array, it should be placed into 
repeated list
+return MetadataUtils.newRepeatedList(name, childColumnMetadata);
+  case MAP:
+  case OTHER:
+throw new UnsupportedOperationException(String.format("Unsupported 
data type: %s", relDataType.getSqlTypeName()));
+  default:
+if (componentType.isStruct()) {
+  // for the case when nested type is struct, it should be placed into 
repeated map
+  return MetadataUtils.newMapArray(name, 
childColumnMetadata.tupleSchema());
+} else {
+  // otherwise creates column metadata with repeated data mode
+  return new PrimitiveColumnMetadata(
+MaterializedField.create(name,
+  Types.overrideMode(
+
TypeInferenceUtils.getDrillMajorTypeFromCalciteType(componentType),
+TypeProtos.DataMode.REPEATED)));
+}
+}
+  }
+
+  /**
+   * Returns {@link MapColumnMetadata} column metadata created based on 
specified {@code RelDataType relDataType} with
+   * converted to {@link ColumnMetadata} {@code relDataType}'s children.
+   *
+   * @param namename of the filed
+   * @param relDataType {@link RelDataType} the source of the children for 
resulting schema
+   * @return {@link MapColumnMetadata} column metadata
+   */
+  private static MapColumnMetadata getStructMetadata(String name, RelDataType 
relDataType) {
+TupleMetadata mapSchema = new TupleSchema();
+relDataType.getFieldList().stream()
+  .map(field -> getColumnMetadata(field.getName(), field.getType()))

Review Comment:
   What would happen here if you have a nested map?  IE something like this:
   ```
   {
  "field1": {
"nested_field1":"something",
"nested_field2": {
   "really_nested":"something",
   "real_nested2":"something_else"
  
 }
   }
   }
   ```
   Or are nested schemata not supported for `VALUES()` operator?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] jnturton opened a new pull request, #2547: DRILL-8223: Refactor auth modes dropping DRILL_PROCESS and allowing credential providers everywhere

[GitBox]

jnturton opened a new pull request, #2547:
URL: https://github.com/apache/drill/pull/2547

   # [DRILL-8223](https://issues.apache.org/jira/browse/DRILL-8223): Refactor 
auth modes dropping DRILL_PROCESS and allowing credential providers everywhere
   
   ## Description
   
   Remove the abstract CredentialedStoragePluginConfig (formerly 
AbstractSecuredStoragePluginConfig) class and promote the credential provider 
members to the parent StoragePluginConfig. Since having a credential provider 
provider is optional, there is no harm in giving the capability to every 
plugin's config.
   
   Drop the DRILL_PROCESS auth mode since this case is adequately covered by 
using SHARED_USER with no credentials specified.
   
   Bug fix. In storage-jdbc and when there are no JDBC credentials for the 
query user, only forgo an attempt to connect if the auth mode is 
USER_TRANSLATION. If it is SHARED_USER, proceed with an attempt to connect 
(examples of this case are unsecured DBs and BigQuery which requires OAuth 
tokens in the JDBC URL instead of a JDBC username and password).
   
   ## Documentation
   New auth mode documentation once the feature has stabilised.
   
   ## Testing
   TODO
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] jnturton merged pull request #2546: DRILL-8222: Fix wrong func impl of concat_delim, when null-value exist in middle args

[GitBox]

jnturton merged PR #2546:
URL: https://github.com/apache/drill/pull/2546


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] jnturton commented on pull request #2541: DRILL-8218: Add unit tests for StorageResource REST endpoints

[GitBox]

jnturton commented on PR #2541:
URL: https://github.com/apache/drill/pull/2541#issuecomment-1123691043

   > I saw that the slow test is failing again. If it helps, we can put an 
`@Ignore` annotation on that and run it manually.
   
   Done (in another PR).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] wojustme opened a new pull request, #2546: DRILL-8222: Fix wrong func impl of concat_delim, when null-value exist in middle args

[GitBox]

wojustme opened a new pull request, #2546:
URL: https://github.com/apache/drill/pull/2546

   # [DRILL-8222](https://issues.apache.org/jira/browse/DRILL-8222): Wrong 
function's implementation of concat_delim
   
   
   ## Description
   
   Current function implementation of `concat_delim` is wrong.
   `select concat_delim(',', 'a', null, 'b') as c `
   The result of this query should be 'a,b', but current result is 'a,,b'
   
   ## Documentation
   
   
   ## Testing
   
org.apache.drill.exec.fn.impl.TestVarArgFunctions#testConcatDelimVarArgsWithNullValue
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] jnturton opened a new pull request, #2545: DRILL-8221: Collect bug fixes from master for Drill 1.20.1

[GitBox]

jnturton opened a new pull request, #2545:
URL: https://github.com/apache/drill/pull/2545

   # [DRILL-8221](https://issues.apache.org/jira/browse/DRILL-8221): Collect 
bug fixes from master for Drill 1.20.1
   
   ## Description
   
   Since bugfix commits to master are not currently added to the latest stable 
branch at the time of their merger, they must be cherry picked to the 1.20.0 
branch _en masse_ to prepare for Drill 1.20.1.
   
   ## Documentation
   Discuss whether to create a release in Jira and do release notes.
   
   ## Testing
   Manually test logging and querying different data sources in embedded Drill. 
Unit tests.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] cgivre opened a new pull request, #2544: DRILL-8220: Add User Translation Support for OAuth Enabled Plugins

[GitBox]

cgivre opened a new pull request, #2544:
URL: https://github.com/apache/drill/pull/2544

   # [DRILL-8220](https://issues.apache.org/jira/browse/DRILL-8220): Add User 
Translation Support for OAuth Enabled Plugins
   
   ## Description
   This PR adds support for individual users to provide their own credentials 
for plugins that use OAuth 2.0 as a means of authorization and authentication.  
 Currently, only the HTTP storage plugin supports OAuth, however, this PR moves 
some of the core features out of the HTTP plugin so that other plugins can 
access this. 
   
   - [ ] Finish unit tests
   - [ ] Write documentation
   
   ## Documentation
   (Please describe user-visible changes similar to what should appear in the 
Drill documentation.)
   
   ## Testing
   (Please describe how this PR has been tested.)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] cgivre commented on pull request #2543: DRILL-5955: Revisit Union Vectors

[GitBox]

cgivre commented on PR #2543:
URL: https://github.com/apache/drill/pull/2543#issuecomment-1123010372

   @vdiravka This is looking good.  Quick question, can we add a configuration 
variable to the JSON Format to enable/disable this at the plugin instance level?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] vdiravka opened a new pull request, #2543: DRILL-5955: Revisit Union Vectors

[GitBox]

vdiravka opened a new pull request, #2543:
URL: https://github.com/apache/drill/pull/2543

   # [DRILL-5955](https://issues.apache.org/jira/browse/DRILL-5955): Revisit 
Union Vectors
   
   ## Description
   
   Draft PR for support Union type for JSON2
   
   - [x] Enabled Union/Variant vectors for JSON2
   - [ ] fix validation for Union vectors
   - [ ] fix List vector in the Union vector
   - [ ] check all existing tests work fine
   
   ## Documentation
   TBA
   
   ## Testing
   existing tests will be enabled for JSON2
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] cgivre merged pull request #2542: DRILL-8219: Handle null catalog names returned by DB2 in storage-jdbc

[GitBox]

cgivre merged PR #2542:
URL: https://github.com/apache/drill/pull/2542


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] cgivre commented on pull request #2542: DRILL-8219: Handle null catalog names returned by DB2 in storage-jdbc

[GitBox]

cgivre commented on PR #2542:
URL: https://github.com/apache/drill/pull/2542#issuecomment-1122609657

   > 
   
   Ok... Let's just merge then. 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] jnturton commented on pull request #2542: DRILL-8219: Handle null catalog names returned by DB2 in storage-jdbc

[GitBox]

jnturton commented on PR #2542:
URL: https://github.com/apache/drill/pull/2542#issuecomment-1122604719

   > LGTM +1
   > This is really a minor update, but is it worth adding a unit test?  We can 
skip it if it is a major hassle.
   
   I think a unit test would need a new DB2 testcontainer or a mocked JDBC 
Connection that returns a null like DB2 does. The cost/benefit of doing those 
to exercise a null test doesn't look good to me, but there may be some tricks I 
don't know. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill-test-framework] dependabot[bot] opened a new pull request, #2: Bump junit from 4.12 to 4.13.1 in /framework

[GitBox]

dependabot[bot] opened a new pull request, #2:
URL: https://github.com/apache/drill-test-framework/pull/2

   Bumps [junit](https://github.com/junit-team/junit4) from 4.12 to 4.13.1.
   
   Release notes
   Sourced from https://github.com/junit-team/junit4/releases";>junit's 
releases.
   
   JUnit 4.13.1
   Please refer to the https://github.com/junit-team/junit/blob/HEAD/doc/ReleaseNotes4.13.1.md";>release
 notes for details.
   JUnit 4.13
   Please refer to the https://github.com/junit-team/junit/blob/HEAD/doc/ReleaseNotes4.13.md";>release
 notes for details.
   JUnit 4.13 RC 2
   Please refer to the https://github.com/junit-team/junit4/wiki/4.13-Release-Notes";>release 
notes for details.
   JUnit 4.13 RC 1
   Please refer to the https://github.com/junit-team/junit4/wiki/4.13-Release-Notes";>release 
notes for details.
   JUnit 4.13 Beta 3
   Please refer to the https://github.com/junit-team/junit4/wiki/4.13-Release-Notes";>release 
notes for details.
   JUnit 4.13 Beta 2
   Please refer to the https://github.com/junit-team/junit4/wiki/4.13-Release-Notes";>release 
notes for details.
   JUnit 4.13 Beta 1
   Please refer to the https://github.com/junit-team/junit4/wiki/4.13-Release-Notes";>release 
notes for details.
   
   
   
   Commits
   
   https://github.com/junit-team/junit4/commit/1b683f4ec07bcfa40149f086d32240f805487e66";>1b683f4
 [maven-release-plugin] prepare release r4.13.1
   https://github.com/junit-team/junit4/commit/ce6ce3aadc070db2902698fe0d3dc6729cd631f2";>ce6ce3a
 Draft 4.13.1 release notes
   https://github.com/junit-team/junit4/commit/c29dd8239d6b353e699397eb090a1fd27411fa24";>c29dd82
 Change version to 4.13.1-SNAPSHOT
   https://github.com/junit-team/junit4/commit/1d174861f0b64f97ab0722bb324a760bfb02f567";>1d17486
 Add a link to assertThrows in exception testing
   https://github.com/junit-team/junit4/commit/543905df72ff10364b94dda27552efebf3dd04e9";>543905d
 Use separate line for annotation in Javadoc
   https://github.com/junit-team/junit4/commit/510e906b391e7e46a346e1c852416dc7be934944";>510e906
 Add sub headlines to class Javadoc
   https://github.com/junit-team/junit4/commit/610155b8c22138329f0723eec22521627dbc52ae";>610155b
 Merge pull request from GHSA-269g-pwp5-87pp
   https://github.com/junit-team/junit4/commit/b6cfd1e3d736cc2106242a8be799615b472c7fec";>b6cfd1e
 Explicitly wrap float parameter for consistency (https://github-redirect.dependabot.com/junit-team/junit4/issues/1671";>#1671)
   https://github.com/junit-team/junit4/commit/a5d205c7956dbed302b3bb5ecde5ba4299f0b646";>a5d205c
 Fix GitHub link in FAQ (https://github-redirect.dependabot.com/junit-team/junit4/issues/1672";>#1672)
   https://github.com/junit-team/junit4/commit/3a5c6b4d08f408c8ca6a8e0bae71a9bc5a8f97e8";>3a5c6b4
 Deprecated since jdk9 replacing constructor instance of Double and Float (https://github-redirect.dependabot.com/junit-team/junit4/issues/1660";>#1660)
   Additional commits viewable in https://github.com/junit-team/junit4/compare/r4.12...r4.13.1";>compare 
view
   
   
   
   
   
   [![Dependabot compatibility 
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=junit:junit&package-manager=maven&previous-version=4.12&new-version=4.13.1)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
   
   Dependabot will resolve any conflicts with this PR as long as you don't 
alter it yourself. You can also trigger a rebase manually by commenting 
`@dependabot rebase`.
   
   [//]: # (dependabot-automerge-start)
   [//]: # (dependabot-automerge-end)
   
   ---
   
   
   Dependabot commands and options
   
   
   You can trigger Dependabot actions by commenting on this PR:
   - `@dependabot rebase` will rebase this PR
   - `@dependabot recreate` will recreate this PR, overwriting any edits that 
have been made to it
   - `@dependabot merge` will merge this PR after your CI passes on it
   - `@dependabot squash and merge` will squash and merge this PR after your CI 
passes on it
   - `@dependabot cancel merge` will cancel a previously requested merge and 
block automerging
   - `@dependabot reopen` will reopen this PR if it is closed
   - `@dependabot close` will close this PR and stop Dependabot recreating it. 
You can achieve the same result by closing it manually
   - `@dependabot ignore this major version` will close this PR and stop 
Dependabot creating any more for this major version (unless you reopen the PR 
or upgrade to it yourself)
   - `@dependabot ignore this minor version` will close this PR and stop 
Dependabot creating any more for this minor version (unless you reopen the PR 
or upgrade to it yourself)
   - `@dependabot ignore this dependency` will close this PR and stop 
Dependabot creating any more for this dependency (unless you reopen the PR or 
upgrade to it yourself)
   - `@dependabot use these labels` will set the current labels as the default 
for future PRs for 

[GitHub] [drill-test-framework] dependabot[bot] opened a new pull request, #4: Bump httpclient from 4.5.5 to 4.5.13 in /framework

[GitBox]

dependabot[bot] opened a new pull request, #4:
URL: https://github.com/apache/drill-test-framework/pull/4

   Bumps httpclient from 4.5.5 to 4.5.13.
   
   
   [![Dependabot compatibility 
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=org.apache.httpcomponents:httpclient&package-manager=maven&previous-version=4.5.5&new-version=4.5.13)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
   
   Dependabot will resolve any conflicts with this PR as long as you don't 
alter it yourself. You can also trigger a rebase manually by commenting 
`@dependabot rebase`.
   
   [//]: # (dependabot-automerge-start)
   [//]: # (dependabot-automerge-end)
   
   ---
   
   
   Dependabot commands and options
   
   
   You can trigger Dependabot actions by commenting on this PR:
   - `@dependabot rebase` will rebase this PR
   - `@dependabot recreate` will recreate this PR, overwriting any edits that 
have been made to it
   - `@dependabot merge` will merge this PR after your CI passes on it
   - `@dependabot squash and merge` will squash and merge this PR after your CI 
passes on it
   - `@dependabot cancel merge` will cancel a previously requested merge and 
block automerging
   - `@dependabot reopen` will reopen this PR if it is closed
   - `@dependabot close` will close this PR and stop Dependabot recreating it. 
You can achieve the same result by closing it manually
   - `@dependabot ignore this major version` will close this PR and stop 
Dependabot creating any more for this major version (unless you reopen the PR 
or upgrade to it yourself)
   - `@dependabot ignore this minor version` will close this PR and stop 
Dependabot creating any more for this minor version (unless you reopen the PR 
or upgrade to it yourself)
   - `@dependabot ignore this dependency` will close this PR and stop 
Dependabot creating any more for this dependency (unless you reopen the PR or 
upgrade to it yourself)
   - `@dependabot use these labels` will set the current labels as the default 
for future PRs for this repo and language
   - `@dependabot use these reviewers` will set the current reviewers as the 
default for future PRs for this repo and language
   - `@dependabot use these assignees` will set the current assignees as the 
default for future PRs for this repo and language
   - `@dependabot use this milestone` will set the current milestone as the 
default for future PRs for this repo and language
   
   You can disable automated security fix PRs for this repo from the [Security 
Alerts page](https://github.com/apache/drill-test-framework/network/alerts).
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill-test-framework] dependabot[bot] opened a new pull request, #3: Bump jackson-databind from 2.10.0 to 2.12.6.1 in /framework

[GitBox]

dependabot[bot] opened a new pull request, #3:
URL: https://github.com/apache/drill-test-framework/pull/3

   Bumps [jackson-databind](https://github.com/FasterXML/jackson) from 2.10.0 
to 2.12.6.1.
   
   Commits
   
   See full diff in https://github.com/FasterXML/jackson/commits";>compare view
   
   
   
   
   
   [![Dependabot compatibility 
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=com.fasterxml.jackson.core:jackson-databind&package-manager=maven&previous-version=2.10.0&new-version=2.12.6.1)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
   
   Dependabot will resolve any conflicts with this PR as long as you don't 
alter it yourself. You can also trigger a rebase manually by commenting 
`@dependabot rebase`.
   
   [//]: # (dependabot-automerge-start)
   [//]: # (dependabot-automerge-end)
   
   ---
   
   
   Dependabot commands and options
   
   
   You can trigger Dependabot actions by commenting on this PR:
   - `@dependabot rebase` will rebase this PR
   - `@dependabot recreate` will recreate this PR, overwriting any edits that 
have been made to it
   - `@dependabot merge` will merge this PR after your CI passes on it
   - `@dependabot squash and merge` will squash and merge this PR after your CI 
passes on it
   - `@dependabot cancel merge` will cancel a previously requested merge and 
block automerging
   - `@dependabot reopen` will reopen this PR if it is closed
   - `@dependabot close` will close this PR and stop Dependabot recreating it. 
You can achieve the same result by closing it manually
   - `@dependabot ignore this major version` will close this PR and stop 
Dependabot creating any more for this major version (unless you reopen the PR 
or upgrade to it yourself)
   - `@dependabot ignore this minor version` will close this PR and stop 
Dependabot creating any more for this minor version (unless you reopen the PR 
or upgrade to it yourself)
   - `@dependabot ignore this dependency` will close this PR and stop 
Dependabot creating any more for this dependency (unless you reopen the PR or 
upgrade to it yourself)
   - `@dependabot use these labels` will set the current labels as the default 
for future PRs for this repo and language
   - `@dependabot use these reviewers` will set the current reviewers as the 
default for future PRs for this repo and language
   - `@dependabot use these assignees` will set the current assignees as the 
default for future PRs for this repo and language
   - `@dependabot use this milestone` will set the current milestone as the 
default for future PRs for this repo and language
   
   You can disable automated security fix PRs for this repo from the [Security 
Alerts page](https://github.com/apache/drill-test-framework/network/alerts).
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill-test-framework] jnturton merged pull request #1: Resurrection

[GitBox]

jnturton merged PR #1:
URL: https://github.com/apache/drill-test-framework/pull/1


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill-test-framework] agozhiy commented on a diff in pull request #1: Resurrection

[GitBox]

agozhiy commented on code in PR #1:
URL: https://github.com/apache/drill-test-framework/pull/1#discussion_r869156157


##
framework/pom.xml:
##
@@ -7,12 +7,27 @@
   1.0.0-SNAPSHOT
   
 6.4
-1.2.17
+1.7.26
+1.2.9
 2.10.0
-${env.DRILL_VERSION}
-${env.HADOOP_VERSION}
-${env.DRILL_HOME}/conf
+30.1.1-jre
+2.0.0-SNAPSHOT

Review Comment:
   Correct. The idea of the test framework is to test the latest version, so 
the pom.xml have to be updated accordingly, if the version is changed. We 
assume that previous versions were already tested so no need for more 
flexibility here. And in case of different Drill branches (i.e. hadoop 2) we 
can create corresponding branches of Drill Test Framework.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] jnturton opened a new pull request, #2542: DRILL-8219: Handle null catalog names returned by DB2 in storage-jdbc

[GitBox]

jnturton opened a new pull request, #2542:
URL: https://github.com/apache/drill/pull/2542

   # [DRILL-8219](https://issues.apache.org/jira/browse/DRILL-8219): Handle 
null catalog names returned by DB2 in storage-jdbc
   
   ## Description
   
   DB2 is another DBMS in which clients must connect to a named database, not 
just to the DBMS. A JDBC connection that does the same will receive a single 
record containing a null string from a call to 
Connection::getMetaData::getCatalogs. We need to ignore such a record in order 
to query DB2 correctly.
   
   ## Documentation
   N/A
   
   ## Testing
   Install DB2 using the Docker container ibmcom/db2. Create a test schema and 
table in the existing `testdb`. Configure a new Drill JDBC storage config to 
connect to the DB2 `testdb` database. Test info schema queries and direct 
querying of the test table.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] jnturton opened a new pull request, #2541: DRILL-8218: Add unit tests for StorageResource REST endpoints

[GitBox]

jnturton opened a new pull request, #2541:
URL: https://github.com/apache/drill/pull/2541

   # [DRILL-8218](https://issues.apache.org/jira/browse/DRILL-8218): Add unit 
tests for StorageResource REST endpoints
   
   ## Description
   
   Use the RestClientFixture to test the function of API endpoints in 
StorageResources
   
   ## Documentation
   N/A
   
   ## Testing
   N/A
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill-test-framework] jnturton commented on a diff in pull request #1: Resurrection

[GitBox]

jnturton commented on code in PR #1:
URL: https://github.com/apache/drill-test-framework/pull/1#discussion_r868861190


##
framework/pom.xml:
##
@@ -7,12 +7,27 @@
   1.0.0-SNAPSHOT
   
 6.4
-1.2.17
+1.7.26
+1.2.9
 2.10.0
-${env.DRILL_VERSION}
-${env.HADOOP_VERSION}
-${env.DRILL_HOME}/conf
+30.1.1-jre
+2.0.0-SNAPSHOT

Review Comment:
   So if I want to run the test framework against a Drill 1.20.1 instead of a 
Drill 2.0.0-SNAPSHOT then I should edit the pom file?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] vvysotskyi merged pull request #2534: DRILL-8210: Add substring convertlet

[GitBox]

vvysotskyi merged PR #2534:
URL: https://github.com/apache/drill/pull/2534


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] vvysotskyi merged pull request #2537: DRILL-8213: Replace deprecated RelNode.getRows with RelNode.estimateRowCount

[GitBox]

vvysotskyi merged PR #2537:
URL: https://github.com/apache/drill/pull/2537


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill-test-framework] agozhiy commented on pull request #1: Resurrection

[GitBox]

agozhiy commented on PR #1:
URL: 
https://github.com/apache/drill-test-framework/pull/1#issuecomment-1121369437

   Regarding the UDF tests: the test folder doesn't contain any tests so there 
is no difference what category to use. 
   I searched in git history and it was always empty, not even in a separate 
branch. There is only this PR: 
https://github.com/mapr/drill-test-framework/pull/249, but the tests there are 
very cluster-specific and require drill restarts during the execution. For that 
reason the PR wasn't merged.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill-test-framework] agozhiy commented on a diff in pull request #1: Resurrection

[GitBox]

agozhiy commented on code in PR #1:
URL: https://github.com/apache/drill-test-framework/pull/1#discussion_r868211411


##
framework/pom.xml:
##
@@ -7,12 +7,27 @@
   1.0.0-SNAPSHOT
   
 6.4
-1.2.17
+1.7.26
+1.2.9
 2.10.0
-${env.DRILL_VERSION}
-${env.HADOOP_VERSION}
-${env.DRILL_HOME}/conf
+30.1.1-jre
+2.0.0-SNAPSHOT

Review Comment:
   It is bound to the Drill version (or the hadoop version used in Drill). 
There is no reason for it to be an environment variable. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] cgivre merged pull request #2536: DRILL-8212: Join queries fail with StackOverflowError

[GitBox]

cgivre merged PR #2536:
URL: https://github.com/apache/drill/pull/2536


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill-test-framework] jnturton commented on a diff in pull request #1: Resurrection

[GitBox]

jnturton commented on code in PR #1:
URL: https://github.com/apache/drill-test-framework/pull/1#discussion_r867959310


##
framework/pom.xml:
##
@@ -7,12 +7,27 @@
   1.0.0-SNAPSHOT
   
 6.4
-1.2.17
+1.7.26
+1.2.9
 2.10.0
-${env.DRILL_VERSION}
-${env.HADOOP_VERSION}
-${env.DRILL_HOME}/conf
+30.1.1-jre
+2.0.0-SNAPSHOT

Review Comment:
   Is it better that this stops being an env var?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill-test-framework] jnturton commented on pull request #1: Resurrection

[GitBox]

jnturton commented on PR #1:
URL: 
https://github.com/apache/drill-test-framework/pull/1#issuecomment-1121029474

   Thanks for this!
   
   - Are the new modes that can be seen in the JSON config files, e.g. `rm`, 
`cp`, `dfs_cp`, `post_rm`, the new "data preparation phases" that you mention 
above?
   - What does it mean that the UDF tests have moved from the `functional` to 
the `excluded` category? That they have been disabled by default?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] jnturton merged pull request #2540: DRILL-8215: Remove SecurityContext from PluginConfigWrapper

[GitBox]

jnturton merged PR #2540:
URL: https://github.com/apache/drill/pull/2540


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] jnturton commented on a diff in pull request #2540: DRILL-8215: Remove SecurityContext from PluginConfigWrapper

[GitBox]

jnturton commented on code in PR #2540:
URL: https://github.com/apache/drill/pull/2540#discussion_r867783408


##
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/CredentialResources.java:
##
@@ -125,11 +125,18 @@ public List 
getConfigsFor(@PathParam("group") String plugin
   default:
 return Collections.emptyList();
 }
-return StreamSupport.stream(
+List results = StreamSupport.stream(
 
Spliterators.spliteratorUnknownSize(storage.storedConfigs(filter).entrySet().iterator(),
 Spliterator.ORDERED), false)
-  .map(entry -> new PluginConfigWrapper(entry.getKey(), entry.getValue(), 
sc))
+  .map(entry -> new PluginConfigWrapper(entry.getKey(), entry.getValue()))
   .sorted(PLUGIN_COMPARATOR)
   .collect(Collectors.toList());
+
+if (results.isEmpty()) {
+  return Collections.emptyList();

Review Comment:
   Does this logic do anything? If `results` is empty then it already is an 
empty List, is it not?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] jnturton merged pull request #2532: DRILL-8208: Create builder for SqlSelect

[GitBox]

jnturton merged PR #2532:
URL: https://github.com/apache/drill/pull/2532


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] cgivre opened a new pull request, #2540: DRILL-8215: Remove SecurityContext from PluginConfigWrapper

[GitBox]

cgivre opened a new pull request, #2540:
URL: https://github.com/apache/drill/pull/2540

   # [DRILL-8215](https://issues.apache.org/jira/browse/DRILL-8215): Remove 
SecurityContext from PluginConfigWrapper
   
   ## Description
   DRILL-8155 introduced a small regression in the `PluginConfigWrapper` which 
caused some serialization/deserialization errors.  This fix removes the 
`SecurityContext` from that class and as a result, fixes the issue.
   
   This PR also addresses 
[DRILL-8217](https://issues.apache.org/jira/browse/DRILL-8217) which the page 
to enter user credentials throws an exception if no plugins are eligible. 
   
   ## Documentation
   N/A
   
   ## Testing
   Manually tested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] lgtm-com[bot] commented on pull request #2539: DRILL-8216: Use EVF-based JSON reader for Values operator

[GitBox]

lgtm-com[bot] commented on PR #2539:
URL: https://github.com/apache/drill/pull/2539#issuecomment-1120440515

   This pull request **fixes 1 alert** when merging 
d2908869c5c841ac9397995d2e4a65d8ca2ca9cf into 
8b498206b35708fa0ae8ef3be3cb41e9d3c13838 - [view on 
LGTM.com](https://lgtm.com/projects/g/apache/drill/rev/pr-2b7cc30cf7339872c0e96c054f643707d9f0)
   
   **fixed alerts:**
   
   * 1 for Result of multiplication cast to wider type


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] vvysotskyi opened a new pull request, #2539: DRILL-8216: Use EVF-based JSON reader for Values operator

[GitBox]

vvysotskyi opened a new pull request, #2539:
URL: https://github.com/apache/drill/pull/2539

   # [DRILL-8216](https://issues.apache.org/jira/browse/DRILL-8216): Use 
EVF-based JSON reader for Values operator
   
   ## Description
   The newer Calcite version simplifies and removes casts for literals. It 
causes wrong results for `drillTypeOf` and `sqlTypeOf` functions since the 
Values operator uses an old JSON reader which reads integers with bigInt type.
   This PR uses EVF-based JSON reader for the Values operator and passes actual 
data types discovered during planning as the provided schema.
   
   ## Documentation
   NA
   
   ## Testing
   Unit tests pass


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] lgtm-com[bot] commented on pull request #2538: DRILL-8214: Replace EnumerableTableScan usage with LogicalTableScan

[GitBox]

lgtm-com[bot] commented on PR #2538:
URL: https://github.com/apache/drill/pull/2538#issuecomment-1120295482

   This pull request **introduces 1 alert** when merging 
276fd9823bc6ded1531fc9b3f0d21412e102acf0 into 
8b498206b35708fa0ae8ef3be3cb41e9d3c13838 - [view on 
LGTM.com](https://lgtm.com/projects/g/apache/drill/rev/pr-31899a25f61d999a7a0ef13995a18ea98375)
   
   **new alerts:**
   
   * 1 for Useless type test


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] vvysotskyi opened a new pull request, #2538: DRILL-8214: Replace EnumerableTableScan usage with LogicalTableScan

[GitBox]

vvysotskyi opened a new pull request, #2538:
URL: https://github.com/apache/drill/pull/2538

   # [DRILL-8214](https://issues.apache.org/jira/browse/DRILL-8214): Replace 
EnumerableTableScan usage with LogicalTableScan
   
   ## Description
   The newer Calcite version returns LogicalTableScan instead of 
EnumerableTableScan in RelOptTableImpl, so Drill shouldn't rely on this class 
also and LogicalTableScan where possible to avoid planning issues.
   
   ## Documentation
   NA
   
   ## Testing
   Unit tests passing
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] vvysotskyi opened a new pull request, #2537: DRILL-8213: Replace deprecated RelNode.getRows with RelNode.estimateRowCount

[GitBox]

vvysotskyi opened a new pull request, #2537:
URL: https://github.com/apache/drill/pull/2537

   # [DRILL-8213](https://issues.apache.org/jira/browse/DRILL-8213): Replace 
deprecated RelNode.getRows with RelNode.estimateRowCount
   
   ## Description
   In the newer Calcite version RelNode.getRows was removed, so replacing its 
usage with RelNode.estimateRowCount
   
   ## Documentation
   NA
   
   ## Testing
   Unit tests pass
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] vvysotskyi opened a new pull request, #2536: DRILL-8212: Join queries fail with StackOverflowError

[GitBox]

vvysotskyi opened a new pull request, #2536:
URL: https://github.com/apache/drill/pull/2536

   # [DRILL-8212](https://issues.apache.org/jira/browse/DRILL-8212): Join 
queries fail with StackOverflowError
   
   ## Description
   With the newer Calcite version, some join queries fail with 
StackOverflowError. In the new version, Calcite uses selectivity code for 
computing cost for some conditions.
   
   ## Documentation
   NA
   
   ## Testing
   Unit tests that were failing pass
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] vvysotskyi opened a new pull request, #2535: DRILL-8211: Replace deprecated RelNode.getChildExps with Project.getProjects

[GitBox]

vvysotskyi opened a new pull request, #2535:
URL: https://github.com/apache/drill/pull/2535

   # [DRILL-8211](https://issues.apache.org/jira/browse/DRILL-8211): Replace 
deprecated RelNode.getChildExps with Project.getProjects
   
   ## Description
   In the newer Calcite version `RelNode.getChildExps` was removed, so 
replacing it with `Project.getProjects`
   
   ## Documentation
   NA
   
   ## Testing
   Unit tests pass.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] vvysotskyi opened a new pull request, #2534: DRILL-8210: Add substring convertlet

[GitBox]

vvysotskyi opened a new pull request, #2534:
URL: https://github.com/apache/drill/pull/2534

   # [DRILL-8210](https://issues.apache.org/jira/browse/DRILL-8210): Add 
substring convertlet
   
   ## Description
   See Jira ticket description for details
   
   ## Documentation
   NA
   
   ## Testing
   Unit test pass on newer Calcite
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] vvysotskyi opened a new pull request, #2533: DRILL-8209: Introduce rule for converting join with distinct input to semi-join

[GitBox]

vvysotskyi opened a new pull request, #2533:
URL: https://github.com/apache/drill/pull/2533

   # [DRILL-8209](https://issues.apache.org/jira/browse/DRILL-8209): Introduce 
rule for converting join with distinct input to semi-join
   
   ## Description
   Newer Calcite changed the order of applying rules. AggregateRemoveRule is 
applied before SemiJoinRule, so SemiJoinRule cannot be applied later, since 
aggregate is pruned from planning.
   
   Adding rule that converts join to semi-join if the right side has distinct 
columns.
   Fixed cost and row count computation for semi-join.
   
   ## Documentation
   NA
   
   ## Testing
   Unit tests pass
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] vvysotskyi opened a new pull request, #2532: DRILL-8208: Create builder for SqlSelect

[GitBox]

vvysotskyi opened a new pull request, #2532:
URL: https://github.com/apache/drill/pull/2532

   # [DRILL-8208](https://issues.apache.org/jira/browse/DRILL-8208): Create 
builder for SqlSelect
   
   ## Description
   The newer Calcite version adds more fields to the constructor of SqlSelect. 
In most cases, nulls are passed for these new arguments. Using builder will 
reduce the number of places where it should be added.
   
   ## Documentation
   NA
   
   ## Testing
   Unit tests pass
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] jnturton merged pull request #2531: DRILL-8207: Fix Username Typo and password @JsonIgnore in JDBC SerDe

[GitBox]

jnturton merged PR #2531:
URL: https://github.com/apache/drill/pull/2531


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] jnturton opened a new pull request, #2531: DRILL-8207: Fix Username Typo and password @JsonIgnore in JDBC SerDe

[GitBox]

jnturton opened a new pull request, #2531:
URL: https://github.com/apache/drill/pull/2531

   # [DRILL-8207](https://issues.apache.org/jira/browse/DRILL-8207): Fix 
Username Typo and password @JsonIgnore in JDBC SerDe
   
   ## Description
   
   This second PR removes the @JsonIgnore annotation from 
`JsonIgnore::getPassword()`, solving a Jackson deserialsiation error affecting 
that class, and reverts a regression in 
`CredentialProviderUtils::getCredentialsProvider`.
   
   ## Documentation
   N/A
   
   ## Testing
   Unit tests, manually start Drill with template storage configs. Edit `rdbms` 
so that it holds a valid pg database config, run a query against pg, 
individually add and remove the username and password direct credentials from 
the storage config.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] luocooong commented on a diff in pull request #2500: DRILL-8172: Use the specified memory usage for Travis CI

[GitBox]

luocooong commented on code in PR #2500:
URL: https://github.com/apache/drill/pull/2500#discussion_r866652340


##
.travis.yml:
##
@@ -45,6 +45,9 @@ cache:
 before_install:
   - export JAVA_HOME="/usr/lib/jvm/java-8-openjdk-arm64"
   - export PATH="$JAVA_HOME/bin:$PATH"
+  - export MEMORYMB=2048

Review Comment:
   The GitHub CI uses 7GB and Travis CI uses close to 8GB.
   In the GitHub CI, we using the java managed heap is 2500 MB, and java direct 
memory is 4500MB, 168MB is reserved for OS. In the Travis CI, we can up the 
managed heap to 2500MB, but I also recommend keeping the direct memory at 5GB.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] vdiravka commented on pull request #2530: DRILL-8207: Fix typo in JDBC config

[GitBox]

vdiravka commented on PR #2530:
URL: https://github.com/apache/drill/pull/2530#issuecomment-1119421766

   Looks like we can be case insesitive here by adding:
   ```
   ObjectMapper mapper = new ObjectMapper();
   mapper.configure(MapperFeature.ACCEPT_CASE_INSENSITIVE_PROPERTIES, true);
   ```
   in
   ```
 public LogicalPlanPersistence(DrillConfig conf, ScanResult scanResult, 
ObjectMapper mapper) {
   // The UI allows comments in JSON. Since we use the mapper
   // here to serialize plugin configs to/from the pe
   this.mapper = mapper
   .configure(JsonParser.Feature.ALLOW_COMMENTS, true);
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] cgivre merged pull request #2530: DRILL-8207: Fix typo in JDBC config

[GitBox]

cgivre merged PR #2530:
URL: https://github.com/apache/drill/pull/2530


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] cgivre opened a new pull request, #2530: DRILL-8207: Fix typo in JDBC config

[GitBox]

cgivre opened a new pull request, #2530:
URL: https://github.com/apache/drill/pull/2530

   # [DRILL-8207](https://issues.apache.org/jira/browse/DRILL-8207): Fix 
Username Typo in JDBC SerDe
   
   ## Description
   Fixes a Serialization/Deserialization bug in the JDBC plugin
   
   ## Documentation
   No user facing changes.
   
   ## Testing
   Ran unit tests


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill-test-framework] agozhiy opened a new pull request, #1: Resurrection

[GitBox]

agozhiy opened a new pull request, #1:
URL: https://github.com/apache/drill-test-framework/pull/1

   - Added an option to run in a docker container
   - Added new data preparation phases
   - New concept of data preparation: changed from using bash scripts to main 
java application
   - Changed logger implementation from log4j to slf4j w/ logback
   - Some other refactoring


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] vdiravka commented on a diff in pull request #2500: DRILL-8172: Use the specified memory usage for Travis CI

[GitBox]

vdiravka commented on code in PR #2500:
URL: https://github.com/apache/drill/pull/2500#discussion_r865819857


##
.travis.yml:
##
@@ -45,6 +45,9 @@ cache:
 before_install:
   - export JAVA_HOME="/usr/lib/jvm/java-8-openjdk-arm64"
   - export PATH="$JAVA_HOME/bin:$PATH"
+  - export MEMORYMB=2048

Review Comment:
   I think 2048 is small value. The default one is 2500M.
   Now Travis build hangs:
   ```
   [INFO] Running org.apache.drill.exec.fn.impl.TestAggregateFunctions
   No output has been received in the last 10m0s, this potentially indicates a 
stalled build or something wrong with the build itself.
   ```
   Possibly it is due to lack of heap memory.
   @luocooong What do you think about reverting this value to the default one?



##
.travis.yml:
##
@@ -45,6 +45,9 @@ cache:
 before_install:
   - export JAVA_HOME="/usr/lib/jvm/java-8-openjdk-arm64"
   - export PATH="$JAVA_HOME/bin:$PATH"
+  - export MEMORYMB=2048
+  - export DIRECTMEMORYMB=5120

Review Comment:
   One note about `DIRECTMEMORYMB`. Do you think `4500M` is really small?
   GitHub Actions has the same 
[7Gb](https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources)
 memory limit and these builds pass amost always.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] jnturton merged pull request #2529: [MINOR UPDATE] Update AWS Java SDK to 1.12.211

[GitBox]

jnturton merged PR #2529:
URL: https://github.com/apache/drill/pull/2529


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] cgivre opened a new pull request, #2529: [MINOR UPDATE] Update AWS Java SDK to 1.12.211

[GitBox]

cgivre opened a new pull request, #2529:
URL: https://github.com/apache/drill/pull/2529

   # [MINOR UPDATE] Update AWS Java SDK to 1.12.211
   
   ## Description
   Bump AWS Java SDK to latest version.
   
   ## Documentation
   N/A
   
   ## Testing
   Manually tested with S3 bucket.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] cgivre merged pull request #2526: DRILL-8204: Allow Provided Schema for HTTP Plugin in JSON Mode

[GitBox]

cgivre merged PR #2526:
URL: https://github.com/apache/drill/pull/2526


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] cgivre commented on a diff in pull request #2526: DRILL-8204: Allow Provided Schema for HTTP Plugin in JSON Mode

[GitBox]

cgivre commented on code in PR #2526:
URL: https://github.com/apache/drill/pull/2526#discussion_r864097045


##
contrib/storage-http/JSON_Options.md:
##
@@ -0,0 +1,125 @@
+# JSON Options and Configuration 
+
+Drill has a collection of JSON configuration options to allow you to configure 
how Drill interprets JSON files.  These are set at the global level, however 
the HTTP plugin
+allows you to configure these options individually per connection and override 
the Drill defaults.  The options are:
+
+* `allowNanInf`:  Configures the connection to interpret `NaN` and `Inf` values
+* `allTextMode`:  By default, Drill attempts to infer data types from JSON 
data. If the data is malformed, Drill may throw schema change exceptions. If 
your data is
+  inconsistent, you can enable `allTextMode` which when true, Drill will read 
all JSON values as strings, rather than try to infer the data type.
+* `readNumbersAsDouble`:  By default Drill will attempt to interpret integers, 
floating point number types and strings.  One challenge is when data is 
consistent, Drill may
+  throw schema change exceptions. In addition to `allTextMode`, you can make 
Drill less sensitive by setting the `readNumbersAsDouble` to `true` which 
causes Drill to read all
+  numeric fields in JSON data as `double` data type rather than trying to 
distinguish between ints and doubles.
+* `enableEscapeAnyChar`:  Allows a user to escape any character with a \
+* `skipMalformedRecords`:  Allows Drill to skip malformed records and recover 
without throwing exceptions.
+* `skipMalformedDocument`:  Allows Drill to skip entire malformed documents 
without throwing errors.
+
+All of these can be set by adding the `jsonOptions` to your connection 
configuration as shown below:
+
+```json
+
+"jsonOptions": {
+  "allTextMode": true, 
+  "readNumbersAsDouble": true
+}
+
+```
+
+## Schema Provisioning
+One of the challenges of querying APIs is inconsistent data.  Drill allows you 
to provide a schema for individual endpoints.  You can do this in one of three 
ways: 
+
+1. By providing a schema inline [See: Specifying Schema as Table Function 
Parameter](https://drill.apache.org/docs/plugin-configuration-basics/#specifying-the-schema-as-table-function-parameter)
+2. By providing a schema in the configuration for the endpoint.
+3. By providing a serialized TupleMetadata of the desired schema.  This is an 
advanced functionality and should only be used by advanced Drill users.
+
+The schema provisioning currently supports complex types of Arrays and Maps at 
any nesting level.
+
+### Example Schema Provisioning:
+```json
+"jsonOptions": {
+  "providedSchema": [

Review Comment:
   Fixed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill-site] jnturton merged pull request #30: Update 011-running-drill-on-docker.md

[GitBox]

jnturton merged PR #30:
URL: https://github.com/apache/drill-site/pull/30


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] cgivre merged pull request #2528: [MINOR UPDATE] Fix default auth mode in AuthMode.parseOrDefault

[GitBox]

cgivre merged PR #2528:
URL: https://github.com/apache/drill/pull/2528


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] cgivre commented on a diff in pull request #2528: [MINOR UPDATE] Fix default auth mode in AuthMode.parseOrDefault

[GitBox]

cgivre commented on code in PR #2528:
URL: https://github.com/apache/drill/pull/2528#discussion_r863794931


##
logical/src/main/java/org/apache/drill/common/logical/StoragePluginConfig.java:
##
@@ -105,8 +106,8 @@ public enum AuthMode {
  */
 USER_TRANSLATION;
 
-public static AuthMode parseOrDefault(String authMode) {
-  return !Strings.isNullOrEmpty(authMode) ? 
AuthMode.valueOf(authMode.toUpperCase()) : DRILL_PROCESS;
+public static AuthMode parseOrDefault(String authMode, AuthMode defavlt) {

Review Comment:
   lol... Ok... well in that case, +1 (pending CI).  Thank you for turning this 
around so quickly.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] jnturton commented on a diff in pull request #2528: [MINOR UPDATE] Fix default auth mode in AuthMode.parseOrDefault

[GitBox]

jnturton commented on code in PR #2528:
URL: https://github.com/apache/drill/pull/2528#discussion_r863767452


##
logical/src/main/java/org/apache/drill/common/logical/StoragePluginConfig.java:
##
@@ -105,8 +106,8 @@ public enum AuthMode {
  */
 USER_TRANSLATION;
 
-public static AuthMode parseOrDefault(String authMode) {
-  return !Strings.isNullOrEmpty(authMode) ? 
AuthMode.valueOf(authMode.toUpperCase()) : DRILL_PROCESS;
+public static AuthMode parseOrDefault(String authMode, AuthMode defavlt) {

Review Comment:
   @cgivre I did wonder if that variable name would get a reaction. `default` 
is a reserved word in Java so this is a little Roman alphabet hack like writing 
`Class clazz;`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] cgivre commented on a diff in pull request #2528: [MINOR UPDATE] Fix default auth mode in AuthMode.parseOrDefault

[GitBox]

cgivre commented on code in PR #2528:
URL: https://github.com/apache/drill/pull/2528#discussion_r863751804


##
logical/src/main/java/org/apache/drill/common/logical/StoragePluginConfig.java:
##
@@ -105,8 +106,8 @@ public enum AuthMode {
  */
 USER_TRANSLATION;
 
-public static AuthMode parseOrDefault(String authMode) {
-  return !Strings.isNullOrEmpty(authMode) ? 
AuthMode.valueOf(authMode.toUpperCase()) : DRILL_PROCESS;
+public static AuthMode parseOrDefault(String authMode, AuthMode defavlt) {

Review Comment:
   I think you meant `default` rather than `defavlt` 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] jnturton opened a new pull request, #2528: [MINOR UPDATE] Fix default auth mode in AuthMode.parseOrDefault

[GitBox]

jnturton opened a new pull request, #2528:
URL: https://github.com/apache/drill/pull/2528

   ... and test the case of JDBC.
   
   ## Description
   
   A utility method for parsing the auth mode enum incorrectly defaults to 
`DRILL_PROCESS`, even when called by storage plugins that desire a different 
default.
   
   ## Documentation
   New user and dev docs for auth modes pending.
   
   ## Testing
   TestDataSource#testInitWithoutUserAndPassword now checks the default 
authMode applied when it is missing from the storage config for the case of the 
JDBC plugin.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill-site] cgivre merged pull request #29: Update 125-http-storage-plugin.md

[GitBox]

cgivre merged PR #29:
URL: https://github.com/apache/drill-site/pull/29


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] luocooong merged pull request #2527: DRILL-8035: Update Janino to 3.1.7 version

[GitBox]

luocooong merged PR #2527:
URL: https://github.com/apache/drill/pull/2527


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] vvysotskyi commented on a diff in pull request #2526: DRILL-8204: Allow Provided Schema for HTTP Plugin in JSON Mode

[GitBox]

vvysotskyi commented on code in PR #2526:
URL: https://github.com/apache/drill/pull/2526#discussion_r863495712


##
contrib/storage-http/JSON_Options.md:
##
@@ -0,0 +1,125 @@
+# JSON Options and Configuration 
+
+Drill has a collection of JSON configuration options to allow you to configure 
how Drill interprets JSON files.  These are set at the global level, however 
the HTTP plugin
+allows you to configure these options individually per connection and override 
the Drill defaults.  The options are:
+
+* `allowNanInf`:  Configures the connection to interpret `NaN` and `Inf` values
+* `allTextMode`:  By default, Drill attempts to infer data types from JSON 
data. If the data is malformed, Drill may throw schema change exceptions. If 
your data is
+  inconsistent, you can enable `allTextMode` which when true, Drill will read 
all JSON values as strings, rather than try to infer the data type.
+* `readNumbersAsDouble`:  By default Drill will attempt to interpret integers, 
floating point number types and strings.  One challenge is when data is 
consistent, Drill may
+  throw schema change exceptions. In addition to `allTextMode`, you can make 
Drill less sensitive by setting the `readNumbersAsDouble` to `true` which 
causes Drill to read all
+  numeric fields in JSON data as `double` data type rather than trying to 
distinguish between ints and doubles.
+* `enableEscapeAnyChar`:  Allows a user to escape any character with a \
+* `skipMalformedRecords`:  Allows Drill to skip malformed records and recover 
without throwing exceptions.
+* `skipMalformedDocument`:  Allows Drill to skip entire malformed documents 
without throwing errors.
+
+All of these can be set by adding the `jsonOptions` to your connection 
configuration as shown below:
+
+```json
+
+"jsonOptions": {
+  "allTextMode": true, 
+  "readNumbersAsDouble": true
+}
+
+```
+
+## Schema Provisioning
+One of the challenges of querying APIs is inconsistent data.  Drill allows you 
to provide a schema for individual endpoints.  You can do this in one of three 
ways: 
+
+1. By providing a schema inline [See: Specifying Schema as Table Function 
Parameter](https://drill.apache.org/docs/plugin-configuration-basics/#specifying-the-schema-as-table-function-parameter)
+2. By providing a schema in the configuration for the endpoint.
+3. By providing a serialized TupleMetadata of the desired schema.  This is an 
advanced functionality and should only be used by advanced Drill users.
+
+The schema provisioning currently supports complex types of Arrays and Maps at 
any nesting level.
+
+### Example Schema Provisioning:
+```json
+"jsonOptions": {
+  "providedSchema": [

Review Comment:
   It is not necessary to have only JSON-supported types because Drill should 
be able to do the auto conversion from one type to another one. `TupleMetadata` 
has also JSON representation (see 
https://drill.apache.org/docs/create-or-replace-schema/#creating-a-schema), so 
at least this format can be used instead of the custom one.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] cgivre closed issue #2523: Due to CVE-2022-26612 (9.8 critical), upgrade Hadoop from 3.2.2 to 3.2.3

[GitBox]

cgivre closed issue #2523: Due to CVE-2022-26612 (9.8 critical), upgrade Hadoop 
from 3.2.2 to 3.2.3
URL: https://github.com/apache/drill/issues/2523


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] cgivre commented on issue #2523: Due to CVE-2022-26612 (9.8 critical), upgrade Hadoop from 3.2.2 to 3.2.3

[GitBox]

cgivre commented on issue #2523:
URL: https://github.com/apache/drill/issues/2523#issuecomment-1115556385

   Fixed in DRILL-8200


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] cgivre commented on a diff in pull request #2526: DRILL-8204: Allow Provided Schema for HTTP Plugin in JSON Mode

[GitBox]

cgivre commented on code in PR #2526:
URL: https://github.com/apache/drill/pull/2526#discussion_r863050797


##
contrib/storage-http/JSON_Options.md:
##
@@ -0,0 +1,125 @@
+# JSON Options and Configuration 
+
+Drill has a collection of JSON configuration options to allow you to configure 
how Drill interprets JSON files.  These are set at the global level, however 
the HTTP plugin
+allows you to configure these options individually per connection and override 
the Drill defaults.  The options are:
+
+* `allowNanInf`:  Configures the connection to interpret `NaN` and `Inf` values
+* `allTextMode`:  By default, Drill attempts to infer data types from JSON 
data. If the data is malformed, Drill may throw schema change exceptions. If 
your data is
+  inconsistent, you can enable `allTextMode` which when true, Drill will read 
all JSON values as strings, rather than try to infer the data type.
+* `readNumbersAsDouble`:  By default Drill will attempt to interpret integers, 
floating point number types and strings.  One challenge is when data is 
consistent, Drill may
+  throw schema change exceptions. In addition to `allTextMode`, you can make 
Drill less sensitive by setting the `readNumbersAsDouble` to `true` which 
causes Drill to read all
+  numeric fields in JSON data as `double` data type rather than trying to 
distinguish between ints and doubles.
+* `enableEscapeAnyChar`:  Allows a user to escape any character with a \
+* `skipMalformedRecords`:  Allows Drill to skip malformed records and recover 
without throwing exceptions.
+* `skipMalformedDocument`:  Allows Drill to skip entire malformed documents 
without throwing errors.
+
+All of these can be set by adding the `jsonOptions` to your connection 
configuration as shown below:
+
+```json
+
+"jsonOptions": {
+  "allTextMode": true, 
+  "readNumbersAsDouble": true
+}
+
+```
+
+## Schema Provisioning
+One of the challenges of querying APIs is inconsistent data.  Drill allows you 
to provide a schema for individual endpoints.  You can do this in one of three 
ways: 
+
+1. By providing a schema inline [See: Specifying Schema as Table Function 
Parameter](https://drill.apache.org/docs/plugin-configuration-basics/#specifying-the-schema-as-table-function-parameter)
+2. By providing a schema in the configuration for the endpoint.
+3. By providing a serialized TupleMetadata of the desired schema.  This is an 
advanced functionality and should only be used by advanced Drill users.
+
+The schema provisioning currently supports complex types of Arrays and Maps at 
any nesting level.
+
+### Example Schema Provisioning:
+```json
+"jsonOptions": {
+  "providedSchema": [

Review Comment:
   You can sort of do that now.  The issue I ran into with the `TupleMetadata` 
representation is that it gets very complicated when you have nested fields.  
The current implementation allows you to either use the simplified 
implementation or you can pass a serialized `TupleMetadata` object.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] cgivre commented on a diff in pull request #2526: DRILL-8204: Allow Provided Schema for HTTP Plugin in JSON Mode

[GitBox]

cgivre commented on code in PR #2526:
URL: https://github.com/apache/drill/pull/2526#discussion_r863277815


##
contrib/storage-http/JSON_Options.md:
##
@@ -0,0 +1,125 @@
+# JSON Options and Configuration 
+
+Drill has a collection of JSON configuration options to allow you to configure 
how Drill interprets JSON files.  These are set at the global level, however 
the HTTP plugin
+allows you to configure these options individually per connection and override 
the Drill defaults.  The options are:
+
+* `allowNanInf`:  Configures the connection to interpret `NaN` and `Inf` values
+* `allTextMode`:  By default, Drill attempts to infer data types from JSON 
data. If the data is malformed, Drill may throw schema change exceptions. If 
your data is
+  inconsistent, you can enable `allTextMode` which when true, Drill will read 
all JSON values as strings, rather than try to infer the data type.
+* `readNumbersAsDouble`:  By default Drill will attempt to interpret integers, 
floating point number types and strings.  One challenge is when data is 
consistent, Drill may
+  throw schema change exceptions. In addition to `allTextMode`, you can make 
Drill less sensitive by setting the `readNumbersAsDouble` to `true` which 
causes Drill to read all
+  numeric fields in JSON data as `double` data type rather than trying to 
distinguish between ints and doubles.
+* `enableEscapeAnyChar`:  Allows a user to escape any character with a \
+* `skipMalformedRecords`:  Allows Drill to skip malformed records and recover 
without throwing exceptions.
+* `skipMalformedDocument`:  Allows Drill to skip entire malformed documents 
without throwing errors.
+
+All of these can be set by adding the `jsonOptions` to your connection 
configuration as shown below:
+
+```json
+
+"jsonOptions": {
+  "allTextMode": true, 
+  "readNumbersAsDouble": true
+}
+
+```
+
+## Schema Provisioning
+One of the challenges of querying APIs is inconsistent data.  Drill allows you 
to provide a schema for individual endpoints.  You can do this in one of three 
ways: 
+
+1. By providing a schema inline [See: Specifying Schema as Table Function 
Parameter](https://drill.apache.org/docs/plugin-configuration-basics/#specifying-the-schema-as-table-function-parameter)
+2. By providing a schema in the configuration for the endpoint.
+3. By providing a serialized TupleMetadata of the desired schema.  This is an 
advanced functionality and should only be used by advanced Drill users.
+
+The schema provisioning currently supports complex types of Arrays and Maps at 
any nesting level.
+
+### Example Schema Provisioning:
+```json
+"jsonOptions": {
+  "providedSchema": [

Review Comment:
   One more thing to mention.  The way I implemented it only supports the data 
types that are supported by the JSON reader. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] cgivre commented on a diff in pull request #2526: DRILL-8204: Allow Provided Schema for HTTP Plugin in JSON Mode

[GitBox]

cgivre commented on code in PR #2526:
URL: https://github.com/apache/drill/pull/2526#discussion_r863050797


##
contrib/storage-http/JSON_Options.md:
##
@@ -0,0 +1,125 @@
+# JSON Options and Configuration 
+
+Drill has a collection of JSON configuration options to allow you to configure 
how Drill interprets JSON files.  These are set at the global level, however 
the HTTP plugin
+allows you to configure these options individually per connection and override 
the Drill defaults.  The options are:
+
+* `allowNanInf`:  Configures the connection to interpret `NaN` and `Inf` values
+* `allTextMode`:  By default, Drill attempts to infer data types from JSON 
data. If the data is malformed, Drill may throw schema change exceptions. If 
your data is
+  inconsistent, you can enable `allTextMode` which when true, Drill will read 
all JSON values as strings, rather than try to infer the data type.
+* `readNumbersAsDouble`:  By default Drill will attempt to interpret integers, 
floating point number types and strings.  One challenge is when data is 
consistent, Drill may
+  throw schema change exceptions. In addition to `allTextMode`, you can make 
Drill less sensitive by setting the `readNumbersAsDouble` to `true` which 
causes Drill to read all
+  numeric fields in JSON data as `double` data type rather than trying to 
distinguish between ints and doubles.
+* `enableEscapeAnyChar`:  Allows a user to escape any character with a \
+* `skipMalformedRecords`:  Allows Drill to skip malformed records and recover 
without throwing exceptions.
+* `skipMalformedDocument`:  Allows Drill to skip entire malformed documents 
without throwing errors.
+
+All of these can be set by adding the `jsonOptions` to your connection 
configuration as shown below:
+
+```json
+
+"jsonOptions": {
+  "allTextMode": true, 
+  "readNumbersAsDouble": true
+}
+
+```
+
+## Schema Provisioning
+One of the challenges of querying APIs is inconsistent data.  Drill allows you 
to provide a schema for individual endpoints.  You can do this in one of three 
ways: 
+
+1. By providing a schema inline [See: Specifying Schema as Table Function 
Parameter](https://drill.apache.org/docs/plugin-configuration-basics/#specifying-the-schema-as-table-function-parameter)
+2. By providing a schema in the configuration for the endpoint.
+3. By providing a serialized TupleMetadata of the desired schema.  This is an 
advanced functionality and should only be used by advanced Drill users.
+
+The schema provisioning currently supports complex types of Arrays and Maps at 
any nesting level.
+
+### Example Schema Provisioning:
+```json
+"jsonOptions": {
+  "providedSchema": [

Review Comment:
   You can sort of do that now.  The issue I ran into with the `TupleMetadata` 
representation is that it gets weird when you have nested fields.  The current 
implementation allows you to either use the simplified implementation or you 
can pass a serialized `TupleMetadata` object.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] vvysotskyi commented on a diff in pull request #2526: DRILL-8204: Allow Provided Schema for HTTP Plugin in JSON Mode

[GitBox]

vvysotskyi commented on code in PR #2526:
URL: https://github.com/apache/drill/pull/2526#discussion_r863046227


##
contrib/storage-http/JSON_Options.md:
##
@@ -0,0 +1,125 @@
+# JSON Options and Configuration 
+
+Drill has a collection of JSON configuration options to allow you to configure 
how Drill interprets JSON files.  These are set at the global level, however 
the HTTP plugin
+allows you to configure these options individually per connection and override 
the Drill defaults.  The options are:
+
+* `allowNanInf`:  Configures the connection to interpret `NaN` and `Inf` values
+* `allTextMode`:  By default, Drill attempts to infer data types from JSON 
data. If the data is malformed, Drill may throw schema change exceptions. If 
your data is
+  inconsistent, you can enable `allTextMode` which when true, Drill will read 
all JSON values as strings, rather than try to infer the data type.
+* `readNumbersAsDouble`:  By default Drill will attempt to interpret integers, 
floating point number types and strings.  One challenge is when data is 
consistent, Drill may
+  throw schema change exceptions. In addition to `allTextMode`, you can make 
Drill less sensitive by setting the `readNumbersAsDouble` to `true` which 
causes Drill to read all
+  numeric fields in JSON data as `double` data type rather than trying to 
distinguish between ints and doubles.
+* `enableEscapeAnyChar`:  Allows a user to escape any character with a \
+* `skipMalformedRecords`:  Allows Drill to skip malformed records and recover 
without throwing exceptions.
+* `skipMalformedDocument`:  Allows Drill to skip entire malformed documents 
without throwing errors.
+
+All of these can be set by adding the `jsonOptions` to your connection 
configuration as shown below:
+
+```json
+
+"jsonOptions": {
+  "allTextMode": true, 
+  "readNumbersAsDouble": true
+}
+
+```
+
+## Schema Provisioning
+One of the challenges of querying APIs is inconsistent data.  Drill allows you 
to provide a schema for individual endpoints.  You can do this in one of three 
ways: 
+
+1. By providing a schema inline [See: Specifying Schema as Table Function 
Parameter](https://drill.apache.org/docs/plugin-configuration-basics/#specifying-the-schema-as-table-function-parameter)
+2. By providing a schema in the configuration for the endpoint.
+3. By providing a serialized TupleMetadata of the desired schema.  This is an 
advanced functionality and should only be used by advanced Drill users.
+
+The schema provisioning currently supports complex types of Arrays and Maps at 
any nesting level.
+
+### Example Schema Provisioning:
+```json
+"jsonOptions": {
+  "providedSchema": [

Review Comment:
   Can we use fields, schema serialization, and deserialization logic from 
TupleMetadata? It has a good human-readable string representation that is much 
more compact than JSON



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] cgivre commented on pull request #2526: DRILL-8204: Allow Provided Schema for HTTP Plugin in JSON Mode

[GitBox]

cgivre commented on PR #2526:
URL: https://github.com/apache/drill/pull/2526#issuecomment-1115142541

   @vvysotskyi Thanks for your comments and assistance!  I got this to work and 
the HTTP plugin now supports inline schema! 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [drill] cgivre commented on pull request #2526: DRILL-8204: Allow Provided Schema for HTTP Plugin in JSON Mode

[GitBox]

cgivre commented on PR #2526:
URL: https://github.com/apache/drill/pull/2526#issuecomment-1115032166

   > 
   
   @vvysotskyi Thanks for the review.  I added the logic to the 
`HttpBatchReader` to support inline schema, however in doing so, I seem to have 
a found a bug with the `SchemaNegotiator` in that the schema always seems to be 
`null` with the inline schema. 
   
   I created [DRILL-8205](https://issues.apache.org/jira/browse/DRILL-8205) to 
address this.  I left the logic and a unit test so once we have a fix for 
DRILL-8205 it should all work as expected. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org