[GitHub] drill pull request: DRILL-3823, DRILL-4507: Add unit tests.

[hsuanyi]

GitHub user hsuanyi opened a pull request:

https://github.com/apache/drill/pull/490

DRILL-3823, DRILL-4507: Add unit tests.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hsuanyi/incubator-drill DRILL-3823

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/490.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #490


commit 0ba6056baad9981088195c7809fd0abb93a29c94
Author: hsuanyi 
Date:   2016-04-29T03:58:25Z

DRILL-3823, DRILL-4507: Add unit tests.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4573: Zero copy LIKE, REGEXP_MATCHES, SU...

[jcmcote]

Github user jcmcote commented on the pull request:

https://github.com/apache/drill/pull/458#issuecomment-215608196
  
Hey Sean,

You did not apply the correct patch. I made the code review changes you 
asked and updated the patch in the jira ticket. But somehow the code that made 
it into the git repo is the initial patch.

Have a look at the one in the jira ticket you will see it has comments in 
the CharSequenceWrapper.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Permission denied for queries on individual file.

[François Méthot]

Thanks for trying it out.

In our case we don't use impersonation but we do have security enabled.

We will keep investigating, It could be a combination of

Cloudera/HDFS/Drill Security
+ the fact that when we moved to 1.6, we also downgraded from Java 8 _31
 to Java 7 _60 because of this issue: DRILL-4609


I will see if I can reproduce the issue on a separate Cloudera cluster, it
will take time but I will report my finding.


Francois








On Wed, Apr 27, 2016 at 4:56 PM, Abhishek Girish 
wrote:

> Hello Francois,
>
> I tried this on latest master (1.7.0) and on a MapR cluster. Was able to
> query an individual file with just read permissions:
>
> > select c_first_name from dfs.tmp.`customer.parquet` limit 1;
> +---+
> | c_first_name  |
> +---+
> | Javier|
> +---+
> 1 row selected (1.1 seconds)
>
>
> # hadoop fs -ls /tmp/customer.parquet
> -r--r--r--   3 root root7778841 2016-04-27 13:50 /tmp/customer.parquet
>
> Note: I did not use impersonation or have security enabled on my cluster.
>
> -Abhishek
>
> On Wed, Apr 27, 2016 at 12:24 PM, François Méthot 
> wrote:
>
> > Has anyone experienced the same issue?  We are Using HDFS managed by
> > Cloudera Manager.
> >
> > A simple upgrade to 1.6 caused queries done directly on individual files
> to
> > fail with "Permission denied: user=drill, access=EXECUTE" error.
> > Also, Using filter with "dir0" file structure also causes the issue to
> > happen.
> >ex: select col1 from hdfs.`/datasrc/` where dir0>= 1234567;
> >
> > We ended up giving "execute" access to all the data file.
> >
> > We would really like to know if this is the intend to have drill to
> expect
> > execute access permission on data files.
> >
> > Thanks
> >
> > On Tue, Apr 26, 2016 at 11:23 AM, François Méthot 
> > wrote:
> >
> > > Hi,
> > >
> > >   We just switched to version 1.6. Using java 1.7_60
> > >
> > > We noticed that we can no longer query individual files stored in HDFS
> > > from CLI and WebUI.
> > >
> > > select col1 from hdfs.`/data/file1.parquet`;
> > >
> > > Error: SYSTEM ERROR: RemoteException: Permission denied: user=drill,
> > > access=EXECUTE inode=/data/file1.parquet":mygroup:drill:-rw-rw-r--
> > >
> > > If we give execution permission to the file
> > >
> > > hdfs fs -chmod +x /data/file1.parquet
> > >
> > > Then the query works.
> > >
> > > If we query the parent folder (hdfs.`/data/`), the query works as well.
> > >
> > > Is it the expected behavior in 1.6?
> > >
> > > Francois
> > >
> > >
> >
>

[GitHub] drill pull request: DRILL-4132 Ability to submit simple type of ph...

[hnfgns]

Github user hnfgns commented on a diff in the pull request:

https://github.com/apache/drill/pull/368#discussion_r61521260
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/DrillSeparatePlanningTest.java
 ---
@@ -0,0 +1,350 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec;
+
+import static org.junit.Assert.*;
+import io.netty.buffer.DrillBuf;
+
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Properties;
+import java.util.concurrent.ExecutionException;
+
+import org.apache.drill.BaseTestQuery;
+import org.apache.drill.common.DrillAutoCloseables;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.util.TestTools;
+import org.apache.drill.exec.client.DrillClient;
+import org.apache.drill.exec.client.PrintingResultsListener;
+import org.apache.drill.exec.client.QuerySubmitter.Format;
+import org.apache.drill.exec.exception.SchemaChangeException;
+import org.apache.drill.exec.memory.BufferAllocator;
+import org.apache.drill.exec.proto.BitControl.PlanFragment;
+import org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint;
+import org.apache.drill.exec.proto.UserBitShared.QueryData;
+import org.apache.drill.exec.proto.UserBitShared.QueryId;
+import org.apache.drill.exec.proto.UserBitShared.QueryResult.QueryState;
+import org.apache.drill.exec.proto.UserBitShared.QueryType;
+import org.apache.drill.exec.proto.UserProtos.QueryPlanFragments;
+import org.apache.drill.exec.record.RecordBatchLoader;
+import org.apache.drill.exec.record.VectorWrapper;
+import org.apache.drill.exec.rpc.ConnectionThrottle;
+import org.apache.drill.exec.rpc.DrillRpcFuture;
+import org.apache.drill.exec.rpc.RpcException;
+import org.apache.drill.exec.rpc.user.AwaitableUserResultsListener;
+import org.apache.drill.exec.rpc.user.QueryDataBatch;
+import org.apache.drill.exec.rpc.user.UserResultsListener;
+import org.apache.drill.exec.util.VectorUtil;
+import org.apache.drill.exec.vector.ValueVector;
+import org.junit.Test;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+/**
+ * Class to test different planning use cases (separate form query 
execution)
+ *
+ */
+public class DrillSeparatePlanningTest extends BaseTestQuery {
+
+  static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(DrillSeparatePlanningTest.class);
+
+  static final String WORKING_PATH = TestTools.getWorkingPath();
+  static final String TEST_RES_PATH = WORKING_PATH + "/src/test/resources";
+
+  //final String query = "SELECT sales_city, COUNT(*) cnt FROM 
cp.`region.json` GROUP BY sales_city";
+  //final String query = "SELECT * FROM cp.`employee.json` where  
employee_id > 1 and  employee_id < 1000";
+  //final String query = "SELECT o_orderkey, o_custkey FROM 
dfs.tmp.`multilevel` where dir0 = 1995 and o_orderkey > 100 and o_orderkey < 
1000 limit 5";
+  //final String query = "SELECT sum(o_totalprice) FROM 
dfs.tmp.`multilevel` where dir0 = 1995 and o_orderkey > 100 and o_orderkey < 
1000";
+  //final String query = "SELECT o_orderkey FROM dfs.tmp.`multilevel` 
order by o_orderkey";
+  //final String query = "SELECT dir1, sum(o_totalprice) FROM 
dfs.tmp.`multilevel` where dir0 = 1995 group by dir1 order by dir1";
+  //final String query = String.format("SELECT dir0, sum(o_totalprice) 
FROM dfs_test.`%s/multilevel/json` group by dir0 order by dir0", TEST_RES_PATH);
+
+
+  @Test(timeout=3)
+  public void testSingleFragmentQuery() throws Exception {
+final String query = "SELECT * FROM cp.`employee.json` where  
employee_id > 1 and  employee_id < 1000";
+
+QueryPlanFragments planFragments = getFragmentsHelper(query);
+
+assertNotNull(planFragments);
+
+assertEquals(1, planFragments.getFragmentsCount());
+

[GitHub] drill pull request: DRILL-4132 Ability to submit simple type of ph...

[yufeldman]

Github user yufeldman commented on a diff in the pull request:

https://github.com/apache/drill/pull/368#discussion_r61516213
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/DrillSeparatePlanningTest.java
 ---
@@ -0,0 +1,350 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec;
+
+import static org.junit.Assert.*;
+import io.netty.buffer.DrillBuf;
+
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Properties;
+import java.util.concurrent.ExecutionException;
+
+import org.apache.drill.BaseTestQuery;
+import org.apache.drill.common.DrillAutoCloseables;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.util.TestTools;
+import org.apache.drill.exec.client.DrillClient;
+import org.apache.drill.exec.client.PrintingResultsListener;
+import org.apache.drill.exec.client.QuerySubmitter.Format;
+import org.apache.drill.exec.exception.SchemaChangeException;
+import org.apache.drill.exec.memory.BufferAllocator;
+import org.apache.drill.exec.proto.BitControl.PlanFragment;
+import org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint;
+import org.apache.drill.exec.proto.UserBitShared.QueryData;
+import org.apache.drill.exec.proto.UserBitShared.QueryId;
+import org.apache.drill.exec.proto.UserBitShared.QueryResult.QueryState;
+import org.apache.drill.exec.proto.UserBitShared.QueryType;
+import org.apache.drill.exec.proto.UserProtos.QueryPlanFragments;
+import org.apache.drill.exec.record.RecordBatchLoader;
+import org.apache.drill.exec.record.VectorWrapper;
+import org.apache.drill.exec.rpc.ConnectionThrottle;
+import org.apache.drill.exec.rpc.DrillRpcFuture;
+import org.apache.drill.exec.rpc.RpcException;
+import org.apache.drill.exec.rpc.user.AwaitableUserResultsListener;
+import org.apache.drill.exec.rpc.user.QueryDataBatch;
+import org.apache.drill.exec.rpc.user.UserResultsListener;
+import org.apache.drill.exec.util.VectorUtil;
+import org.apache.drill.exec.vector.ValueVector;
+import org.junit.Test;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+/**
+ * Class to test different planning use cases (separate form query 
execution)
+ *
+ */
+public class DrillSeparatePlanningTest extends BaseTestQuery {
+
+  static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(DrillSeparatePlanningTest.class);
+
+  static final String WORKING_PATH = TestTools.getWorkingPath();
+  static final String TEST_RES_PATH = WORKING_PATH + "/src/test/resources";
+
+  //final String query = "SELECT sales_city, COUNT(*) cnt FROM 
cp.`region.json` GROUP BY sales_city";
+  //final String query = "SELECT * FROM cp.`employee.json` where  
employee_id > 1 and  employee_id < 1000";
+  //final String query = "SELECT o_orderkey, o_custkey FROM 
dfs.tmp.`multilevel` where dir0 = 1995 and o_orderkey > 100 and o_orderkey < 
1000 limit 5";
+  //final String query = "SELECT sum(o_totalprice) FROM 
dfs.tmp.`multilevel` where dir0 = 1995 and o_orderkey > 100 and o_orderkey < 
1000";
+  //final String query = "SELECT o_orderkey FROM dfs.tmp.`multilevel` 
order by o_orderkey";
+  //final String query = "SELECT dir1, sum(o_totalprice) FROM 
dfs.tmp.`multilevel` where dir0 = 1995 group by dir1 order by dir1";
+  //final String query = String.format("SELECT dir0, sum(o_totalprice) 
FROM dfs_test.`%s/multilevel/json` group by dir0 order by dir0", TEST_RES_PATH);
+
+
+  @Test(timeout=3)
+  public void testSingleFragmentQuery() throws Exception {
+final String query = "SELECT * FROM cp.`employee.json` where  
employee_id > 1 and  employee_id < 1000";
+
+QueryPlanFragments planFragments = getFragmentsHelper(query);
+
+assertNotNull(planFragments);
+
+assertEquals(1, planFragments.getFragmentsCount());
+   

[GitHub] drill pull request: DRILL-4132 Ability to submit simple type of ph...

[yufeldman]

Github user yufeldman commented on a diff in the pull request:

https://github.com/apache/drill/pull/368#discussion_r61516040
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/work/user/PlanSplitter.java 
---
@@ -0,0 +1,142 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.work.user;
+
+import java.util.List;
+
+import org.apache.drill.exec.ops.QueryContext;
+import org.apache.drill.exec.physical.PhysicalPlan;
+import org.apache.drill.exec.physical.base.PhysicalOperator;
+import org.apache.drill.exec.planner.fragment.Fragment;
+import org.apache.drill.exec.planner.fragment.MakeFragmentsVisitor;
+import org.apache.drill.exec.planner.fragment.SimpleParallelizer;
+import 
org.apache.drill.exec.planner.fragment.contrib.SimpleParallelizerMultiPlans;
+import org.apache.drill.exec.planner.sql.DrillSqlWorker;
+import org.apache.drill.exec.proto.BitControl.PlanFragment;
+import org.apache.drill.exec.proto.UserBitShared.DrillPBError;
+import org.apache.drill.exec.proto.UserBitShared.QueryId;
+import org.apache.drill.exec.proto.UserBitShared.QueryResult.QueryState;
+import org.apache.drill.exec.proto.UserProtos.GetQueryPlanFragments;
+import org.apache.drill.exec.proto.UserProtos.QueryPlanFragments;
+import org.apache.drill.exec.rpc.user.UserServer.UserClientConnection;
+import org.apache.drill.exec.server.DrillbitContext;
+import org.apache.drill.exec.util.MemoryAllocationUtilities;
+import org.apache.drill.exec.util.Pointer;
+import org.apache.drill.exec.util.Utilities;
+import org.apache.drill.exec.work.QueryWorkUnit;
+
+import com.google.common.collect.Lists;
+
+/**
+ * Helper class to return PlanFragments based on the query plan
+ * or based on split query plan
+ *
+ */
+public class PlanSplitter {
+  static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(PlanSplitter.class);
+
+  private static PlanSplitter s_instance = new PlanSplitter();
+
+  private PlanSplitter() {
+
+  }
+
+  public static PlanSplitter getInstance() {
+return s_instance;
+  }
+
+  /**
+   * Method to plan the query and return list of fragments
+   * it will return query plan "as is" or split plans based on the req 
setting: split_plan
+   * @param dContext
+   * @param queryId
+   * @param req
+   * @param connection
+   * @return
+   */
+  public QueryPlanFragments planFragments(DrillbitContext dContext, 
QueryId queryId,
+  GetQueryPlanFragments req, UserClientConnection connection) {
+QueryPlanFragments.Builder responseBuilder = 
QueryPlanFragments.newBuilder();
+QueryContext queryContext = new QueryContext(connection.getSession(), 
dContext, queryId);
+
+responseBuilder.setQueryId(queryId);
+
+try {
+  responseBuilder.addAllFragments(getFragments(dContext, req, 
queryContext, queryId));
+  responseBuilder.setStatus(QueryState.COMPLETED);
+} catch (Exception e) {
+  final String errorMessage = String.format("Failed to produce 
PlanFragments for query id \"%s\" with "
+  + "request to %s plan", queryId, (req.getSplitPlan() ? "split" : 
"no split"));
+  DrillPBError error = 
DrillPBError.newBuilder().setMessage(errorMessage).setErrorType(DrillPBError.ErrorType.PLAN).build();
+
+  responseBuilder.setStatus(QueryState.FAILED);
+  responseBuilder.setError(error);
+}
+return responseBuilder.build();
+  }
+
+  private List getFragments(final DrillbitContext dContext, 
final GetQueryPlanFragments req,
+  final QueryContext queryContext, final QueryId queryId) throws 
Exception {
+final PhysicalPlan plan;
+final String query = req.getQuery();
+switch(req.getType()) {
+case SQL:
+  final Pointer textPlan = new Pointer<>();
+  plan = DrillSqlWorker.getPlan(queryContext, query, textPlan);
+  break;
+case 

[GitHub] drill pull request: DRILL-4132 Ability to submit simple type of ph...

[yufeldman]

Github user yufeldman commented on a diff in the pull request:

https://github.com/apache/drill/pull/368#discussion_r61515624
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/util/Utilities.java ---
@@ -17,13 +17,27 @@
  */
 package org.apache.drill.exec.util;
 
+import java.util.LinkedList;
+import java.util.List;
+
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.exec.ExecConstants;
 import org.apache.drill.exec.expr.fn.impl.DateUtility;
+import org.apache.drill.exec.memory.RootAllocatorFactory;
 import org.apache.drill.exec.ops.FragmentContext;
+import org.apache.drill.exec.ops.QueryContext;
+import org.apache.drill.exec.physical.PhysicalPlan;
+import org.apache.drill.exec.physical.base.PhysicalOperator;
+import org.apache.drill.exec.physical.config.ExternalSort;
 import org.apache.drill.exec.proto.BitControl.QueryContextInformation;
 import org.apache.drill.exec.proto.ExecProtos;
 import org.apache.drill.exec.proto.helper.QueryIdHelper;
+import org.apache.drill.exec.server.options.OptionManager;
 
 public class Utilities {
+
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(Utilities.class);
--- End diff --

yeah, sorry - forgot to remove all the imports, etc. will do


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4132 Ability to submit simple type of ph...

[yufeldman]

Github user yufeldman commented on a diff in the pull request:

https://github.com/apache/drill/pull/368#discussion_r61515592
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/contrib/SimpleParallelizerMultiPlans.java
 ---
@@ -0,0 +1,228 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.fragment.contrib;
+
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.util.DrillStringUtils;
+import org.apache.drill.exec.ops.QueryContext;
+import org.apache.drill.exec.physical.base.Exchange;
+import org.apache.drill.exec.physical.base.FragmentRoot;
+import org.apache.drill.exec.physical.base.PhysicalOperator;
+import org.apache.drill.exec.planner.PhysicalPlanReader;
+import org.apache.drill.exec.planner.fragment.Fragment;
+import org.apache.drill.exec.planner.fragment.PlanningSet;
+import org.apache.drill.exec.planner.fragment.SimpleParallelizer;
+import org.apache.drill.exec.planner.fragment.Wrapper;
+import 
org.apache.drill.exec.planner.fragment.Materializer.IndexedFragmentNode;
+import org.apache.drill.exec.proto.BitControl.PlanFragment;
+import org.apache.drill.exec.proto.BitControl.QueryContextInformation;
+import org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint;
+import org.apache.drill.exec.proto.ExecProtos.FragmentHandle;
+import org.apache.drill.exec.proto.UserBitShared.QueryId;
+import org.apache.drill.exec.rpc.user.UserSession;
+import org.apache.drill.exec.server.options.OptionList;
+import org.apache.drill.exec.work.QueryWorkUnit;
+import org.apache.drill.exec.work.foreman.ForemanSetupException;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Lists;
+
+/**
+ * SimpleParallelizerMultiPlans class is an extension to SimpleParallelizer
+ * to help with getting PlanFragments for split plan.
+ * Split plan is essentially ability to create multiple Physical Operator 
plans from original Physical Operator plan
+ * to be able to run plans separately.
+ * Moving functionality specific to splitting the plan to this class
+ * allows not to pollute parent class with non-authentic functionality
+ *
+ */
+public class SimpleParallelizerMultiPlans extends SimpleParallelizer {
+
+  static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(SimpleParallelizerMultiPlans.class);
+
+  public SimpleParallelizerMultiPlans(QueryContext context) {
+super(context);
+  }
+
+  /**
+   * Create multiple physical plans from original query planning, it will 
allow execute them eventually independently
+   * @param options
+   * @param foremanNode
+   * @param queryId
+   * @param activeEndpoints
+   * @param reader
+   * @param rootFragment
+   * @param session
+   * @param queryContextInfo
+   * @return
+   * @throws ExecutionSetupException
+   */
+  public List getSplitFragments(OptionList options, 
DrillbitEndpoint foremanNode, QueryId queryId,
+  Collection activeEndpoints, PhysicalPlanReader 
reader, Fragment rootFragment,
+  UserSession session, QueryContextInformation queryContextInfo) 
throws ExecutionSetupException {
+
+final PlanningSet planningSet = getFragmentsHelper(activeEndpoints, 
rootFragment);
+
+return generateWorkUnits(
+options, foremanNode, queryId, reader, rootFragment, planningSet, 
session, queryContextInfo);
+  }
+
+  /**
+   * Split plan into multiple plans based on parallelization
+   * Ideally it is applicable only to plans with two major fragments: 
Screen and UnionExchange
+   * But there could be cases where we can remove even multiple exchanges 
like in case of "order by"
--- End diff --

After discussing 

[jira] [Created] (DRILL-4645) Regex_replace() function is broken

[Krystal (JIRA)]

Krystal created DRILL-4645:
--

 Summary: Regex_replace() function is broken
 Key: DRILL-4645
 URL: https://issues.apache.org/jira/browse/DRILL-4645
 Project: Apache Drill
  Issue Type: Bug
  Components: Functions - Drill
Reporter: Krystal


git.commit.id.abbrev=5705d45

select regexp_replace(cast(columns[2] as VARCHAR(4)), 'a', 'b') from `t1.csv` 
limit 2;
+--+
|  EXPR$0  |
+--+
| org.apache.drill.exec.expr.fn.impl.CharSequenceWrapper@7a6e1b48  |
| org.apache.drill.exec.expr.fn.impl.CharSequenceWrapper@7a6e1b48  |
+--+

Content of table t1.csv:

select * from `t1.csv`;
+--+
|   columns|
+--+
| ["1","a","2015-01-01"]   |
| ["2","b","2015-01-02"]   |
| ["3","c","2015-01-03"]   |
| ["4","","2015-01-04"]|
| ["5","e","2015-01-05"]   |
| ["6","f","2015-01-06"]   |
| ["7","g","2015-01-07"]   |
| ["","h","2015-01-08"]|
| ["9","i",""] |
| ["10","j","2015-01-10"]  |
+--+




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] drill pull request: DRILL-4132 Ability to submit simple type of ph...

[hnfgns]

Github user hnfgns commented on the pull request:

https://github.com/apache/drill/pull/368#issuecomment-215548496
  
Overall looks good 👍 Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4132 Ability to submit simple type of ph...

[hnfgns]

Github user hnfgns commented on a diff in the pull request:

https://github.com/apache/drill/pull/368#discussion_r61494824
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/DrillSeparatePlanningTest.java
 ---
@@ -0,0 +1,350 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec;
+
+import static org.junit.Assert.*;
+import io.netty.buffer.DrillBuf;
+
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Properties;
+import java.util.concurrent.ExecutionException;
+
+import org.apache.drill.BaseTestQuery;
+import org.apache.drill.common.DrillAutoCloseables;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.util.TestTools;
+import org.apache.drill.exec.client.DrillClient;
+import org.apache.drill.exec.client.PrintingResultsListener;
+import org.apache.drill.exec.client.QuerySubmitter.Format;
+import org.apache.drill.exec.exception.SchemaChangeException;
+import org.apache.drill.exec.memory.BufferAllocator;
+import org.apache.drill.exec.proto.BitControl.PlanFragment;
+import org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint;
+import org.apache.drill.exec.proto.UserBitShared.QueryData;
+import org.apache.drill.exec.proto.UserBitShared.QueryId;
+import org.apache.drill.exec.proto.UserBitShared.QueryResult.QueryState;
+import org.apache.drill.exec.proto.UserBitShared.QueryType;
+import org.apache.drill.exec.proto.UserProtos.QueryPlanFragments;
+import org.apache.drill.exec.record.RecordBatchLoader;
+import org.apache.drill.exec.record.VectorWrapper;
+import org.apache.drill.exec.rpc.ConnectionThrottle;
+import org.apache.drill.exec.rpc.DrillRpcFuture;
+import org.apache.drill.exec.rpc.RpcException;
+import org.apache.drill.exec.rpc.user.AwaitableUserResultsListener;
+import org.apache.drill.exec.rpc.user.QueryDataBatch;
+import org.apache.drill.exec.rpc.user.UserResultsListener;
+import org.apache.drill.exec.util.VectorUtil;
+import org.apache.drill.exec.vector.ValueVector;
+import org.junit.Test;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+/**
+ * Class to test different planning use cases (separate form query 
execution)
+ *
+ */
+public class DrillSeparatePlanningTest extends BaseTestQuery {
+
+  static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(DrillSeparatePlanningTest.class);
+
+  static final String WORKING_PATH = TestTools.getWorkingPath();
+  static final String TEST_RES_PATH = WORKING_PATH + "/src/test/resources";
+
+  //final String query = "SELECT sales_city, COUNT(*) cnt FROM 
cp.`region.json` GROUP BY sales_city";
+  //final String query = "SELECT * FROM cp.`employee.json` where  
employee_id > 1 and  employee_id < 1000";
+  //final String query = "SELECT o_orderkey, o_custkey FROM 
dfs.tmp.`multilevel` where dir0 = 1995 and o_orderkey > 100 and o_orderkey < 
1000 limit 5";
+  //final String query = "SELECT sum(o_totalprice) FROM 
dfs.tmp.`multilevel` where dir0 = 1995 and o_orderkey > 100 and o_orderkey < 
1000";
+  //final String query = "SELECT o_orderkey FROM dfs.tmp.`multilevel` 
order by o_orderkey";
+  //final String query = "SELECT dir1, sum(o_totalprice) FROM 
dfs.tmp.`multilevel` where dir0 = 1995 group by dir1 order by dir1";
+  //final String query = String.format("SELECT dir0, sum(o_totalprice) 
FROM dfs_test.`%s/multilevel/json` group by dir0 order by dir0", TEST_RES_PATH);
+
+
+  @Test(timeout=3)
+  public void testSingleFragmentQuery() throws Exception {
+final String query = "SELECT * FROM cp.`employee.json` where  
employee_id > 1 and  employee_id < 1000";
+
+QueryPlanFragments planFragments = getFragmentsHelper(query);
+
+assertNotNull(planFragments);
+
+assertEquals(1, planFragments.getFragmentsCount());
+

[GitHub] drill pull request: DRILL-4132 Ability to submit simple type of ph...

[hnfgns]

Github user hnfgns commented on a diff in the pull request:

https://github.com/apache/drill/pull/368#discussion_r61494244
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/work/user/PlanSplitter.java 
---
@@ -0,0 +1,142 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.work.user;
+
+import java.util.List;
+
+import org.apache.drill.exec.ops.QueryContext;
+import org.apache.drill.exec.physical.PhysicalPlan;
+import org.apache.drill.exec.physical.base.PhysicalOperator;
+import org.apache.drill.exec.planner.fragment.Fragment;
+import org.apache.drill.exec.planner.fragment.MakeFragmentsVisitor;
+import org.apache.drill.exec.planner.fragment.SimpleParallelizer;
+import 
org.apache.drill.exec.planner.fragment.contrib.SimpleParallelizerMultiPlans;
+import org.apache.drill.exec.planner.sql.DrillSqlWorker;
+import org.apache.drill.exec.proto.BitControl.PlanFragment;
+import org.apache.drill.exec.proto.UserBitShared.DrillPBError;
+import org.apache.drill.exec.proto.UserBitShared.QueryId;
+import org.apache.drill.exec.proto.UserBitShared.QueryResult.QueryState;
+import org.apache.drill.exec.proto.UserProtos.GetQueryPlanFragments;
+import org.apache.drill.exec.proto.UserProtos.QueryPlanFragments;
+import org.apache.drill.exec.rpc.user.UserServer.UserClientConnection;
+import org.apache.drill.exec.server.DrillbitContext;
+import org.apache.drill.exec.util.MemoryAllocationUtilities;
+import org.apache.drill.exec.util.Pointer;
+import org.apache.drill.exec.util.Utilities;
+import org.apache.drill.exec.work.QueryWorkUnit;
+
+import com.google.common.collect.Lists;
+
+/**
+ * Helper class to return PlanFragments based on the query plan
+ * or based on split query plan
+ *
+ */
+public class PlanSplitter {
+  static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(PlanSplitter.class);
+
+  private static PlanSplitter s_instance = new PlanSplitter();
+
+  private PlanSplitter() {
+
+  }
+
+  public static PlanSplitter getInstance() {
+return s_instance;
+  }
+
+  /**
+   * Method to plan the query and return list of fragments
+   * it will return query plan "as is" or split plans based on the req 
setting: split_plan
+   * @param dContext
+   * @param queryId
+   * @param req
+   * @param connection
+   * @return
+   */
+  public QueryPlanFragments planFragments(DrillbitContext dContext, 
QueryId queryId,
+  GetQueryPlanFragments req, UserClientConnection connection) {
+QueryPlanFragments.Builder responseBuilder = 
QueryPlanFragments.newBuilder();
+QueryContext queryContext = new QueryContext(connection.getSession(), 
dContext, queryId);
+
+responseBuilder.setQueryId(queryId);
+
+try {
+  responseBuilder.addAllFragments(getFragments(dContext, req, 
queryContext, queryId));
+  responseBuilder.setStatus(QueryState.COMPLETED);
+} catch (Exception e) {
+  final String errorMessage = String.format("Failed to produce 
PlanFragments for query id \"%s\" with "
+  + "request to %s plan", queryId, (req.getSplitPlan() ? "split" : 
"no split"));
+  DrillPBError error = 
DrillPBError.newBuilder().setMessage(errorMessage).setErrorType(DrillPBError.ErrorType.PLAN).build();
+
+  responseBuilder.setStatus(QueryState.FAILED);
+  responseBuilder.setError(error);
+}
+return responseBuilder.build();
+  }
+
+  private List getFragments(final DrillbitContext dContext, 
final GetQueryPlanFragments req,
+  final QueryContext queryContext, final QueryId queryId) throws 
Exception {
+final PhysicalPlan plan;
+final String query = req.getQuery();
+switch(req.getType()) {
+case SQL:
+  final Pointer textPlan = new Pointer<>();
+  plan = DrillSqlWorker.getPlan(queryContext, query, textPlan);
+  break;
+case PHYSICAL:
 

[GitHub] drill pull request: DRILL-4132 Ability to submit simple type of ph...

[hnfgns]

Github user hnfgns commented on a diff in the pull request:

https://github.com/apache/drill/pull/368#discussion_r61492206
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/work/user/PlanSplitter.java 
---
@@ -0,0 +1,142 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.work.user;
+
+import java.util.List;
+
+import org.apache.drill.exec.ops.QueryContext;
+import org.apache.drill.exec.physical.PhysicalPlan;
+import org.apache.drill.exec.physical.base.PhysicalOperator;
+import org.apache.drill.exec.planner.fragment.Fragment;
+import org.apache.drill.exec.planner.fragment.MakeFragmentsVisitor;
+import org.apache.drill.exec.planner.fragment.SimpleParallelizer;
+import 
org.apache.drill.exec.planner.fragment.contrib.SimpleParallelizerMultiPlans;
+import org.apache.drill.exec.planner.sql.DrillSqlWorker;
+import org.apache.drill.exec.proto.BitControl.PlanFragment;
+import org.apache.drill.exec.proto.UserBitShared.DrillPBError;
+import org.apache.drill.exec.proto.UserBitShared.QueryId;
+import org.apache.drill.exec.proto.UserBitShared.QueryResult.QueryState;
+import org.apache.drill.exec.proto.UserProtos.GetQueryPlanFragments;
+import org.apache.drill.exec.proto.UserProtos.QueryPlanFragments;
+import org.apache.drill.exec.rpc.user.UserServer.UserClientConnection;
+import org.apache.drill.exec.server.DrillbitContext;
+import org.apache.drill.exec.util.MemoryAllocationUtilities;
+import org.apache.drill.exec.util.Pointer;
+import org.apache.drill.exec.util.Utilities;
+import org.apache.drill.exec.work.QueryWorkUnit;
+
+import com.google.common.collect.Lists;
+
+/**
+ * Helper class to return PlanFragments based on the query plan
+ * or based on split query plan
+ *
+ */
+public class PlanSplitter {
+  static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(PlanSplitter.class);
+
+  private static PlanSplitter s_instance = new PlanSplitter();
+
+  private PlanSplitter() {
+
+  }
+
+  public static PlanSplitter getInstance() {
+return s_instance;
--- End diff --

This seems a very cheap object used once per query. Why to make it 
singleton? We should create it in the stack/heap as we need.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4132 Ability to submit simple type of ph...

[hnfgns]

Github user hnfgns commented on a diff in the pull request:

https://github.com/apache/drill/pull/368#discussion_r61491885
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/util/Utilities.java ---
@@ -17,13 +17,27 @@
  */
 package org.apache.drill.exec.util;
 
+import java.util.LinkedList;
+import java.util.List;
+
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.exec.ExecConstants;
 import org.apache.drill.exec.expr.fn.impl.DateUtility;
+import org.apache.drill.exec.memory.RootAllocatorFactory;
 import org.apache.drill.exec.ops.FragmentContext;
+import org.apache.drill.exec.ops.QueryContext;
+import org.apache.drill.exec.physical.PhysicalPlan;
+import org.apache.drill.exec.physical.base.PhysicalOperator;
+import org.apache.drill.exec.physical.config.ExternalSort;
 import org.apache.drill.exec.proto.BitControl.QueryContextInformation;
 import org.apache.drill.exec.proto.ExecProtos;
 import org.apache.drill.exec.proto.helper.QueryIdHelper;
+import org.apache.drill.exec.server.options.OptionManager;
 
 public class Utilities {
+
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(Utilities.class);
--- End diff --

Please revert the changes in this class after migration of the real logic.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4132 Ability to submit simple type of ph...

[hnfgns]

Github user hnfgns commented on a diff in the pull request:

https://github.com/apache/drill/pull/368#discussion_r61491435
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/contrib/SimpleParallelizerMultiPlans.java
 ---
@@ -0,0 +1,228 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.fragment.contrib;
+
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.util.DrillStringUtils;
+import org.apache.drill.exec.ops.QueryContext;
+import org.apache.drill.exec.physical.base.Exchange;
+import org.apache.drill.exec.physical.base.FragmentRoot;
+import org.apache.drill.exec.physical.base.PhysicalOperator;
+import org.apache.drill.exec.planner.PhysicalPlanReader;
+import org.apache.drill.exec.planner.fragment.Fragment;
+import org.apache.drill.exec.planner.fragment.PlanningSet;
+import org.apache.drill.exec.planner.fragment.SimpleParallelizer;
+import org.apache.drill.exec.planner.fragment.Wrapper;
+import 
org.apache.drill.exec.planner.fragment.Materializer.IndexedFragmentNode;
+import org.apache.drill.exec.proto.BitControl.PlanFragment;
+import org.apache.drill.exec.proto.BitControl.QueryContextInformation;
+import org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint;
+import org.apache.drill.exec.proto.ExecProtos.FragmentHandle;
+import org.apache.drill.exec.proto.UserBitShared.QueryId;
+import org.apache.drill.exec.rpc.user.UserSession;
+import org.apache.drill.exec.server.options.OptionList;
+import org.apache.drill.exec.work.QueryWorkUnit;
+import org.apache.drill.exec.work.foreman.ForemanSetupException;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Lists;
+
+/**
+ * SimpleParallelizerMultiPlans class is an extension to SimpleParallelizer
+ * to help with getting PlanFragments for split plan.
+ * Split plan is essentially ability to create multiple Physical Operator 
plans from original Physical Operator plan
+ * to be able to run plans separately.
+ * Moving functionality specific to splitting the plan to this class
+ * allows not to pollute parent class with non-authentic functionality
+ *
+ */
+public class SimpleParallelizerMultiPlans extends SimpleParallelizer {
+
+  static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(SimpleParallelizerMultiPlans.class);
+
+  public SimpleParallelizerMultiPlans(QueryContext context) {
+super(context);
+  }
+
+  /**
+   * Create multiple physical plans from original query planning, it will 
allow execute them eventually independently
+   * @param options
+   * @param foremanNode
+   * @param queryId
+   * @param activeEndpoints
+   * @param reader
+   * @param rootFragment
+   * @param session
+   * @param queryContextInfo
+   * @return
+   * @throws ExecutionSetupException
+   */
+  public List getSplitFragments(OptionList options, 
DrillbitEndpoint foremanNode, QueryId queryId,
+  Collection activeEndpoints, PhysicalPlanReader 
reader, Fragment rootFragment,
+  UserSession session, QueryContextInformation queryContextInfo) 
throws ExecutionSetupException {
+
+final PlanningSet planningSet = getFragmentsHelper(activeEndpoints, 
rootFragment);
+
+return generateWorkUnits(
+options, foremanNode, queryId, reader, rootFragment, planningSet, 
session, queryContextInfo);
+  }
+
+  /**
+   * Split plan into multiple plans based on parallelization
+   * Ideally it is applicable only to plans with two major fragments: 
Screen and UnionExchange
+   * But there could be cases where we can remove even multiple exchanges 
like in case of "order by"
--- End diff --

I am not comfortable 

[GitHub] drill pull request: DRILL-4132 Ability to submit simple type of ph...

[hnfgns]

Github user hnfgns commented on a diff in the pull request:

https://github.com/apache/drill/pull/368#discussion_r61491214
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/contrib/SimpleParallelizerMultiPlans.java
 ---
@@ -0,0 +1,228 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.fragment.contrib;
+
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.util.DrillStringUtils;
+import org.apache.drill.exec.ops.QueryContext;
+import org.apache.drill.exec.physical.base.Exchange;
+import org.apache.drill.exec.physical.base.FragmentRoot;
+import org.apache.drill.exec.physical.base.PhysicalOperator;
+import org.apache.drill.exec.planner.PhysicalPlanReader;
+import org.apache.drill.exec.planner.fragment.Fragment;
+import org.apache.drill.exec.planner.fragment.PlanningSet;
+import org.apache.drill.exec.planner.fragment.SimpleParallelizer;
+import org.apache.drill.exec.planner.fragment.Wrapper;
+import 
org.apache.drill.exec.planner.fragment.Materializer.IndexedFragmentNode;
+import org.apache.drill.exec.proto.BitControl.PlanFragment;
+import org.apache.drill.exec.proto.BitControl.QueryContextInformation;
+import org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint;
+import org.apache.drill.exec.proto.ExecProtos.FragmentHandle;
+import org.apache.drill.exec.proto.UserBitShared.QueryId;
+import org.apache.drill.exec.rpc.user.UserSession;
+import org.apache.drill.exec.server.options.OptionList;
+import org.apache.drill.exec.work.QueryWorkUnit;
+import org.apache.drill.exec.work.foreman.ForemanSetupException;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Lists;
+
+/**
+ * SimpleParallelizerMultiPlans class is an extension to SimpleParallelizer
+ * to help with getting PlanFragments for split plan.
+ * Split plan is essentially ability to create multiple Physical Operator 
plans from original Physical Operator plan
+ * to be able to run plans separately.
+ * Moving functionality specific to splitting the plan to this class
+ * allows not to pollute parent class with non-authentic functionality
+ *
+ */
+public class SimpleParallelizerMultiPlans extends SimpleParallelizer {
+
+  static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(SimpleParallelizerMultiPlans.class);
+
+  public SimpleParallelizerMultiPlans(QueryContext context) {
+super(context);
+  }
+
+  /**
+   * Create multiple physical plans from original query planning, it will 
allow execute them eventually independently
+   * @param options
+   * @param foremanNode
+   * @param queryId
+   * @param activeEndpoints
+   * @param reader
+   * @param rootFragment
+   * @param session
+   * @param queryContextInfo
+   * @return
+   * @throws ExecutionSetupException
+   */
+  public List getSplitFragments(OptionList options, 
DrillbitEndpoint foremanNode, QueryId queryId,
--- End diff --

Name advice: splitFragments/split.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4132 Ability to submit simple type of ph...

[hnfgns]

Github user hnfgns commented on a diff in the pull request:

https://github.com/apache/drill/pull/368#discussion_r61491072
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/contrib/SimpleParallelizerMultiPlans.java
 ---
@@ -0,0 +1,228 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.fragment.contrib;
+
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.util.DrillStringUtils;
+import org.apache.drill.exec.ops.QueryContext;
+import org.apache.drill.exec.physical.base.Exchange;
+import org.apache.drill.exec.physical.base.FragmentRoot;
+import org.apache.drill.exec.physical.base.PhysicalOperator;
+import org.apache.drill.exec.planner.PhysicalPlanReader;
+import org.apache.drill.exec.planner.fragment.Fragment;
+import org.apache.drill.exec.planner.fragment.PlanningSet;
+import org.apache.drill.exec.planner.fragment.SimpleParallelizer;
+import org.apache.drill.exec.planner.fragment.Wrapper;
+import 
org.apache.drill.exec.planner.fragment.Materializer.IndexedFragmentNode;
+import org.apache.drill.exec.proto.BitControl.PlanFragment;
+import org.apache.drill.exec.proto.BitControl.QueryContextInformation;
+import org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint;
+import org.apache.drill.exec.proto.ExecProtos.FragmentHandle;
+import org.apache.drill.exec.proto.UserBitShared.QueryId;
+import org.apache.drill.exec.rpc.user.UserSession;
+import org.apache.drill.exec.server.options.OptionList;
+import org.apache.drill.exec.work.QueryWorkUnit;
+import org.apache.drill.exec.work.foreman.ForemanSetupException;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Lists;
+
+/**
+ * SimpleParallelizerMultiPlans class is an extension to SimpleParallelizer
+ * to help with getting PlanFragments for split plan.
+ * Split plan is essentially ability to create multiple Physical Operator 
plans from original Physical Operator plan
+ * to be able to run plans separately.
+ * Moving functionality specific to splitting the plan to this class
+ * allows not to pollute parent class with non-authentic functionality
+ *
+ */
+public class SimpleParallelizerMultiPlans extends SimpleParallelizer {
--- End diff --

Name advise: SplittingParallelizer + documentation seems pretty good here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4132 Ability to submit simple type of ph...

[hnfgns]

Github user hnfgns commented on a diff in the pull request:

https://github.com/apache/drill/pull/368#discussion_r61490732
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java ---
@@ -321,6 +329,38 @@ public void close() {
 return listener.getResults();
   }
 
+  public DrillRpcFuture planQuery(QueryType type, 
String query, boolean isSplitPlan) {
+GetQueryPlanFragments runQuery = 
GetQueryPlanFragments.newBuilder().setQuery(query).setType(type).setSplitPlan(isSplitPlan).build();
+return client.submitPlanQuery(runQuery);
+  }
+
+  public void runQuery(QueryType type, List planFragments, 
UserResultsListener resultsListener)
+  throws RpcException {
+// QueryType can be only executional
+checkArgument((QueryType.EXECUTION == type), "Only EXECUTIONAL type 
query is supported with PlanFragments");
+// setting Plan on RunQuery will be used for logging purposes and 
therefore can not be null
+// since there is no Plan string provided we will create a JsonArray 
out of individual fragment Plans
+ArrayNode jsonArray = objectMapper.createArrayNode();
+for (PlanFragment fragment : planFragments) {
+  try {
+jsonArray.add(objectMapper.readTree(fragment.getFragmentJson()));
+  } catch (IOException e) {
+logger.error("Exception while trying to read PlanFragment JSON for 
%s", fragment.getHandle().getQueryId(), e);
+throw new RpcException(e);
+  }
+}
+final String fragmentsToJsonString;
+try {
+  fragmentsToJsonString = objectMapper.writeValueAsString(jsonArray);
--- End diff --

That's because we are not telling Jackson how to handle cyclicity. Can you 
try using DrillbitContext#getPlanReader instead?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Store format options in CTAS

[Amit Hadke]

I've created https://issues.apache.org/jira/browse/DRILL-4644

Trying to move session options into CTAS grammar.
It would be great if I can get a feedback on it.

Other options are -
STORE WITH or WITH (type => 'csv', fieldDelimiter => ',')
USING(type => 'csv', fieldDelimiter => ',')
use past tense STORED AS (type => 'csv', fieldDelimiter => ',')
STORAGE (type => 'csv', fieldDelimiter => ',')

Thanks,
~ Amit.

[jira] [Created] (DRILL-4644) Allow setting format options in CTAS

[amit hadke (JIRA)]

amit hadke created DRILL-4644:
-

 Summary: Allow setting format options in CTAS
 Key: DRILL-4644
 URL: https://issues.apache.org/jira/browse/DRILL-4644
 Project: Apache Drill
  Issue Type: Improvement
Reporter: amit hadke
Assignee: amit hadke
Priority: Minor


User has to set session options in order to specify how CTAS should store a 
table.(store.format='json')

Add a new option in CTAS 'STORE AS' that takes optional parameters similar to 
format attributes used in table functions.

For example

CREATE TABLE  STORE AS(type => 'csv',
fieldDelimiter => ',') AS 




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] drill pull request: DRILL-4642: Remove customized RexBuilder.ensur...

[hsuanyi]

Github user hsuanyi commented on the pull request:

https://github.com/apache/drill/pull/489#issuecomment-215490663
  
@jinfengni Can you help review?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4642: Remove customized RexBuilder.ensur...

[hsuanyi]

Github user hsuanyi commented on the pull request:

https://github.com/apache/drill/pull/489#issuecomment-215490190
  
DrillCalcite can be found at here:
https://github.com/hsuanyi/incubator-calcite/tree/DRILL-4642


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4642: Remove customized RexBuilder.ensur...

[hsuanyi]

GitHub user hsuanyi opened a pull request:

https://github.com/apache/drill/pull/489

DRILL-4642: Remove customized RexBuilder.ensureType()

Bump calcite version to 1.4.0-drill-r12



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hsuanyi/incubator-drill DRILL-4642

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/489.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #489


commit 163e7b713fbb1a21603f4c1de77acd0481088158
Author: Hsuan-Yi Chu 
Date:   2016-04-25T22:09:01Z

DRILL-4642: Remove customized RexBuilder.ensureType()

Bump calcite version to 1.4.0-drill-r12




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-3894: Upgrade functions MaxDir, MinDir.....

[vdiravka]

Github user vdiravka commented on the pull request:

https://github.com/apache/drill/pull/467#issuecomment-215476544
  
I used the same approach like in lpad .. functions. You can use Query 
Directory Functions with filename param or without it.
And for code cleaning I've replaced `MaxDirOnePar` with `MaxDir`, 
`MaxDirTwoPar` with `MaxDirTwoArg` like in the lpad functions and I've replaced 
`parameterAmount==2` label with `functionClassName?ends_with("TwoArg")`. 
Also I've added UTest for new Query Directory Functions implementations.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: Update 005-supported-sql-commands.md

[limbosongs]

GitHub user limbosongs opened a pull request:

https://github.com/apache/drill/pull/488

Update 005-supported-sql-commands.md

Fixed link to CTAS

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/limbosongs/drill patch-1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/488.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #488


commit 1a067069fdb3659555e9cfeb4646990e56f1ee7f
Author: limbosongs 
Date:   2016-04-28T08:13:42Z

Update 005-supported-sql-commands.md

Fixed link to CTAS




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4132 Ability to submit simple type of ph...

[yufeldman]

Github user yufeldman commented on the pull request:

https://github.com/apache/drill/pull/368#issuecomment-215326874
  
addressed review comments


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---