date:20171212

Drill over Hive

2017-12-12 Thread Dor Ben Dov

Hi

This is our query
select * from hive.mytable where key >= '0001:10:2017:0410:3157781' 
  and key < '0001:10:2017:0410:3157782';

Same query in Drill over hbase takes less than a sec, in hive it takes few 
seconds and with drill over hive it quits with a timeout.
When you look in the sql profile of drill (using the web), you see a table full 
scan for millions of records, while actually it was supposed to return 9 rows.

Does Drill on top of hive is using the key to access only the relevant region 
of the table?

Regards,
Dor Ben Dov
Research Manager, Digital - BSS - Research
+972-9-7764043   (office)
+972-52-3573492 (mobile)
[cid:image001.png@01D28AC9.B559B360]

Follow us on Facebook, 
Twitter, 
LinkedIn, 
YouTube, 
Google+ and the Amdocs blog 
network| Digital BSS 
Research@Yammer

This message and the information contained herein is proprietary and 
confidential and subject to the Amdocs policy statement,

you may review at https://www.amdocs.com/about/email-disclaimer

[GitHub] drill issue #1070: DRILL-6004: Direct buffer bounds checking should be disab...

2017-12-12 Thread vrozov

Github user vrozov commented on the issue:

https://github.com/apache/drill/pull/1070
  
@paul-rogers Please review


---

[GitHub] drill pull request #1070: DRILL-6004: Direct buffer bounds checking should b...

2017-12-12 Thread vrozov

GitHub user vrozov opened a pull request:

https://github.com/apache/drill/pull/1070

DRILL-6004: Direct buffer bounds checking should be disabled by default



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vrozov/drill DRILL-6004

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/1070.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1070


commit b863f2a0c9c595dacb924747dea6c449c9ee9917
Author: Vlad Rozov 
Date:   2017-12-12T21:37:38Z

DRILL-6004: Direct buffer bounds checking should be disabled by default




---

Re: [ANNOUNCE] New Committer: Vitalii Diravka

2017-12-12 Thread weijie tong

Congratulations Vitalii

On Wed, 13 Dec 2017 at 5:50 AM Khurram Faraaz  wrote:

> Congratulations Vitalii.
>
>
> Regards,
>
> Khurram
>
> 
> From: Vlad Rozov 
> Sent: Tuesday, December 12, 2017 1:10:09 PM
> To: dev@drill.apache.org
> Subject: Re: [ANNOUNCE] New Committer: Vitalii Diravka
>
> Congrats!
>
> Thank you,
>
> Vlad
>
> On 12/11/17 11:18, Timothy Farkas wrote:
> > Congrats!
> > 
> > From: Gautam Parai 
> > Sent: Monday, December 11, 2017 7:32:27 AM
> > To: dev@drill.apache.org
> > Subject: Re: [ANNOUNCE] New Committer: Vitalii Diravka
> >
> > Congratulations Vitalii!
> >
> >
> > Gautam
> >
> > 
> > From: Vitalii Diravka 
> > Sent: Monday, December 11, 2017 2:36:00 AM
> > To: dev@drill.apache.org
> > Subject: Re: [ANNOUNCE] New Committer: Vitalii Diravka
> >
> > Thanks to everyone! I highly appreciate the opportunity to work with you
> to
> > make Drill better!
> >
> > Kind regards
> > Vitalii
> >
> > On Mon, Dec 11, 2017 at 11:30 AM, Saurabh Mahapatra <
> > saurabhmahapatr...@gmail.com> wrote:
> >
> >> Yes and yes!!! Congrats!
> >>
> >> On Mon, Dec 11, 2017 at 12:28 AM, Arina Yelchiyeva <
> >> arina.yelchiy...@gmail.com> wrote:
> >>
> >>> Congratulations!
> >>>
> >>> Kind regards
> >>> Arina
> >>>
> >>> On Mon, Dec 11, 2017 at 8:06 AM, Prasad Nagaraj Subramanya <
> >>> prasadn...@gmail.com> wrote:
> >>>
>  Congratulations Vitali!
> 
>  Thanks,
>  Prasad
> 
>  On Sun, Dec 10, 2017 at 9:58 PM, Robert Hou  wrote:
> 
> > Congratulations!
> >
> >
> > --Robert
> >
> > 
> > From: Paul Rogers 
> > Sent: Sunday, December 10, 2017 4:29 PM
> > To: dev@drill.apache.org
> > Subject: Re: [ANNOUNCE] New Committer: Vitalii Diravka
> >
> > Congrats! Well deserved.
> >
> > - Paul
> >
> >> On Dec 10, 2017, at 3:16 PM, AnilKumar B 
>  wrote:
> >> Congratulations Vitalii
> >>
> >> Thanks & Regards,
> >> B Anil Kumar.
> >>
> >> On Sun, Dec 10, 2017 at 3:12 PM, rahul challapalli <
> >> challapallira...@gmail.com> wrote:
> >>
> >>> Congratulations Vitalii!
> >>>
> >>> On Sun, Dec 10, 2017 at 3:05 PM, Kunal Khatua 
>  wrote:
>  Congratulations!!
> 
>  -Original Message-
>  From: Aman Sinha [mailto:amansi...@apache.org]
>  Sent: Sunday, December 10, 2017 11:06 AM
>  To: dev@drill.apache.org
>  Subject: [ANNOUNCE] New Committer: Vitalii Diravka
> 
>  The Project Management Committee (PMC) for Apache Drill has
> >> invited
>  Vitalii Diravka  to become a committer, and we are pleased to
>  announce
> >>> that
>  he has accepted.
> 
>  Vitalii has been an active contributor to Drill over the last 1
> >> 1/2
> >>> years.
>  His contributions have spanned areas such as: CASTing issues with
>  Date/Timestamp, Parquet metadata and SQL enhancements, among
> >>> others.
>  Welcome Vitalii, and thank you for your contributions.  Keep up
> >> the
> > good
>  work !
> 
>  - Aman
>  (on behalf of the Apache Drill PMC)
> 
> >
>
>

[GitHub] drill issue #1045: DRILL-5730 Test Mocking Improvements

2017-12-12 Thread ilooner

Github user ilooner commented on the issue:

https://github.com/apache/drill/pull/1045
  
@paul-rogers 

I have split the FragmentContext into Four interfaces. We discussed 3, but 
it felt like 4 was more appropriate since some methods didn't fit well in some 
of the contexts.

- **ExecutorFragmentContext**: This is the highest level context that 
exposes access to everything. This is used by the FragmentExecutor and is also 
passed to the BatchCreators and RootCreators. 
- **RootFragmentContext**: This is exposed to the root executors. It has a 
few methods only needed by the root, but doesn't have access to everything like 
the ExecutorFragmentContext.
- **ExchangeFragmentContext**: This contains the network functions, and is 
used by the senders and recievers.
- **FragmentContext**: This is the barebones context which has no network 
functions and only exposes the bare minimum needed by operators.

These contexts are interfaces that inherit each other in the following way:

ExecutorFragmentContext -> RootFragmentContext -> ExchangeFragmentContext 
-> FragmentContext

There is only one implementation of the interfaces **FragmentContextImpl**. 
The facade pattern is used to cast the context to the appropriate interface 
depending on where it's used.

Let me know if you'd like to change the hierarchy or organization. Once we 
finalize how we want to organize things I will add javadoc.




---

[GitHub] drill pull request #1045: DRILL-5730 Test Mocking Improvements

2017-12-12 Thread ilooner

Github user ilooner commented on a diff in the pull request:

https://github.com/apache/drill/pull/1045#discussion_r156538150
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java ---
@@ -17,483 +17,216 @@
  */
 package org.apache.drill.exec.ops;
 
+import java.io.IOException;
+import java.util.Collection;
 import java.util.List;
-import java.util.Map;
-import java.util.concurrent.Executor;
+import java.util.concurrent.ExecutorService;
 
+import com.google.common.annotations.VisibleForTesting;
 import org.apache.calcite.schema.SchemaPlus;
 import org.apache.drill.common.config.DrillConfig;
-import org.apache.drill.common.exceptions.ExecutionSetupException;
-import org.apache.drill.common.exceptions.UserException;
-import org.apache.drill.common.types.TypeProtos.MinorType;
-import org.apache.drill.exec.ExecConstants;
 import org.apache.drill.exec.compile.CodeCompiler;
-import org.apache.drill.exec.exception.OutOfMemoryException;
+import org.apache.drill.exec.coord.ClusterCoordinator;
+import org.apache.drill.exec.exception.ClassTransformationException;
+import org.apache.drill.exec.expr.ClassGenerator;
+import org.apache.drill.exec.expr.CodeGenerator;
 import org.apache.drill.exec.expr.fn.FunctionImplementationRegistry;
-import org.apache.drill.exec.expr.holders.ValueHolder;
 import org.apache.drill.exec.memory.BufferAllocator;
 import org.apache.drill.exec.physical.base.PhysicalOperator;
-import org.apache.drill.exec.planner.physical.PlannerSettings;
-import org.apache.drill.exec.proto.BitControl.PlanFragment;
-import org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint;
-import org.apache.drill.exec.proto.ExecProtos.FragmentHandle;
-import org.apache.drill.exec.proto.GeneralRPCProtos.Ack;
-import org.apache.drill.exec.proto.helper.QueryIdHelper;
-import org.apache.drill.exec.rpc.RpcException;
-import org.apache.drill.exec.rpc.RpcOutcomeListener;
-import org.apache.drill.exec.rpc.UserClientConnection;
-import org.apache.drill.exec.rpc.control.ControlTunnel;
-import org.apache.drill.exec.rpc.control.WorkEventBus;
-import org.apache.drill.exec.server.DrillbitContext;
-import org.apache.drill.exec.server.options.FragmentOptionManager;
-import org.apache.drill.exec.server.options.OptionList;
-import org.apache.drill.exec.server.options.OptionManager;
+import org.apache.drill.exec.physical.impl.OperatorCreatorRegistry;
+import org.apache.drill.exec.planner.PhysicalPlanReader;
+import org.apache.drill.exec.proto.CoordinationProtos;
+import org.apache.drill.exec.proto.ExecProtos;
+import org.apache.drill.exec.rpc.control.Controller;
 import org.apache.drill.exec.server.options.OptionSet;
-import org.apache.drill.exec.store.PartitionExplorer;
-import org.apache.drill.exec.store.SchemaConfig;
 import org.apache.drill.exec.testing.ExecutionControls;
-import org.apache.drill.exec.util.ImpersonationUtil;
-import org.apache.drill.exec.work.batch.IncomingBuffers;
-
-import com.google.common.annotations.VisibleForTesting;
-import com.google.common.base.Function;
-import com.google.common.base.Preconditions;
-import com.google.common.collect.Lists;
-import com.google.common.collect.Maps;
 
 import io.netty.buffer.DrillBuf;
+import org.apache.drill.exec.work.batch.IncomingBuffers;
 
 /**
- * Contextual objects required for execution of a particular fragment.
- * This is the implementation; use FragmentContextInterface
- * in code to allow tests to use test-time implementations.
+ * Fragment context interface: separates implementation from definition.
+ * Allows unit testing by mocking or reimplementing services with
+ * test-time versions. The name is awkward, chosen to avoid renaming
+ * the implementation class which is used in many places in legacy code.
+ * New code should use this interface, and the names should eventually
+ * be swapped with {@link FragmentContextImpl} becoming
+ * FragmentContextImpl and this interface becoming
+ * {@link FragmentContextImpl}.
  */
 
-public class FragmentContext extends BaseFragmentContext implements 
AutoCloseable, UdfUtilities, FragmentContextInterface {
-  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(FragmentContext.class);
-
-  private final Map tunnels = 
Maps.newHashMap();
-  private final List contexts = Lists.newLinkedList();
-
-  private final DrillbitContext context;
-  private final UserClientConnection connection; // is null if this 
context is for non-root fragment
-  private final QueryContext queryContext; // is null if this context is 
for non-root fragment
-  private final FragmentStats

[GitHub] drill pull request #1045: DRILL-5730 Test Mocking Improvements

2017-12-12 Thread ilooner

Github user ilooner commented on a diff in the pull request:

https://github.com/apache/drill/pull/1045#discussion_r156538123
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java ---
@@ -17,483 +17,216 @@
  */
 package org.apache.drill.exec.ops;
 
+import java.io.IOException;
+import java.util.Collection;
 import java.util.List;
-import java.util.Map;
-import java.util.concurrent.Executor;
+import java.util.concurrent.ExecutorService;
 
+import com.google.common.annotations.VisibleForTesting;
 import org.apache.calcite.schema.SchemaPlus;
 import org.apache.drill.common.config.DrillConfig;
-import org.apache.drill.common.exceptions.ExecutionSetupException;
-import org.apache.drill.common.exceptions.UserException;
-import org.apache.drill.common.types.TypeProtos.MinorType;
-import org.apache.drill.exec.ExecConstants;
 import org.apache.drill.exec.compile.CodeCompiler;
-import org.apache.drill.exec.exception.OutOfMemoryException;
+import org.apache.drill.exec.coord.ClusterCoordinator;
+import org.apache.drill.exec.exception.ClassTransformationException;
+import org.apache.drill.exec.expr.ClassGenerator;
+import org.apache.drill.exec.expr.CodeGenerator;
 import org.apache.drill.exec.expr.fn.FunctionImplementationRegistry;
-import org.apache.drill.exec.expr.holders.ValueHolder;
 import org.apache.drill.exec.memory.BufferAllocator;
 import org.apache.drill.exec.physical.base.PhysicalOperator;
-import org.apache.drill.exec.planner.physical.PlannerSettings;
-import org.apache.drill.exec.proto.BitControl.PlanFragment;
-import org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint;
-import org.apache.drill.exec.proto.ExecProtos.FragmentHandle;
-import org.apache.drill.exec.proto.GeneralRPCProtos.Ack;
-import org.apache.drill.exec.proto.helper.QueryIdHelper;
-import org.apache.drill.exec.rpc.RpcException;
-import org.apache.drill.exec.rpc.RpcOutcomeListener;
-import org.apache.drill.exec.rpc.UserClientConnection;
-import org.apache.drill.exec.rpc.control.ControlTunnel;
-import org.apache.drill.exec.rpc.control.WorkEventBus;
-import org.apache.drill.exec.server.DrillbitContext;
-import org.apache.drill.exec.server.options.FragmentOptionManager;
-import org.apache.drill.exec.server.options.OptionList;
-import org.apache.drill.exec.server.options.OptionManager;
+import org.apache.drill.exec.physical.impl.OperatorCreatorRegistry;
+import org.apache.drill.exec.planner.PhysicalPlanReader;
+import org.apache.drill.exec.proto.CoordinationProtos;
+import org.apache.drill.exec.proto.ExecProtos;
+import org.apache.drill.exec.rpc.control.Controller;
 import org.apache.drill.exec.server.options.OptionSet;
-import org.apache.drill.exec.store.PartitionExplorer;
-import org.apache.drill.exec.store.SchemaConfig;
 import org.apache.drill.exec.testing.ExecutionControls;
-import org.apache.drill.exec.util.ImpersonationUtil;
-import org.apache.drill.exec.work.batch.IncomingBuffers;
-
-import com.google.common.annotations.VisibleForTesting;
-import com.google.common.base.Function;
-import com.google.common.base.Preconditions;
-import com.google.common.collect.Lists;
-import com.google.common.collect.Maps;
 
 import io.netty.buffer.DrillBuf;
+import org.apache.drill.exec.work.batch.IncomingBuffers;
 
 /**
- * Contextual objects required for execution of a particular fragment.
- * This is the implementation; use FragmentContextInterface
- * in code to allow tests to use test-time implementations.
+ * Fragment context interface: separates implementation from definition.
+ * Allows unit testing by mocking or reimplementing services with
+ * test-time versions. The name is awkward, chosen to avoid renaming
+ * the implementation class which is used in many places in legacy code.
+ * New code should use this interface, and the names should eventually
+ * be swapped with {@link FragmentContextImpl} becoming
+ * FragmentContextImpl and this interface becoming
+ * {@link FragmentContextImpl}.
  */
 
-public class FragmentContext extends BaseFragmentContext implements 
AutoCloseable, UdfUtilities, FragmentContextInterface {
-  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(FragmentContext.class);
-
-  private final Map tunnels = 
Maps.newHashMap();
-  private final List contexts = Lists.newLinkedList();
-
-  private final DrillbitContext context;
-  private final UserClientConnection connection; // is null if this 
context is for non-root fragment
-  private final QueryContext queryContext; // is null if this context is 
for non-root fragment
-  private final FragmentStats

[GitHub] drill pull request #1045: DRILL-5730 Test Mocking Improvements

2017-12-12 Thread ilooner

Github user ilooner commented on a diff in the pull request:

https://github.com/apache/drill/pull/1045#discussion_r156534310
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java ---
@@ -17,483 +17,216 @@
  */
 package org.apache.drill.exec.ops;
 
+import java.io.IOException;
+import java.util.Collection;
 import java.util.List;
-import java.util.Map;
-import java.util.concurrent.Executor;
+import java.util.concurrent.ExecutorService;
 
+import com.google.common.annotations.VisibleForTesting;
 import org.apache.calcite.schema.SchemaPlus;
 import org.apache.drill.common.config.DrillConfig;
-import org.apache.drill.common.exceptions.ExecutionSetupException;
-import org.apache.drill.common.exceptions.UserException;
-import org.apache.drill.common.types.TypeProtos.MinorType;
-import org.apache.drill.exec.ExecConstants;
 import org.apache.drill.exec.compile.CodeCompiler;
-import org.apache.drill.exec.exception.OutOfMemoryException;
+import org.apache.drill.exec.coord.ClusterCoordinator;
+import org.apache.drill.exec.exception.ClassTransformationException;
+import org.apache.drill.exec.expr.ClassGenerator;
+import org.apache.drill.exec.expr.CodeGenerator;
 import org.apache.drill.exec.expr.fn.FunctionImplementationRegistry;
-import org.apache.drill.exec.expr.holders.ValueHolder;
 import org.apache.drill.exec.memory.BufferAllocator;
 import org.apache.drill.exec.physical.base.PhysicalOperator;
-import org.apache.drill.exec.planner.physical.PlannerSettings;
-import org.apache.drill.exec.proto.BitControl.PlanFragment;
-import org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint;
-import org.apache.drill.exec.proto.ExecProtos.FragmentHandle;
-import org.apache.drill.exec.proto.GeneralRPCProtos.Ack;
-import org.apache.drill.exec.proto.helper.QueryIdHelper;
-import org.apache.drill.exec.rpc.RpcException;
-import org.apache.drill.exec.rpc.RpcOutcomeListener;
-import org.apache.drill.exec.rpc.UserClientConnection;
-import org.apache.drill.exec.rpc.control.ControlTunnel;
-import org.apache.drill.exec.rpc.control.WorkEventBus;
-import org.apache.drill.exec.server.DrillbitContext;
-import org.apache.drill.exec.server.options.FragmentOptionManager;
-import org.apache.drill.exec.server.options.OptionList;
-import org.apache.drill.exec.server.options.OptionManager;
+import org.apache.drill.exec.physical.impl.OperatorCreatorRegistry;
+import org.apache.drill.exec.planner.PhysicalPlanReader;
+import org.apache.drill.exec.proto.CoordinationProtos;
+import org.apache.drill.exec.proto.ExecProtos;
+import org.apache.drill.exec.rpc.control.Controller;
 import org.apache.drill.exec.server.options.OptionSet;
-import org.apache.drill.exec.store.PartitionExplorer;
-import org.apache.drill.exec.store.SchemaConfig;
 import org.apache.drill.exec.testing.ExecutionControls;
-import org.apache.drill.exec.util.ImpersonationUtil;
-import org.apache.drill.exec.work.batch.IncomingBuffers;
-
-import com.google.common.annotations.VisibleForTesting;
-import com.google.common.base.Function;
-import com.google.common.base.Preconditions;
-import com.google.common.collect.Lists;
-import com.google.common.collect.Maps;
 
 import io.netty.buffer.DrillBuf;
+import org.apache.drill.exec.work.batch.IncomingBuffers;
 
 /**
- * Contextual objects required for execution of a particular fragment.
- * This is the implementation; use FragmentContextInterface
- * in code to allow tests to use test-time implementations.
+ * Fragment context interface: separates implementation from definition.
+ * Allows unit testing by mocking or reimplementing services with
+ * test-time versions. The name is awkward, chosen to avoid renaming
+ * the implementation class which is used in many places in legacy code.
+ * New code should use this interface, and the names should eventually
+ * be swapped with {@link FragmentContextImpl} becoming
+ * FragmentContextImpl and this interface becoming
+ * {@link FragmentContextImpl}.
  */
 
-public class FragmentContext extends BaseFragmentContext implements 
AutoCloseable, UdfUtilities, FragmentContextInterface {
-  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(FragmentContext.class);
-
-  private final Map tunnels = 
Maps.newHashMap();
-  private final List contexts = Lists.newLinkedList();
-
-  private final DrillbitContext context;
-  private final UserClientConnection connection; // is null if this 
context is for non-root fragment
-  private final QueryContext queryContext; // is null if this context is 
for non-root fragment
-  private final FragmentStats

[GitHub] drill pull request #1045: DRILL-5730 Test Mocking Improvements

2017-12-12 Thread ilooner

Github user ilooner commented on a diff in the pull request:

https://github.com/apache/drill/pull/1045#discussion_r156534345
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java ---
@@ -17,483 +17,216 @@
  */
 package org.apache.drill.exec.ops;
 
+import java.io.IOException;
+import java.util.Collection;
 import java.util.List;
-import java.util.Map;
-import java.util.concurrent.Executor;
+import java.util.concurrent.ExecutorService;
 
+import com.google.common.annotations.VisibleForTesting;
 import org.apache.calcite.schema.SchemaPlus;
 import org.apache.drill.common.config.DrillConfig;
-import org.apache.drill.common.exceptions.ExecutionSetupException;
-import org.apache.drill.common.exceptions.UserException;
-import org.apache.drill.common.types.TypeProtos.MinorType;
-import org.apache.drill.exec.ExecConstants;
 import org.apache.drill.exec.compile.CodeCompiler;
-import org.apache.drill.exec.exception.OutOfMemoryException;
+import org.apache.drill.exec.coord.ClusterCoordinator;
+import org.apache.drill.exec.exception.ClassTransformationException;
+import org.apache.drill.exec.expr.ClassGenerator;
+import org.apache.drill.exec.expr.CodeGenerator;
 import org.apache.drill.exec.expr.fn.FunctionImplementationRegistry;
-import org.apache.drill.exec.expr.holders.ValueHolder;
 import org.apache.drill.exec.memory.BufferAllocator;
 import org.apache.drill.exec.physical.base.PhysicalOperator;
-import org.apache.drill.exec.planner.physical.PlannerSettings;
-import org.apache.drill.exec.proto.BitControl.PlanFragment;
-import org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint;
-import org.apache.drill.exec.proto.ExecProtos.FragmentHandle;
-import org.apache.drill.exec.proto.GeneralRPCProtos.Ack;
-import org.apache.drill.exec.proto.helper.QueryIdHelper;
-import org.apache.drill.exec.rpc.RpcException;
-import org.apache.drill.exec.rpc.RpcOutcomeListener;
-import org.apache.drill.exec.rpc.UserClientConnection;
-import org.apache.drill.exec.rpc.control.ControlTunnel;
-import org.apache.drill.exec.rpc.control.WorkEventBus;
-import org.apache.drill.exec.server.DrillbitContext;
-import org.apache.drill.exec.server.options.FragmentOptionManager;
-import org.apache.drill.exec.server.options.OptionList;
-import org.apache.drill.exec.server.options.OptionManager;
+import org.apache.drill.exec.physical.impl.OperatorCreatorRegistry;
+import org.apache.drill.exec.planner.PhysicalPlanReader;
+import org.apache.drill.exec.proto.CoordinationProtos;
+import org.apache.drill.exec.proto.ExecProtos;
+import org.apache.drill.exec.rpc.control.Controller;
 import org.apache.drill.exec.server.options.OptionSet;
-import org.apache.drill.exec.store.PartitionExplorer;
-import org.apache.drill.exec.store.SchemaConfig;
 import org.apache.drill.exec.testing.ExecutionControls;
-import org.apache.drill.exec.util.ImpersonationUtil;
-import org.apache.drill.exec.work.batch.IncomingBuffers;
-
-import com.google.common.annotations.VisibleForTesting;
-import com.google.common.base.Function;
-import com.google.common.base.Preconditions;
-import com.google.common.collect.Lists;
-import com.google.common.collect.Maps;
 
 import io.netty.buffer.DrillBuf;
+import org.apache.drill.exec.work.batch.IncomingBuffers;
 
 /**
- * Contextual objects required for execution of a particular fragment.
- * This is the implementation; use FragmentContextInterface
- * in code to allow tests to use test-time implementations.
+ * Fragment context interface: separates implementation from definition.
+ * Allows unit testing by mocking or reimplementing services with
+ * test-time versions. The name is awkward, chosen to avoid renaming
+ * the implementation class which is used in many places in legacy code.
+ * New code should use this interface, and the names should eventually
+ * be swapped with {@link FragmentContextImpl} becoming
+ * FragmentContextImpl and this interface becoming
+ * {@link FragmentContextImpl}.
  */
 
-public class FragmentContext extends BaseFragmentContext implements 
AutoCloseable, UdfUtilities, FragmentContextInterface {
-  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(FragmentContext.class);
-
-  private final Map tunnels = 
Maps.newHashMap();
-  private final List contexts = Lists.newLinkedList();
-
-  private final DrillbitContext context;
-  private final UserClientConnection connection; // is null if this 
context is for non-root fragment
-  private final QueryContext queryContext; // is null if this context is 
for non-root fragment
-  private final FragmentStats

[GitHub] drill pull request #1045: DRILL-5730 Test Mocking Improvements

2017-12-12 Thread ilooner

Github user ilooner commented on a diff in the pull request:

https://github.com/apache/drill/pull/1045#discussion_r156534369
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java ---
@@ -17,483 +17,216 @@
  */
 package org.apache.drill.exec.ops;
 
+import java.io.IOException;
+import java.util.Collection;
 import java.util.List;
-import java.util.Map;
-import java.util.concurrent.Executor;
+import java.util.concurrent.ExecutorService;
 
+import com.google.common.annotations.VisibleForTesting;
 import org.apache.calcite.schema.SchemaPlus;
 import org.apache.drill.common.config.DrillConfig;
-import org.apache.drill.common.exceptions.ExecutionSetupException;
-import org.apache.drill.common.exceptions.UserException;
-import org.apache.drill.common.types.TypeProtos.MinorType;
-import org.apache.drill.exec.ExecConstants;
 import org.apache.drill.exec.compile.CodeCompiler;
-import org.apache.drill.exec.exception.OutOfMemoryException;
+import org.apache.drill.exec.coord.ClusterCoordinator;
+import org.apache.drill.exec.exception.ClassTransformationException;
+import org.apache.drill.exec.expr.ClassGenerator;
+import org.apache.drill.exec.expr.CodeGenerator;
 import org.apache.drill.exec.expr.fn.FunctionImplementationRegistry;
-import org.apache.drill.exec.expr.holders.ValueHolder;
 import org.apache.drill.exec.memory.BufferAllocator;
 import org.apache.drill.exec.physical.base.PhysicalOperator;
-import org.apache.drill.exec.planner.physical.PlannerSettings;
-import org.apache.drill.exec.proto.BitControl.PlanFragment;
-import org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint;
-import org.apache.drill.exec.proto.ExecProtos.FragmentHandle;
-import org.apache.drill.exec.proto.GeneralRPCProtos.Ack;
-import org.apache.drill.exec.proto.helper.QueryIdHelper;
-import org.apache.drill.exec.rpc.RpcException;
-import org.apache.drill.exec.rpc.RpcOutcomeListener;
-import org.apache.drill.exec.rpc.UserClientConnection;
-import org.apache.drill.exec.rpc.control.ControlTunnel;
-import org.apache.drill.exec.rpc.control.WorkEventBus;
-import org.apache.drill.exec.server.DrillbitContext;
-import org.apache.drill.exec.server.options.FragmentOptionManager;
-import org.apache.drill.exec.server.options.OptionList;
-import org.apache.drill.exec.server.options.OptionManager;
+import org.apache.drill.exec.physical.impl.OperatorCreatorRegistry;
+import org.apache.drill.exec.planner.PhysicalPlanReader;
+import org.apache.drill.exec.proto.CoordinationProtos;
+import org.apache.drill.exec.proto.ExecProtos;
+import org.apache.drill.exec.rpc.control.Controller;
 import org.apache.drill.exec.server.options.OptionSet;
-import org.apache.drill.exec.store.PartitionExplorer;
-import org.apache.drill.exec.store.SchemaConfig;
 import org.apache.drill.exec.testing.ExecutionControls;
-import org.apache.drill.exec.util.ImpersonationUtil;
-import org.apache.drill.exec.work.batch.IncomingBuffers;
-
-import com.google.common.annotations.VisibleForTesting;
-import com.google.common.base.Function;
-import com.google.common.base.Preconditions;
-import com.google.common.collect.Lists;
-import com.google.common.collect.Maps;
 
 import io.netty.buffer.DrillBuf;
+import org.apache.drill.exec.work.batch.IncomingBuffers;
 
 /**
- * Contextual objects required for execution of a particular fragment.
- * This is the implementation; use FragmentContextInterface
- * in code to allow tests to use test-time implementations.
+ * Fragment context interface: separates implementation from definition.
+ * Allows unit testing by mocking or reimplementing services with
+ * test-time versions. The name is awkward, chosen to avoid renaming
+ * the implementation class which is used in many places in legacy code.
+ * New code should use this interface, and the names should eventually
+ * be swapped with {@link FragmentContextImpl} becoming
+ * FragmentContextImpl and this interface becoming
+ * {@link FragmentContextImpl}.
  */
 
-public class FragmentContext extends BaseFragmentContext implements 
AutoCloseable, UdfUtilities, FragmentContextInterface {
-  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(FragmentContext.class);
-
-  private final Map tunnels = 
Maps.newHashMap();
-  private final List contexts = Lists.newLinkedList();
-
-  private final DrillbitContext context;
-  private final UserClientConnection connection; // is null if this 
context is for non-root fragment
-  private final QueryContext queryContext; // is null if this context is 
for non-root fragment
-  private final FragmentStats

[GitHub] drill pull request #1045: DRILL-5730 Test Mocking Improvements

2017-12-12 Thread ilooner

Github user ilooner commented on a diff in the pull request:

https://github.com/apache/drill/pull/1045#discussion_r156534273
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java ---
@@ -17,483 +17,216 @@
  */
 package org.apache.drill.exec.ops;
 
+import java.io.IOException;
+import java.util.Collection;
 import java.util.List;
-import java.util.Map;
-import java.util.concurrent.Executor;
+import java.util.concurrent.ExecutorService;
 
+import com.google.common.annotations.VisibleForTesting;
 import org.apache.calcite.schema.SchemaPlus;
 import org.apache.drill.common.config.DrillConfig;
-import org.apache.drill.common.exceptions.ExecutionSetupException;
-import org.apache.drill.common.exceptions.UserException;
-import org.apache.drill.common.types.TypeProtos.MinorType;
-import org.apache.drill.exec.ExecConstants;
 import org.apache.drill.exec.compile.CodeCompiler;
-import org.apache.drill.exec.exception.OutOfMemoryException;
+import org.apache.drill.exec.coord.ClusterCoordinator;
+import org.apache.drill.exec.exception.ClassTransformationException;
+import org.apache.drill.exec.expr.ClassGenerator;
+import org.apache.drill.exec.expr.CodeGenerator;
 import org.apache.drill.exec.expr.fn.FunctionImplementationRegistry;
-import org.apache.drill.exec.expr.holders.ValueHolder;
 import org.apache.drill.exec.memory.BufferAllocator;
 import org.apache.drill.exec.physical.base.PhysicalOperator;
-import org.apache.drill.exec.planner.physical.PlannerSettings;
-import org.apache.drill.exec.proto.BitControl.PlanFragment;
-import org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint;
-import org.apache.drill.exec.proto.ExecProtos.FragmentHandle;
-import org.apache.drill.exec.proto.GeneralRPCProtos.Ack;
-import org.apache.drill.exec.proto.helper.QueryIdHelper;
-import org.apache.drill.exec.rpc.RpcException;
-import org.apache.drill.exec.rpc.RpcOutcomeListener;
-import org.apache.drill.exec.rpc.UserClientConnection;
-import org.apache.drill.exec.rpc.control.ControlTunnel;
-import org.apache.drill.exec.rpc.control.WorkEventBus;
-import org.apache.drill.exec.server.DrillbitContext;
-import org.apache.drill.exec.server.options.FragmentOptionManager;
-import org.apache.drill.exec.server.options.OptionList;
-import org.apache.drill.exec.server.options.OptionManager;
+import org.apache.drill.exec.physical.impl.OperatorCreatorRegistry;
+import org.apache.drill.exec.planner.PhysicalPlanReader;
+import org.apache.drill.exec.proto.CoordinationProtos;
+import org.apache.drill.exec.proto.ExecProtos;
+import org.apache.drill.exec.rpc.control.Controller;
 import org.apache.drill.exec.server.options.OptionSet;
-import org.apache.drill.exec.store.PartitionExplorer;
-import org.apache.drill.exec.store.SchemaConfig;
 import org.apache.drill.exec.testing.ExecutionControls;
-import org.apache.drill.exec.util.ImpersonationUtil;
-import org.apache.drill.exec.work.batch.IncomingBuffers;
-
-import com.google.common.annotations.VisibleForTesting;
-import com.google.common.base.Function;
-import com.google.common.base.Preconditions;
-import com.google.common.collect.Lists;
-import com.google.common.collect.Maps;
 
 import io.netty.buffer.DrillBuf;
+import org.apache.drill.exec.work.batch.IncomingBuffers;
 
 /**
- * Contextual objects required for execution of a particular fragment.
- * This is the implementation; use FragmentContextInterface
- * in code to allow tests to use test-time implementations.
+ * Fragment context interface: separates implementation from definition.
+ * Allows unit testing by mocking or reimplementing services with
+ * test-time versions. The name is awkward, chosen to avoid renaming
+ * the implementation class which is used in many places in legacy code.
+ * New code should use this interface, and the names should eventually
+ * be swapped with {@link FragmentContextImpl} becoming
+ * FragmentContextImpl and this interface becoming
+ * {@link FragmentContextImpl}.
  */
 
-public class FragmentContext extends BaseFragmentContext implements 
AutoCloseable, UdfUtilities, FragmentContextInterface {
-  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(FragmentContext.class);
-
-  private final Map tunnels = 
Maps.newHashMap();
-  private final List contexts = Lists.newLinkedList();
-
-  private final DrillbitContext context;
-  private final UserClientConnection connection; // is null if this 
context is for non-root fragment
-  private final QueryContext queryContext; // is null if this context is 
for non-root fragment
-  private final FragmentStats

[GitHub] drill pull request #1045: DRILL-5730 Test Mocking Improvements

2017-12-12 Thread ilooner

Github user ilooner commented on a diff in the pull request:

https://github.com/apache/drill/pull/1045#discussion_r156534237
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java ---
@@ -17,483 +17,216 @@
  */
 package org.apache.drill.exec.ops;
 
+import java.io.IOException;
+import java.util.Collection;
 import java.util.List;
-import java.util.Map;
-import java.util.concurrent.Executor;
+import java.util.concurrent.ExecutorService;
 
+import com.google.common.annotations.VisibleForTesting;
 import org.apache.calcite.schema.SchemaPlus;
 import org.apache.drill.common.config.DrillConfig;
-import org.apache.drill.common.exceptions.ExecutionSetupException;
-import org.apache.drill.common.exceptions.UserException;
-import org.apache.drill.common.types.TypeProtos.MinorType;
-import org.apache.drill.exec.ExecConstants;
 import org.apache.drill.exec.compile.CodeCompiler;
-import org.apache.drill.exec.exception.OutOfMemoryException;
+import org.apache.drill.exec.coord.ClusterCoordinator;
+import org.apache.drill.exec.exception.ClassTransformationException;
+import org.apache.drill.exec.expr.ClassGenerator;
+import org.apache.drill.exec.expr.CodeGenerator;
 import org.apache.drill.exec.expr.fn.FunctionImplementationRegistry;
-import org.apache.drill.exec.expr.holders.ValueHolder;
 import org.apache.drill.exec.memory.BufferAllocator;
 import org.apache.drill.exec.physical.base.PhysicalOperator;
-import org.apache.drill.exec.planner.physical.PlannerSettings;
-import org.apache.drill.exec.proto.BitControl.PlanFragment;
-import org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint;
-import org.apache.drill.exec.proto.ExecProtos.FragmentHandle;
-import org.apache.drill.exec.proto.GeneralRPCProtos.Ack;
-import org.apache.drill.exec.proto.helper.QueryIdHelper;
-import org.apache.drill.exec.rpc.RpcException;
-import org.apache.drill.exec.rpc.RpcOutcomeListener;
-import org.apache.drill.exec.rpc.UserClientConnection;
-import org.apache.drill.exec.rpc.control.ControlTunnel;
-import org.apache.drill.exec.rpc.control.WorkEventBus;
-import org.apache.drill.exec.server.DrillbitContext;
-import org.apache.drill.exec.server.options.FragmentOptionManager;
-import org.apache.drill.exec.server.options.OptionList;
-import org.apache.drill.exec.server.options.OptionManager;
+import org.apache.drill.exec.physical.impl.OperatorCreatorRegistry;
+import org.apache.drill.exec.planner.PhysicalPlanReader;
+import org.apache.drill.exec.proto.CoordinationProtos;
+import org.apache.drill.exec.proto.ExecProtos;
+import org.apache.drill.exec.rpc.control.Controller;
 import org.apache.drill.exec.server.options.OptionSet;
-import org.apache.drill.exec.store.PartitionExplorer;
-import org.apache.drill.exec.store.SchemaConfig;
 import org.apache.drill.exec.testing.ExecutionControls;
-import org.apache.drill.exec.util.ImpersonationUtil;
-import org.apache.drill.exec.work.batch.IncomingBuffers;
-
-import com.google.common.annotations.VisibleForTesting;
-import com.google.common.base.Function;
-import com.google.common.base.Preconditions;
-import com.google.common.collect.Lists;
-import com.google.common.collect.Maps;
 
 import io.netty.buffer.DrillBuf;
+import org.apache.drill.exec.work.batch.IncomingBuffers;
 
 /**
- * Contextual objects required for execution of a particular fragment.
- * This is the implementation; use FragmentContextInterface
- * in code to allow tests to use test-time implementations.
+ * Fragment context interface: separates implementation from definition.
+ * Allows unit testing by mocking or reimplementing services with
+ * test-time versions. The name is awkward, chosen to avoid renaming
+ * the implementation class which is used in many places in legacy code.
+ * New code should use this interface, and the names should eventually
+ * be swapped with {@link FragmentContextImpl} becoming
+ * FragmentContextImpl and this interface becoming
+ * {@link FragmentContextImpl}.
  */
 
-public class FragmentContext extends BaseFragmentContext implements 
AutoCloseable, UdfUtilities, FragmentContextInterface {
-  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(FragmentContext.class);
-
-  private final Map tunnels = 
Maps.newHashMap();
-  private final List contexts = Lists.newLinkedList();
-
-  private final DrillbitContext context;
-  private final UserClientConnection connection; // is null if this 
context is for non-root fragment
-  private final QueryContext queryContext; // is null if this context is 
for non-root fragment
-  private final FragmentStats

[jira] [Created] (DRILL-6030) Managed sort should minimize number of batches in a k-way merge

2017-12-12 Thread Vlad Rozov (JIRA)

Vlad Rozov created DRILL-6030:
-

 Summary: Managed sort should minimize number of batches in a k-way 
merge
 Key: DRILL-6030
 URL: https://issues.apache.org/jira/browse/DRILL-6030
 Project: Apache Drill
  Issue Type: Improvement
Reporter: Vlad Rozov
Assignee: Vlad Rozov


The time complexity of the algorithm is O(n*k*log(k)) where k is a number of 
batches to merge and n is a number of records in each batch (assuming equal 
size batches). As n*k is the total number of record to merge and it can be 
quite large, minimizing k should give better results.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [ANNOUNCE] New Committer: Vitalii Diravka

2017-12-12 Thread Khurram Faraaz

Congratulations Vitalii.


Regards,

Khurram


From: Vlad Rozov 
Sent: Tuesday, December 12, 2017 1:10:09 PM
To: dev@drill.apache.org
Subject: Re: [ANNOUNCE] New Committer: Vitalii Diravka

Congrats!

Thank you,

Vlad

On 12/11/17 11:18, Timothy Farkas wrote:
> Congrats!
> 
> From: Gautam Parai 
> Sent: Monday, December 11, 2017 7:32:27 AM
> To: dev@drill.apache.org
> Subject: Re: [ANNOUNCE] New Committer: Vitalii Diravka
>
> Congratulations Vitalii!
>
>
> Gautam
>
> 
> From: Vitalii Diravka 
> Sent: Monday, December 11, 2017 2:36:00 AM
> To: dev@drill.apache.org
> Subject: Re: [ANNOUNCE] New Committer: Vitalii Diravka
>
> Thanks to everyone! I highly appreciate the opportunity to work with you to
> make Drill better!
>
> Kind regards
> Vitalii
>
> On Mon, Dec 11, 2017 at 11:30 AM, Saurabh Mahapatra <
> saurabhmahapatr...@gmail.com> wrote:
>
>> Yes and yes!!! Congrats!
>>
>> On Mon, Dec 11, 2017 at 12:28 AM, Arina Yelchiyeva <
>> arina.yelchiy...@gmail.com> wrote:
>>
>>> Congratulations!
>>>
>>> Kind regards
>>> Arina
>>>
>>> On Mon, Dec 11, 2017 at 8:06 AM, Prasad Nagaraj Subramanya <
>>> prasadn...@gmail.com> wrote:
>>>
 Congratulations Vitali!

 Thanks,
 Prasad

 On Sun, Dec 10, 2017 at 9:58 PM, Robert Hou  wrote:

> Congratulations!
>
>
> --Robert
>
> 
> From: Paul Rogers 
> Sent: Sunday, December 10, 2017 4:29 PM
> To: dev@drill.apache.org
> Subject: Re: [ANNOUNCE] New Committer: Vitalii Diravka
>
> Congrats! Well deserved.
>
> - Paul
>
>> On Dec 10, 2017, at 3:16 PM, AnilKumar B 
 wrote:
>> Congratulations Vitalii
>>
>> Thanks & Regards,
>> B Anil Kumar.
>>
>> On Sun, Dec 10, 2017 at 3:12 PM, rahul challapalli <
>> challapallira...@gmail.com> wrote:
>>
>>> Congratulations Vitalii!
>>>
>>> On Sun, Dec 10, 2017 at 3:05 PM, Kunal Khatua 
 wrote:
 Congratulations!!

 -Original Message-
 From: Aman Sinha [mailto:amansi...@apache.org]
 Sent: Sunday, December 10, 2017 11:06 AM
 To: dev@drill.apache.org
 Subject: [ANNOUNCE] New Committer: Vitalii Diravka

 The Project Management Committee (PMC) for Apache Drill has
>> invited
 Vitalii Diravka  to become a committer, and we are pleased to
 announce
>>> that
 he has accepted.

 Vitalii has been an active contributor to Drill over the last 1
>> 1/2
>>> years.
 His contributions have spanned areas such as: CASTing issues with
 Date/Timestamp, Parquet metadata and SQL enhancements, among
>>> others.
 Welcome Vitalii, and thank you for your contributions.  Keep up
>> the
> good
 work !

 - Aman
 (on behalf of the Apache Drill PMC)

>

[jira] [Created] (DRILL-6029) Add Travis info to Drill contribution doc page

2017-12-12 Thread Bridget Bevens (JIRA)

Bridget Bevens created DRILL-6029:
-

 Summary: Add Travis info to Drill contribution doc page
 Key: DRILL-6029
 URL: https://issues.apache.org/jira/browse/DRILL-6029
 Project: Apache Drill
  Issue Type: Task
  Components: Documentation
Reporter: Bridget Bevens
Assignee: Bridget Bevens






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [ANNOUNCE] New Committer: Vitalii Diravka

2017-12-12 Thread Vlad Rozov


Congrats!

Thank you,

Vlad

On 12/11/17 11:18, Timothy Farkas wrote:

Congrats!

From: Gautam Parai 
Sent: Monday, December 11, 2017 7:32:27 AM
To: dev@drill.apache.org
Subject: Re: [ANNOUNCE] New Committer: Vitalii Diravka

Congratulations Vitalii!


Gautam


From: Vitalii Diravka 
Sent: Monday, December 11, 2017 2:36:00 AM
To: dev@drill.apache.org
Subject: Re: [ANNOUNCE] New Committer: Vitalii Diravka

Thanks to everyone! I highly appreciate the opportunity to work with you to
make Drill better!

Kind regards
Vitalii

On Mon, Dec 11, 2017 at 11:30 AM, Saurabh Mahapatra <
saurabhmahapatr...@gmail.com> wrote:


Yes and yes!!! Congrats!

On Mon, Dec 11, 2017 at 12:28 AM, Arina Yelchiyeva <
arina.yelchiy...@gmail.com> wrote:


Congratulations!

Kind regards
Arina

On Mon, Dec 11, 2017 at 8:06 AM, Prasad Nagaraj Subramanya <
prasadn...@gmail.com> wrote:


Congratulations Vitali!

Thanks,
Prasad

On Sun, Dec 10, 2017 at 9:58 PM, Robert Hou  wrote:


Congratulations!


--Robert


From: Paul Rogers 
Sent: Sunday, December 10, 2017 4:29 PM
To: dev@drill.apache.org
Subject: Re: [ANNOUNCE] New Committer: Vitalii Diravka

Congrats! Well deserved.

- Paul


On Dec 10, 2017, at 3:16 PM, AnilKumar B 

wrote:

Congratulations Vitalii

Thanks & Regards,
B Anil Kumar.

On Sun, Dec 10, 2017 at 3:12 PM, rahul challapalli <
challapallira...@gmail.com> wrote:


Congratulations Vitalii!

On Sun, Dec 10, 2017 at 3:05 PM, Kunal Khatua 

wrote:

Congratulations!!

-Original Message-
From: Aman Sinha [mailto:amansi...@apache.org]
Sent: Sunday, December 10, 2017 11:06 AM
To: dev@drill.apache.org
Subject: [ANNOUNCE] New Committer: Vitalii Diravka

The Project Management Committee (PMC) for Apache Drill has

invited

Vitalii Diravka  to become a committer, and we are pleased to

announce

that

he has accepted.

Vitalii has been an active contributor to Drill over the last 1

1/2

years.

His contributions have spanned areas such as: CASTing issues with
Date/Timestamp, Parquet metadata and SQL enhancements, among

others.

Welcome Vitalii, and thank you for your contributions.  Keep up

the

good

work !

- Aman
(on behalf of the Apache Drill PMC)

[GitHub] drill pull request #1045: DRILL-5730 Test Mocking Improvements

2017-12-12 Thread ilooner

Github user ilooner commented on a diff in the pull request:

https://github.com/apache/drill/pull/1045#discussion_r156494989
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java ---
@@ -17,483 +17,216 @@
  */
 package org.apache.drill.exec.ops;
 
+import java.io.IOException;
+import java.util.Collection;
 import java.util.List;
-import java.util.Map;
-import java.util.concurrent.Executor;
+import java.util.concurrent.ExecutorService;
 
+import com.google.common.annotations.VisibleForTesting;
 import org.apache.calcite.schema.SchemaPlus;
 import org.apache.drill.common.config.DrillConfig;
-import org.apache.drill.common.exceptions.ExecutionSetupException;
-import org.apache.drill.common.exceptions.UserException;
-import org.apache.drill.common.types.TypeProtos.MinorType;
-import org.apache.drill.exec.ExecConstants;
 import org.apache.drill.exec.compile.CodeCompiler;
-import org.apache.drill.exec.exception.OutOfMemoryException;
+import org.apache.drill.exec.coord.ClusterCoordinator;
+import org.apache.drill.exec.exception.ClassTransformationException;
+import org.apache.drill.exec.expr.ClassGenerator;
+import org.apache.drill.exec.expr.CodeGenerator;
 import org.apache.drill.exec.expr.fn.FunctionImplementationRegistry;
-import org.apache.drill.exec.expr.holders.ValueHolder;
 import org.apache.drill.exec.memory.BufferAllocator;
 import org.apache.drill.exec.physical.base.PhysicalOperator;
-import org.apache.drill.exec.planner.physical.PlannerSettings;
-import org.apache.drill.exec.proto.BitControl.PlanFragment;
-import org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint;
-import org.apache.drill.exec.proto.ExecProtos.FragmentHandle;
-import org.apache.drill.exec.proto.GeneralRPCProtos.Ack;
-import org.apache.drill.exec.proto.helper.QueryIdHelper;
-import org.apache.drill.exec.rpc.RpcException;
-import org.apache.drill.exec.rpc.RpcOutcomeListener;
-import org.apache.drill.exec.rpc.UserClientConnection;
-import org.apache.drill.exec.rpc.control.ControlTunnel;
-import org.apache.drill.exec.rpc.control.WorkEventBus;
-import org.apache.drill.exec.server.DrillbitContext;
-import org.apache.drill.exec.server.options.FragmentOptionManager;
-import org.apache.drill.exec.server.options.OptionList;
-import org.apache.drill.exec.server.options.OptionManager;
+import org.apache.drill.exec.physical.impl.OperatorCreatorRegistry;
+import org.apache.drill.exec.planner.PhysicalPlanReader;
+import org.apache.drill.exec.proto.CoordinationProtos;
+import org.apache.drill.exec.proto.ExecProtos;
+import org.apache.drill.exec.rpc.control.Controller;
 import org.apache.drill.exec.server.options.OptionSet;
-import org.apache.drill.exec.store.PartitionExplorer;
-import org.apache.drill.exec.store.SchemaConfig;
 import org.apache.drill.exec.testing.ExecutionControls;
-import org.apache.drill.exec.util.ImpersonationUtil;
-import org.apache.drill.exec.work.batch.IncomingBuffers;
-
-import com.google.common.annotations.VisibleForTesting;
-import com.google.common.base.Function;
-import com.google.common.base.Preconditions;
-import com.google.common.collect.Lists;
-import com.google.common.collect.Maps;
 
 import io.netty.buffer.DrillBuf;
+import org.apache.drill.exec.work.batch.IncomingBuffers;
 
 /**
- * Contextual objects required for execution of a particular fragment.
- * This is the implementation; use FragmentContextInterface
- * in code to allow tests to use test-time implementations.
+ * Fragment context interface: separates implementation from definition.
+ * Allows unit testing by mocking or reimplementing services with
+ * test-time versions. The name is awkward, chosen to avoid renaming
+ * the implementation class which is used in many places in legacy code.
+ * New code should use this interface, and the names should eventually
+ * be swapped with {@link FragmentContextImpl} becoming
+ * FragmentContextImpl and this interface becoming
+ * {@link FragmentContextImpl}.
  */
 
-public class FragmentContext extends BaseFragmentContext implements 
AutoCloseable, UdfUtilities, FragmentContextInterface {
-  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(FragmentContext.class);
-
-  private final Map tunnels = 
Maps.newHashMap();
-  private final List contexts = Lists.newLinkedList();
-
-  private final DrillbitContext context;
-  private final UserClientConnection connection; // is null if this 
context is for non-root fragment
-  private final QueryContext queryContext; // is null if this context is 
for non-root fragment
-  private final FragmentStats

[GitHub] drill pull request #1045: DRILL-5730 Test Mocking Improvements

2017-12-12 Thread ilooner

Github user ilooner commented on a diff in the pull request:

https://github.com/apache/drill/pull/1045#discussion_r156494301
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java ---
@@ -17,483 +17,216 @@
  */
 package org.apache.drill.exec.ops;
 
+import java.io.IOException;
+import java.util.Collection;
 import java.util.List;
-import java.util.Map;
-import java.util.concurrent.Executor;
+import java.util.concurrent.ExecutorService;
 
+import com.google.common.annotations.VisibleForTesting;
 import org.apache.calcite.schema.SchemaPlus;
 import org.apache.drill.common.config.DrillConfig;
-import org.apache.drill.common.exceptions.ExecutionSetupException;
-import org.apache.drill.common.exceptions.UserException;
-import org.apache.drill.common.types.TypeProtos.MinorType;
-import org.apache.drill.exec.ExecConstants;
 import org.apache.drill.exec.compile.CodeCompiler;
-import org.apache.drill.exec.exception.OutOfMemoryException;
+import org.apache.drill.exec.coord.ClusterCoordinator;
+import org.apache.drill.exec.exception.ClassTransformationException;
+import org.apache.drill.exec.expr.ClassGenerator;
+import org.apache.drill.exec.expr.CodeGenerator;
 import org.apache.drill.exec.expr.fn.FunctionImplementationRegistry;
-import org.apache.drill.exec.expr.holders.ValueHolder;
 import org.apache.drill.exec.memory.BufferAllocator;
 import org.apache.drill.exec.physical.base.PhysicalOperator;
-import org.apache.drill.exec.planner.physical.PlannerSettings;
-import org.apache.drill.exec.proto.BitControl.PlanFragment;
-import org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint;
-import org.apache.drill.exec.proto.ExecProtos.FragmentHandle;
-import org.apache.drill.exec.proto.GeneralRPCProtos.Ack;
-import org.apache.drill.exec.proto.helper.QueryIdHelper;
-import org.apache.drill.exec.rpc.RpcException;
-import org.apache.drill.exec.rpc.RpcOutcomeListener;
-import org.apache.drill.exec.rpc.UserClientConnection;
-import org.apache.drill.exec.rpc.control.ControlTunnel;
-import org.apache.drill.exec.rpc.control.WorkEventBus;
-import org.apache.drill.exec.server.DrillbitContext;
-import org.apache.drill.exec.server.options.FragmentOptionManager;
-import org.apache.drill.exec.server.options.OptionList;
-import org.apache.drill.exec.server.options.OptionManager;
+import org.apache.drill.exec.physical.impl.OperatorCreatorRegistry;
+import org.apache.drill.exec.planner.PhysicalPlanReader;
+import org.apache.drill.exec.proto.CoordinationProtos;
+import org.apache.drill.exec.proto.ExecProtos;
+import org.apache.drill.exec.rpc.control.Controller;
 import org.apache.drill.exec.server.options.OptionSet;
-import org.apache.drill.exec.store.PartitionExplorer;
-import org.apache.drill.exec.store.SchemaConfig;
 import org.apache.drill.exec.testing.ExecutionControls;
-import org.apache.drill.exec.util.ImpersonationUtil;
-import org.apache.drill.exec.work.batch.IncomingBuffers;
-
-import com.google.common.annotations.VisibleForTesting;
-import com.google.common.base.Function;
-import com.google.common.base.Preconditions;
-import com.google.common.collect.Lists;
-import com.google.common.collect.Maps;
 
 import io.netty.buffer.DrillBuf;
+import org.apache.drill.exec.work.batch.IncomingBuffers;
 
 /**
- * Contextual objects required for execution of a particular fragment.
- * This is the implementation; use FragmentContextInterface
- * in code to allow tests to use test-time implementations.
+ * Fragment context interface: separates implementation from definition.
+ * Allows unit testing by mocking or reimplementing services with
+ * test-time versions. The name is awkward, chosen to avoid renaming
+ * the implementation class which is used in many places in legacy code.
+ * New code should use this interface, and the names should eventually
+ * be swapped with {@link FragmentContextImpl} becoming
+ * FragmentContextImpl and this interface becoming
+ * {@link FragmentContextImpl}.
  */
 
-public class FragmentContext extends BaseFragmentContext implements 
AutoCloseable, UdfUtilities, FragmentContextInterface {
-  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(FragmentContext.class);
-
-  private final Map tunnels = 
Maps.newHashMap();
-  private final List contexts = Lists.newLinkedList();
-
-  private final DrillbitContext context;
-  private final UserClientConnection connection; // is null if this 
context is for non-root fragment
-  private final QueryContext queryContext; // is null if this context is 
for non-root fragment
-  private final FragmentStats

[GitHub] drill pull request #1045: DRILL-5730 Test Mocking Improvements

2017-12-12 Thread ilooner

Github user ilooner commented on a diff in the pull request:

https://github.com/apache/drill/pull/1045#discussion_r156490496
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java ---
@@ -17,483 +17,216 @@
  */
 package org.apache.drill.exec.ops;
 
+import java.io.IOException;
+import java.util.Collection;
 import java.util.List;
-import java.util.Map;
-import java.util.concurrent.Executor;
+import java.util.concurrent.ExecutorService;
 
+import com.google.common.annotations.VisibleForTesting;
 import org.apache.calcite.schema.SchemaPlus;
 import org.apache.drill.common.config.DrillConfig;
-import org.apache.drill.common.exceptions.ExecutionSetupException;
-import org.apache.drill.common.exceptions.UserException;
-import org.apache.drill.common.types.TypeProtos.MinorType;
-import org.apache.drill.exec.ExecConstants;
 import org.apache.drill.exec.compile.CodeCompiler;
-import org.apache.drill.exec.exception.OutOfMemoryException;
+import org.apache.drill.exec.coord.ClusterCoordinator;
+import org.apache.drill.exec.exception.ClassTransformationException;
+import org.apache.drill.exec.expr.ClassGenerator;
+import org.apache.drill.exec.expr.CodeGenerator;
 import org.apache.drill.exec.expr.fn.FunctionImplementationRegistry;
-import org.apache.drill.exec.expr.holders.ValueHolder;
 import org.apache.drill.exec.memory.BufferAllocator;
 import org.apache.drill.exec.physical.base.PhysicalOperator;
-import org.apache.drill.exec.planner.physical.PlannerSettings;
-import org.apache.drill.exec.proto.BitControl.PlanFragment;
-import org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint;
-import org.apache.drill.exec.proto.ExecProtos.FragmentHandle;
-import org.apache.drill.exec.proto.GeneralRPCProtos.Ack;
-import org.apache.drill.exec.proto.helper.QueryIdHelper;
-import org.apache.drill.exec.rpc.RpcException;
-import org.apache.drill.exec.rpc.RpcOutcomeListener;
-import org.apache.drill.exec.rpc.UserClientConnection;
-import org.apache.drill.exec.rpc.control.ControlTunnel;
-import org.apache.drill.exec.rpc.control.WorkEventBus;
-import org.apache.drill.exec.server.DrillbitContext;
-import org.apache.drill.exec.server.options.FragmentOptionManager;
-import org.apache.drill.exec.server.options.OptionList;
-import org.apache.drill.exec.server.options.OptionManager;
+import org.apache.drill.exec.physical.impl.OperatorCreatorRegistry;
+import org.apache.drill.exec.planner.PhysicalPlanReader;
+import org.apache.drill.exec.proto.CoordinationProtos;
+import org.apache.drill.exec.proto.ExecProtos;
+import org.apache.drill.exec.rpc.control.Controller;
 import org.apache.drill.exec.server.options.OptionSet;
-import org.apache.drill.exec.store.PartitionExplorer;
-import org.apache.drill.exec.store.SchemaConfig;
 import org.apache.drill.exec.testing.ExecutionControls;
-import org.apache.drill.exec.util.ImpersonationUtil;
-import org.apache.drill.exec.work.batch.IncomingBuffers;
-
-import com.google.common.annotations.VisibleForTesting;
-import com.google.common.base.Function;
-import com.google.common.base.Preconditions;
-import com.google.common.collect.Lists;
-import com.google.common.collect.Maps;
 
 import io.netty.buffer.DrillBuf;
+import org.apache.drill.exec.work.batch.IncomingBuffers;
 
 /**
- * Contextual objects required for execution of a particular fragment.
- * This is the implementation; use FragmentContextInterface
- * in code to allow tests to use test-time implementations.
+ * Fragment context interface: separates implementation from definition.
+ * Allows unit testing by mocking or reimplementing services with
+ * test-time versions. The name is awkward, chosen to avoid renaming
+ * the implementation class which is used in many places in legacy code.
+ * New code should use this interface, and the names should eventually
+ * be swapped with {@link FragmentContextImpl} becoming
+ * FragmentContextImpl and this interface becoming
+ * {@link FragmentContextImpl}.
  */
 
-public class FragmentContext extends BaseFragmentContext implements 
AutoCloseable, UdfUtilities, FragmentContextInterface {
-  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(FragmentContext.class);
-
-  private final Map tunnels = 
Maps.newHashMap();
-  private final List contexts = Lists.newLinkedList();
-
-  private final DrillbitContext context;
-  private final UserClientConnection connection; // is null if this 
context is for non-root fragment
-  private final QueryContext queryContext; // is null if this context is 
for non-root fragment
-  private final FragmentStats

Hangout minutes for 12/12/17

2017-12-12 Thread Paul Rogers

Today’s was a very short meeting as many people had other obligations.

Attendees: Vitalli, Volodymyr, Paul

Topics:

* Release 1.12  RC1 is out, please vote
* Thanks to Arina, Vitalli and others on resolving the issues that sank RC0.
* Congratulations to Vitalli on becoming a committer
* Brief discussion of current issues with the JDBC driver

There will be no hangout on 12/26. We’ll convene again in the new year.

Thanks,

- Paul

[GitHub] drill pull request #1045: DRILL-5730 Test Mocking Improvements

2017-12-12 Thread ilooner

Github user ilooner commented on a diff in the pull request:

https://github.com/apache/drill/pull/1045#discussion_r156463711
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java ---
@@ -17,483 +17,216 @@
  */
 package org.apache.drill.exec.ops;
 
+import java.io.IOException;
+import java.util.Collection;
 import java.util.List;
-import java.util.Map;
-import java.util.concurrent.Executor;
+import java.util.concurrent.ExecutorService;
 
+import com.google.common.annotations.VisibleForTesting;
 import org.apache.calcite.schema.SchemaPlus;
 import org.apache.drill.common.config.DrillConfig;
-import org.apache.drill.common.exceptions.ExecutionSetupException;
-import org.apache.drill.common.exceptions.UserException;
-import org.apache.drill.common.types.TypeProtos.MinorType;
-import org.apache.drill.exec.ExecConstants;
 import org.apache.drill.exec.compile.CodeCompiler;
-import org.apache.drill.exec.exception.OutOfMemoryException;
+import org.apache.drill.exec.coord.ClusterCoordinator;
+import org.apache.drill.exec.exception.ClassTransformationException;
+import org.apache.drill.exec.expr.ClassGenerator;
+import org.apache.drill.exec.expr.CodeGenerator;
 import org.apache.drill.exec.expr.fn.FunctionImplementationRegistry;
-import org.apache.drill.exec.expr.holders.ValueHolder;
 import org.apache.drill.exec.memory.BufferAllocator;
 import org.apache.drill.exec.physical.base.PhysicalOperator;
-import org.apache.drill.exec.planner.physical.PlannerSettings;
-import org.apache.drill.exec.proto.BitControl.PlanFragment;
-import org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint;
-import org.apache.drill.exec.proto.ExecProtos.FragmentHandle;
-import org.apache.drill.exec.proto.GeneralRPCProtos.Ack;
-import org.apache.drill.exec.proto.helper.QueryIdHelper;
-import org.apache.drill.exec.rpc.RpcException;
-import org.apache.drill.exec.rpc.RpcOutcomeListener;
-import org.apache.drill.exec.rpc.UserClientConnection;
-import org.apache.drill.exec.rpc.control.ControlTunnel;
-import org.apache.drill.exec.rpc.control.WorkEventBus;
-import org.apache.drill.exec.server.DrillbitContext;
-import org.apache.drill.exec.server.options.FragmentOptionManager;
-import org.apache.drill.exec.server.options.OptionList;
-import org.apache.drill.exec.server.options.OptionManager;
+import org.apache.drill.exec.physical.impl.OperatorCreatorRegistry;
+import org.apache.drill.exec.planner.PhysicalPlanReader;
+import org.apache.drill.exec.proto.CoordinationProtos;
+import org.apache.drill.exec.proto.ExecProtos;
+import org.apache.drill.exec.rpc.control.Controller;
 import org.apache.drill.exec.server.options.OptionSet;
-import org.apache.drill.exec.store.PartitionExplorer;
-import org.apache.drill.exec.store.SchemaConfig;
 import org.apache.drill.exec.testing.ExecutionControls;
-import org.apache.drill.exec.util.ImpersonationUtil;
-import org.apache.drill.exec.work.batch.IncomingBuffers;
-
-import com.google.common.annotations.VisibleForTesting;
-import com.google.common.base.Function;
-import com.google.common.base.Preconditions;
-import com.google.common.collect.Lists;
-import com.google.common.collect.Maps;
 
 import io.netty.buffer.DrillBuf;
+import org.apache.drill.exec.work.batch.IncomingBuffers;
 
 /**
- * Contextual objects required for execution of a particular fragment.
- * This is the implementation; use FragmentContextInterface
- * in code to allow tests to use test-time implementations.
+ * Fragment context interface: separates implementation from definition.
+ * Allows unit testing by mocking or reimplementing services with
+ * test-time versions. The name is awkward, chosen to avoid renaming
+ * the implementation class which is used in many places in legacy code.
+ * New code should use this interface, and the names should eventually
+ * be swapped with {@link FragmentContextImpl} becoming
+ * FragmentContextImpl and this interface becoming
+ * {@link FragmentContextImpl}.
  */
 
-public class FragmentContext extends BaseFragmentContext implements 
AutoCloseable, UdfUtilities, FragmentContextInterface {
-  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(FragmentContext.class);
-
-  private final Map tunnels = 
Maps.newHashMap();
-  private final List contexts = Lists.newLinkedList();
-
-  private final DrillbitContext context;
-  private final UserClientConnection connection; // is null if this 
context is for non-root fragment
-  private final QueryContext queryContext; // is null if this context is 
for non-root fragment
-  private final FragmentStats

[GitHub] drill pull request #1045: DRILL-5730 Test Mocking Improvements

2017-12-12 Thread ilooner

Github user ilooner commented on a diff in the pull request:

https://github.com/apache/drill/pull/1045#discussion_r156460993
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/ops/BaseOperatorContext.java 
---
@@ -161,16 +163,13 @@ public void close() {
 }
 try {
   if (fs != null) {
-fs.close();
-fs = null;
+  fs.close();
--- End diff --

fixed


---

[GitHub] drill pull request #1045: DRILL-5730 Test Mocking Improvements

2017-12-12 Thread ilooner

Github user ilooner commented on a diff in the pull request:

https://github.com/apache/drill/pull/1045#discussion_r156460769
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/ops/BaseOperatorContext.java 
---
@@ -39,27 +39,29 @@
  * version of the operator context and the full production-time context
  * that includes network services.
  */
-
 public abstract class BaseOperatorContext implements OperatorContext {
   static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(BaseOperatorContext.class);
 
-  protected final FragmentContextInterface context;
+  protected final FragmentContext context;
   protected final BufferAllocator allocator;
   protected final PhysicalOperator popConfig;
   protected final BufferManager manager;
+  protected OperatorStatReceiver statsWriter;
   private DrillFileSystem fs;
   private ControlsInjector injector;
 
-  public BaseOperatorContext(FragmentContextInterface context, 
BufferAllocator allocator,
-   PhysicalOperator popConfig) {
+  public BaseOperatorContext(FragmentContext context, BufferAllocator 
allocator,
+ PhysicalOperator popConfig,
+ OperatorStatReceiver stats) {
 this.context = context;
 this.allocator = allocator;
 this.popConfig = popConfig;
 this.manager = new BufferManagerImpl(allocator);
+this.statsWriter = stats;
--- End diff --

done


---

[GitHub] drill pull request #1045: DRILL-5730 Test Mocking Improvements

2017-12-12 Thread ilooner

Github user ilooner commented on a diff in the pull request:

https://github.com/apache/drill/pull/1045#discussion_r156459436
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/ops/BaseOperatorContext.java 
---
@@ -39,27 +39,29 @@
  * version of the operator context and the full production-time context
  * that includes network services.
  */
-
 public abstract class BaseOperatorContext implements OperatorContext {
   static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(BaseOperatorContext.class);
 
-  protected final FragmentContextInterface context;
+  protected final FragmentContext context;
   protected final BufferAllocator allocator;
   protected final PhysicalOperator popConfig;
   protected final BufferManager manager;
+  protected OperatorStatReceiver statsWriter;
--- End diff --

Thanks for catching this. I think it slipped in when I was rebasing an 
resolving conflicts. I will remove it.


---

[GitHub] drill issue #1067: DRILL-3958: Return a valid error message when storage plu...

2017-12-12 Thread kkhatua

Github user kkhatua commented on the issue:

https://github.com/apache/drill/pull/1067
  
Can you put a one line description?


---

[jira] [Created] (DRILL-6028) Allow splitting generated code in ChainedHashTable into blocks to avoid "code too large" error

2017-12-12 Thread Arina Ielchiieva (JIRA)

Arina Ielchiieva created DRILL-6028:
---

 Summary: Allow splitting generated code in ChainedHashTable into 
blocks to avoid "code too large" error
 Key: DRILL-6028
 URL: https://issues.apache.org/jira/browse/DRILL-6028
 Project: Apache Drill
  Issue Type: Improvement
Affects Versions: 1.10.0
Reporter: Arina Ielchiieva
Assignee: Arina Ielchiieva
 Fix For: 1.13.0


Allow splitting generated code in ChainedHashTable into blocks to avoid "code 
too large" error.

REPRODUCE
File {{1200_columns.csv}}
{noformat}
0,1,2,3...1200
0,1,2,3...1200
{noformat}

Query
{noformat}
select columns[0], column[1]...columns[1200] from dfs.`1200_columns.csv`
union
select columns[0], column[1]...columns[1200] from dfs.`1200_columns.csv`
{noformat}

Error
{noformat}
Error: SYSTEM ERROR: CompileException: File 
'org.apache.drill.exec.compile.DrillJavaFileObject[HashTableGen10.java]', Line 
-7886, Column 24: HashTableGen10.java:57650: error: code too large
public boolean isKeyMatchInternalBuild(int incomingRowIdx, int htRowIdx)
   ^ (compiler.err.limit.code)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] drill pull request #1069: DRILL-5994 Added webserver maxThreads configuratio...

2017-12-12 Thread arina-ielchiieva

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/1069#discussion_r156403329
  
--- Diff: exec/java-exec/src/main/resources/drill-module.conf ---
@@ -377,7 +377,9 @@ drill.exec: {
   # refresh time.
   grace_period_ms : 0,
   //port hunting for drillbits. Enabled only for testing purposes.
-  port_hunt : false
+  port_hunt : false,
+  // Max threads of embedded Jetty
+  webserver.threadpool_max: 200
--- End diff --

Please add underscores: `web_server.thread_pool_max`


---

[GitHub] drill pull request #1069: DRILL-5994 Added webserver maxThreads configuratio...

2017-12-12 Thread MitchelLabonte

GitHub user MitchelLabonte opened a pull request:

https://github.com/apache/drill/pull/1069

DRILL-5994 Added webserver maxThreads configuration option to enable â¦

â¦launching on a machine with more than 200 cores

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/MitchelLabonte/drill DRILL-5994

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/1069.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1069


commit af4e99f16be4081f151024688d2fd5cd306b219b
Author: mitchel 
Date:   2017-12-12T15:27:19Z

DRILL-5994 Added webserver maxThreads configuration option to enable 
launching on a machine with more than 200 cores




---

[GitHub] drill pull request #1068: DRILL-6020 Fix NullPointerException when querying ...

2017-12-12 Thread MitchelLabonte

GitHub user MitchelLabonte opened a pull request:

https://github.com/apache/drill/pull/1068

DRILL-6020 Fix NullPointerException when querying JSON untyped path wâ¦

â¦ith Union setting on

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/MitchelLabonte/drill DRILL-6020

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/1068.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1068


commit ccd6af90fc7a4f1fbd8204d84617bd12ff3207d6
Author: mitchel 
Date:   2017-12-12T14:57:55Z

DRILL-6020 Fix NullPointerException when querying JSON untyped path with 
Union setting on




---

[GitHub] drill pull request #1051: DRILL-5963: Query state process improvements

2017-12-12 Thread arina-ielchiieva

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/1051#discussion_r156358222
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java ---
@@ -502,33 +438,82 @@ private void runFragment(List 
fragmentsList) throws ExecutionSetup
   }
 }
 
+assert rootFragment != null;
+
 final FragmentRoot rootOperator;
 try {
   rootOperator = 
drillbitContext.getPlanReader().readFragmentRoot(rootFragment.getFragmentJson());
 } catch (IOException e) {
   throw new ExecutionSetupException(String.format("Unable to parse 
FragmentRoot from fragment: %s", rootFragment.getFragmentJson()));
 }
 queryRM.setCost(rootOperator.getCost());
-admit(null);
-drillbitContext.getWorkBus().addFragmentStatusListener(queryId, 
queryManager.getFragmentStatusListener());
-
drillbitContext.getClusterCoordinator().addDrillbitStatusListener(queryManager.getDrillbitStatusListener());
 
-logger.debug("Submitting fragments to run.");
+fragmentsRunner.setFragmentsInfo(planFragments, rootFragment, 
rootOperator);
 
-// set up the root fragment first so we'll have incoming buffers 
available.
-setupRootFragment(rootFragment, rootOperator);
+startQueryProcessing();
+  }
 
-setupNonRootFragments(planFragments);
+  /**
+   * Enqueues the query and once enqueued, starts sending out query 
fragments for further execution.
+   * Moves query to RUNNING state.
+   */
+  private void startQueryProcessing() {
+enqueue();
+runFragments();
+queryStateProcessor.moveToState(QueryState.RUNNING, null);
+  }
 
-moveToState(QueryState.RUNNING, null);
-logger.debug("Fragments running.");
+  /**
+   * Move query to ENQUEUED state. Enqueues query if queueing is enabled.
+   * Foreman run will be blocked until query is enqueued.
+   * In case of failures (ex: queue timeout exception) will move query to 
FAILED state.
+   */
+  private void enqueue() {
+queryStateProcessor.moveToState(QueryState.ENQUEUED, null);
+
+try {
+  queryRM.admit();
+  queryStateProcessor.moveToState(QueryState.STARTING, null);
+} catch (QueueTimeoutException | QueryQueueException e) {
+  queryStateProcessor.moveToState(QueryState.FAILED, e);
+} finally {
+  String queueName = queryRM.queueName();
+  queryManager.setQueueName(queueName == null ? "Unknown" : queueName);
--- End diff --

@paul-rogers `QueryStateProcessor` is responsible to enforcing valid state 
transitions (directly if fragments were not sent out yet) or though queuing (if 
fragments were sent out), proper cancelling and failing, Also it is responsible 
for query counters. `Foreman` is responsible for execution part and does all 
the work. If we move remaining logic from `Foreman` into  
`QueryStateProcessor`, we'll end up with the same huge unreadable class as 
`Foreman` was before. At this point of refactoring, I suggest we stop and if 
further refactoring is needed, create new Jira.


---

drill vs hive vs native hbase count

2017-12-12 Thread Dor Ben Dov

Hi Everyone,

I tried the same query of 'select count(*) from table x' from hive, drill gui 
(no hive) and natively using hbase count.

The hive and hbase native got the same result, but in Drill - I got a different 
(higher) result.

What might be or causing this gap ?

Regards,
Dor

This message and the information contained herein is proprietary and 
confidential and subject to the Amdocs policy statement,

you may review at https://www.amdocs.com/about/email-disclaimer

[GitHub] drill pull request #1067: DRILL-3958: Return a valid error message when stor...

2017-12-12 Thread rajrahul

GitHub user rajrahul opened a pull request:

https://github.com/apache/drill/pull/1067

DRILL-3958: Return a valid error message when storage plugin fails



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rajrahul/drill DRILL-3958

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/1067.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1067


commit d5db51c795e9a9537c9feb4cf330d3b11045be8f
Author: Rahul Raj 
Date:   2017-12-12T08:51:53Z

DRILL-3958: Return a valid error message when storage plugin fails




---

Drill over Hive

[GitHub] drill issue #1070: DRILL-6004: Direct buffer bounds checking should be disab...

[GitHub] drill pull request #1070: DRILL-6004: Direct buffer bounds checking should b...

Re: [ANNOUNCE] New Committer: Vitalii Diravka

[GitHub] drill issue #1045: DRILL-5730 Test Mocking Improvements

[GitHub] drill pull request #1045: DRILL-5730 Test Mocking Improvements

[GitHub] drill pull request #1045: DRILL-5730 Test Mocking Improvements

[GitHub] drill pull request #1045: DRILL-5730 Test Mocking Improvements

[GitHub] drill pull request #1045: DRILL-5730 Test Mocking Improvements

[GitHub] drill pull request #1045: DRILL-5730 Test Mocking Improvements

[GitHub] drill pull request #1045: DRILL-5730 Test Mocking Improvements

[GitHub] drill pull request #1045: DRILL-5730 Test Mocking Improvements

[jira] [Created] (DRILL-6030) Managed sort should minimize number of batches in a k-way merge

Re: [ANNOUNCE] New Committer: Vitalii Diravka

[jira] [Created] (DRILL-6029) Add Travis info to Drill contribution doc page

Re: [ANNOUNCE] New Committer: Vitalii Diravka

[GitHub] drill pull request #1045: DRILL-5730 Test Mocking Improvements

[GitHub] drill pull request #1045: DRILL-5730 Test Mocking Improvements

[GitHub] drill pull request #1045: DRILL-5730 Test Mocking Improvements

Hangout minutes for 12/12/17

[GitHub] drill pull request #1045: DRILL-5730 Test Mocking Improvements

[GitHub] drill pull request #1045: DRILL-5730 Test Mocking Improvements

[GitHub] drill pull request #1045: DRILL-5730 Test Mocking Improvements

[GitHub] drill pull request #1045: DRILL-5730 Test Mocking Improvements

[GitHub] drill issue #1067: DRILL-3958: Return a valid error message when storage plu...

[jira] [Created] (DRILL-6028) Allow splitting generated code in ChainedHashTable into blocks to avoid "code too large" error

[GitHub] drill pull request #1069: DRILL-5994 Added webserver maxThreads configuratio...

[GitHub] drill pull request #1069: DRILL-5994 Added webserver maxThreads configuratio...

[GitHub] drill pull request #1068: DRILL-6020 Fix NullPointerException when querying ...

[GitHub] drill pull request #1051: DRILL-5963: Query state process improvements

drill vs hive vs native hbase count

[GitHub] drill pull request #1067: DRILL-3958: Return a valid error message when stor...

32 matches

Site Navigation

Mail list logo

Footer information