[GitHub] drill pull request: DRILL-4131: Move RPC allocators under Drill's ...

[adeneche]

Github user adeneche commented on a diff in the pull request:

https://github.com/apache/drill/pull/327#discussion_r50024752
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/service/ServiceEngine.java 
---
@@ -56,22 +60,83 @@
   private final DrillConfig config;
   boolean useIP = false;
   private final boolean allowPortHunting;
+  private final BufferAllocator userAllocator;
+  private final BufferAllocator controlAllocator;
+  private final BufferAllocator dataAllocator;
+
 
   public ServiceEngine(ControlMessageHandler controlMessageHandler, 
UserWorker userWorker, BootStrapContext context,
   WorkEventBus workBus, WorkerBee bee, boolean allowPortHunting) 
throws DrillbitStartupException {
+userAllocator = newAllocator(context, "rpc:user", 
"drill.exec.rpc.user.server.memory.reservation",
+"drill.exec.rpc.user.server.memory.maximum");
+controlAllocator = newAllocator(context, "rpc:bit-control",
+"drill.exec.rpc.bit.server.memory.control.reservation", 
"drill.exec.rpc.bit.server.memory.control.maximum");
+dataAllocator = newAllocator(context, "rpc:bit-control",
--- End diff --

should it be named `rpc:bit-data` ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4275: Refactor e/pstore interfaces and t...

[hnfgns]

Github user hnfgns commented on the pull request:

https://github.com/apache/drill/pull/325#issuecomment-172604790
  
Cool. Let's sync up on tomorrow's weekly hangout. Let me know if that does 
not work for you.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4270: Create a separate WindowFramer tha...

[amansinha100]

Github user amansinha100 commented on the pull request:

https://github.com/apache/drill/pull/322#issuecomment-172619347
  
Renaming sounds fine to avoid misinterpretation.  +1. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (DRILL-4280) Kerberos Authentication

[Keys Botzum (JIRA)]

Keys Botzum created DRILL-4280:
--

 Summary: Kerberos Authentication
 Key: DRILL-4280
 URL: https://issues.apache.org/jira/browse/DRILL-4280
 Project: Apache Drill
  Issue Type: Improvement
Reporter: Keys Botzum


Drill should support Kerberos based authentication from clients. This means 
that both the ODBC and JDBC drivers as well as the web/REST interfaces should 
support inbound Kerberos. For Web this would most likely be SPNEGO while for 
ODBC and JDBC this will be more generic Kerberos.

Since Hive and much of Hadoop supports Kerberos there is a potential for a lot 
of reuse of ideas if not implementation.

Note that this is related to but not the same as 
https://issues.apache.org/jira/browse/DRILL-3584 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-4281) Drill should support inbound impersonation

[Keys Botzum (JIRA)]

Keys Botzum created DRILL-4281:
--

 Summary: Drill should support inbound impersonation
 Key: DRILL-4281
 URL: https://issues.apache.org/jira/browse/DRILL-4281
 Project: Apache Drill
  Issue Type: Improvement
Reporter: Keys Botzum


Today Drill supports impersonation *to* external sources. For example I can 
authenticate to Drill as myself and then Drill will access HDFS using 
impersonation

In many scenarios we also need impersonation to Drill. For example I might use 
some front end tool (such as Tableau) and authenticate to it as myself. That 
tool (server version) then needs to access Drill to perform queries and I want 
those queries to run as myself, not as the Tableau user. While in theory the 
intermediate tool could store the userid & password for every user to the Drill 
this isn't a scalable or very secure solution.

Note that HS2 today does support inbound impersonation as described here:  
https://issues.apache.org/jira/browse/HIVE-5155 

The above is not the best approach as it is tied to the connection object which 
is very coarse grained and potentially expensive. It would be better if there 
was a call on the ODBC/JDBC driver to switch the identity on a existing 
connection. Most modern SQL databases (Oracle, DB2) support such function.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] drill pull request: DRILL-4131: Move RPC allocators under Drill's ...

[jacques-n]

Github user jacques-n commented on a diff in the pull request:

https://github.com/apache/drill/pull/327#discussion_r50033210
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/service/ServiceEngine.java 
---
@@ -56,22 +60,83 @@
   private final DrillConfig config;
   boolean useIP = false;
   private final boolean allowPortHunting;
+  private final BufferAllocator userAllocator;
+  private final BufferAllocator controlAllocator;
+  private final BufferAllocator dataAllocator;
+
 
   public ServiceEngine(ControlMessageHandler controlMessageHandler, 
UserWorker userWorker, BootStrapContext context,
   WorkEventBus workBus, WorkerBee bee, boolean allowPortHunting) 
throws DrillbitStartupException {
+userAllocator = newAllocator(context, "rpc:user", 
"drill.exec.rpc.user.server.memory.reservation",
+"drill.exec.rpc.user.server.memory.maximum");
+controlAllocator = newAllocator(context, "rpc:bit-control",
+"drill.exec.rpc.bit.server.memory.control.reservation", 
"drill.exec.rpc.bit.server.memory.control.maximum");
+dataAllocator = newAllocator(context, "rpc:bit-control",
--- End diff --

nice catch


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4131: Move RPC allocators under Drill's ...

[jacques-n]

Github user jacques-n commented on a diff in the pull request:

https://github.com/apache/drill/pull/327#discussion_r50033277
  
--- Diff: 
exec/memory/base/src/main/java/io/netty/buffer/PooledByteBufAllocatorL.java ---
@@ -59,7 +59,7 @@ public PooledByteBufAllocatorL(MetricRegistry registry) {
 
   public UnsafeDirectLittleEndian allocate(int size) {
 try {
-  return allocator.directBuffer(size, size);
+  return allocator.directBuffer(size, Integer.MAX_VALUE);
--- End diff --

The RPC uses ensureWritable to expand buffers. Buffers are expandable to 
maxCapacity. We were coding maxCapacity == capacity which meant that buffers 
weren't expandable and the RPC layer would error out. This fixes this issue.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4131: Move RPC allocators under Drill's ...

[jacques-n]

Github user jacques-n commented on the pull request:

https://github.com/apache/drill/pull/327#issuecomment-172623959
  
One quick note, I still am receiving failures on a small set of unit tests 
because of metric registration conflicts so I'll be fixing those. (Happens only 
in tests where a single JVM contains multiple drillbits since the registry is 
currently static.) Also, I'm concerned that some portion of allocation is not 
getting tracked since the tests that Vicky just posted on DRILL-4266 don't show 
a zero peak allocation for the RPC data layer (which doesn't make any sense 
unless all the queries were single fragmented).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: Bumps FMPP version to 0.9.15

[laurentgo]

GitHub user laurentgo opened a pull request:

https://github.com/apache/drill/pull/330

Bumps FMPP version to 0.9.15

FMPP 0.9.14 pom.xml specifies a version range for FreeMarker dependency,
which should be okay except that latest SNAPSHOT version doesn't resolve
anymore.
FMPP 0.9.15 now uses a fixed stable version (2.3.21).

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/laurentgo/drill laurent/bumps-fmpp-version

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/330.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #330


commit 4d02c2aef8be9b095df5333ec7affadc113fe566
Author: Laurent Goujon 
Date:   2016-01-19T01:28:39Z

Bumps FMPP version to 0.9.15

FMPP 0.9.14 pom.xml specifies a version range for FreeMarker dependency,
which should be okay except that latest SNAPSHOT version doesn't resolve
anymore.
FMPP 0.9.15 now uses a fixed stable version (2.3.21).




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4256: Create HiveConf per HiveStoragePlu...

[vkorukanti]

GitHub user vkorukanti opened a pull request:

https://github.com/apache/drill/pull/329

DRILL-4256: Create HiveConf per HiveStoragePlugin and reuse it wherev…

…er needed.

Creating new instances of HiveConf() are very costly, we should avoid 
creating new ones as much as possible.
Also get rid of hiveConfigOverride and use HiveConf in HiveStoregPlugin 
wherever we need the HiveConf.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vkorukanti/drill DRILL-4256

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/329.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #329


commit 3769dada12dafc7cd9209551e96184c968d19f73
Author: vkorukanti 
Date:   2016-01-11T23:01:02Z

DRILL-4256: Create HiveConf per HiveStoragePlugin and reuse it wherever 
needed.

Creating new instances of HiveConf() are very costly, we should avoid 
creating new ones as much as possible.
Also get rid of hiveConfigOverride and use HiveConf in HiveStoregPlugin 
wherever we need the HiveConf.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4278: Fix issue where WorkspaceConfig wa...

[jacques-n]

GitHub user jacques-n opened a pull request:

https://github.com/apache/drill/pull/331

DRILL-4278: Fix issue where WorkspaceConfig was not returning consist…ent 
hashCode()s for equal objects.

WorkspaceConfig was generating different values which caused us to cache 
separate instances of the plugin.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jacques-n/drill DRILL-4278

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/331.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #331


commit 6faed2f380385390c270c6699c653956b8364826
Author: Jacques Nadeau 
Date:   2016-01-19T01:42:33Z

DRILL-4278: Fix issue where WorkspaceConfig was not returning consistent 
hashCode()s for equal objects.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4275: Refactor e/pstore interfaces and t...

[jacques-n]

Github user jacques-n commented on the pull request:

https://github.com/apache/drill/pull/325#issuecomment-172747355
  
Let's do a separate hangout. This seems to deep/specific to cover in a
large group.
On Jan 18, 2016 9:51 AM, "Hanifi Gunes"  wrote:

> Cool. Let's sync up on tomorrow's weekly hangout. Let me know if that does
> not work for you.
>
> —
> Reply to this email directly or view it on GitHub
> .
>



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[DISCUSS] DRILL-4132

[yuliya Feldman]

Hello here,
I wanted to start discussion on [1]
Would be nice to have a hangout session with @jacques-n, @hnfgns, 
@StevenMPhillips
Let me know suitable time
Thanks,Yuliya
[1] https://issues.apache.org/jira/browse/DRILL-4132