[GitHub] drill issue #1082: DRILL-5741: Automatically manage memory allocations durin...

[kkhatua]

Github user kkhatua commented on the issue:

https://github.com/apache/drill/pull/1082
  
Done the following changes:
1. The file names have been changed to the following:
* `./bin/auto-setup.sh`
* `./conf/distrib-setup.sh`
* `./conf/drill-setup.sh`
`auto-setup.sh` and `distrib-setup.sh` files are mutually exclusive in 
execution.
i. `distrib-setup.sh` -> if it has executable content. Else 
`auto-setup.sh`
The assumption here is that the distribution might have a 
different mechanism for doing things (along lines of specifying a preferred 
setup service).
ii. `drill-setup.sh` --> This is any additional setup activities beyond 
what the distribution offers. I don't think this should be mutually exclusive 
to the previous step, but should be an add on.

A distribution of Drill should reimplement the setup task done by 
`auto-setup.sh` , if it wants to do the same separately.

2. For managing memory automatically, the `auto-setup.sh` script is looking 
for the `DRILLBIT_MAX_PROC_MEM` variable.
If this is not set (which is the case by default), the existing procedure 
for inferring memory parameters will be followed.

3. If the `DRILLBIT_MAX_PROC_MEM` variable is set, the logic defined within 
the `auto-setup.sh` script is executed to assign and validate that the total 
memory allocation of Heap, Direct and CodeCache is within that value.

4. The JIRA and the PR has been updated with the details of the feature.



---

[GitHub] drill pull request #570: DRILL-4834 decimal implementation is vulnerable to ...

[daveoshinsky]

Github user daveoshinsky commented on a diff in the pull request:

https://github.com/apache/drill/pull/570#discussion_r161630462
  
--- Diff: exec/java-exec/src/main/codegen/templates/SqlAccessors.java ---
@@ -127,6 +127,25 @@ public String getString(int index) {
 }
   <#break>
 
+<#case "VarDecimal">
+
+@Override
+public String getString(int index){
+<#if mode=="Nullable">
+if(ac.isNull(index)){
+  return null;
+}
+
+try {
--- End diff --

When I remove the try/catch here, the build fails as follows.  So, I'm 
leaving it as is for the time being.
[ERROR] 
C:\apache\Drill20180111\rebase6\drill\exec\java-exec\target\generated-sources\org\apache\drill\exec\vector\accessor\VarDecimalAccessor.java:[144,35]
 error: unreported exception InvalidAccessException; must be caught or declared 
to be thrown



---

[jira] [Resolved] (DRILL-6083) RestClientFixture does not connect to the correct webserver port

[Karthikeyan Manivannan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karthikeyan Manivannan resolved DRILL-6083.
---
Resolution: Not A Problem

> RestClientFixture does not connect to the correct webserver port
> 
>
> Key: DRILL-6083
> URL: https://issues.apache.org/jira/browse/DRILL-6083
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Tools, Build  Test
>Affects Versions: Future
>Reporter: Karthikeyan Manivannan
>Assignee: Karthikeyan Manivannan
>Priority: Major
> Fix For: 1.13.0
>
>
> RestClientFixture always connects to the default http port (8047) instead of 
> connecting to the webserver-port of the cluster. The cluster's webserver port 
> won't be 8047 if there are other Drillbits running when the cluster is 
> launched.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (DRILL-6089) Planner changes needed as Hash-Join spill breaks left-side order

[Timothy Farkas (JIRA)]

Timothy Farkas created DRILL-6089:
-

 Summary: Planner changes needed as Hash-Join spill breaks 
left-side order
 Key: DRILL-6089
 URL: https://issues.apache.org/jira/browse/DRILL-6089
 Project: Apache Drill
  Issue Type: Improvement
 Environment: Explanation provided by Boaz:



(As explained in the design document) The new "automatic spill" feature of the 
Hash-Join operator may cause (if spilling occurs) the rows from the left/probe 
side to be returned in a different order than their incoming order (due to 
splitting the rows into partitions).

Currently the Drill planner assumes that left-order is preserved by the 
Hash-Join operator; therefore if not changes, a query relying on that order may 
return wrong results (when the Hash-Join spills).

A fix is needed. Here are few options (ordered from the simpler down to the 
most complex):

# Change the order rule in the planner. Thus whenever an order is needed above 
(downstream) the Hash-Join, the planner would add a sort operator. That would 
be a big execution time waste.
# When the planner needs the left-order above the Hash-Join, it may assess the 
size of the right/build side (need statistics). If the right side is small 
enough, the planner would set an option for the runtime to avoid spilling, 
hence preserving the left-side order. In case spilling becomes necessary, the 
code would return an error (possibly with a message suggesting setting some 
special option and retrying; the special option would add a sort operator and 
allow the hash-join to spill).
# When generating the code for the fragment above the Hash-Join (where 
left-order should be maintained) - at code-gen time check if the hash-join 
below spilled, and if so, add a sort operator. (Nothing like that exists in 
Drill now, so it may be complicated).
Reporter: Timothy Farkas
Assignee: Timothy Farkas






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] drill pull request #1066: DRILL-3993: Changes to support Calcite 1.15

[chunhui-shi]

Github user chunhui-shi commented on a diff in the pull request:

https://github.com/apache/drill/pull/1066#discussion_r161607926
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggTemplate.java
 ---
@@ -1303,6 +1305,8 @@ private void checkGroupAndAggrValues(int 
incomingRowIdx) {
   long memDiff = allocator.getAllocatedMemory() - allocatedBeforeHTput;
   if ( memDiff > 0 ) { logger.warn("Leak: HashTable put() OOM left 
behind {} bytes allocated",memDiff); }
 
+  checkForSpillPossibility(currentPartition);
--- End diff --

Not sure this check 'chooseAPartitionToFlush'  is needed. If an exception 
is desired, I would think modifying doSpill() is better way e.g. modifying this 
line: "  if ( victimPartition < 0 ) { return; } " Otherwise in this process 
chooseAPartitionToFlush will be called twice.

  int victimPartition = chooseAPartitionToFlush(currentPartition, 
forceSpill);

  // In case no partition has more than one batch -- try and "push the 
limits"; maybe next
  // time the spill could work.
  if ( victimPartition < 0 ) { return; } 


---

Re: Odd Error on Login with 1.13-SNAPSHOT

[Sorabh Hamirwasia]

I have opened a PR for review. Would 
be great if you can pull that in and try out, unfortunately just adding the new 
config will not help 


Thanks,
Sorabh



From: Sorabh Hamirwasia
Sent: Sunday, January 14, 2018 9:36 PM
To: dev@drill.apache.org
Subject: Re: Odd Error on Login with 1.13-SNAPSHOT


I have opened DRILL-6088 for 
this issue and will provide a patch by tonight for review. For now to unblock 
you, please add following http configuration in your drill-override.conf.


drill.exec.http.auth.mechanisms: ["FORM"],


I am assuming other user auth related setting to enable PLAIN authentication is 
in place as per [1] or [2].


[1]: https://drill.apache.org/docs/using-libpam4j-as-the-pam-authenticator/

[2]: https://drill.apache.org/docs/using-jpam-as-the-pam-authenticator/


Thanks,
Sorabh


From: Vitalii Diravka 
Sent: Friday, January 12, 2018 2:12 PM
To: dev@drill.apache.org
Subject: Re: Odd Error on Login with 1.13-SNAPSHOT

Hi John,

Probably this Jira can help you [1].
You should verify your drill-override.conf file. Do you want to use any
security mechanism?

[1] 
https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_DRILL-2D5425=DwIBaQ=cskdkSMqhcnjZxdQVpwTXg=gRpEl0WzXE3EMrwj0KFbZXGXRyadOthF2jlYxvhTlQg=eWbxbVf7ysp5Kkl8FpO63Em5a65AnSOP5kYU4KIwQTo=WfamtfbUdrG9eS2v6VLz0FI80E-2a_YjfrjkuztQzME=

Kind regards
Vitalii

On Thu, Jan 11, 2018 at 6:52 PM, John Omernik  wrote:

> I am probably missing something minor here, but I am working with Ted
> Dunning on some PCAP plugin stuff, so I built his 1.13 SNAPSHOT, and when I
> try to login I see
>
> {
>   "errorMessage" : "No configuration setting found for key
> 'drill.exec.http.auth'"
> }
>
>
>
> I am guessing that something was added that I need to fill out in my
> config? Is there a JIRA or something that can guide me on this?
>
>
> Thanks
>

[GitHub] drill pull request #1059: DRILL-5851: Empty table during a join operation wi...

[vdiravka]

Github user vdiravka commented on a diff in the pull request:

https://github.com/apache/drill/pull/1059#discussion_r161510606
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/physical/unit/TestNullInputMiniPlan.java
 ---
@@ -354,9 +354,9 @@ public void testHashJoinRightEmpty() throws Exception {
 .build();
 
 BatchSchema expectedSchema = new SchemaBuilder()
-.addNullable("a", TypeProtos.MinorType.BIGINT)
-.addNullable("b", TypeProtos.MinorType.BIGINT)
-.withSVMode(BatchSchema.SelectionVectorMode.NONE)
+.addNullable("a", TypeProtos.MinorType.BIGINT)
+.addNullable("b", TypeProtos.MinorType.BIGINT)
+.withSVMode(BatchSchema.SelectionVectorMode.NONE)
--- End diff --

Please, return original indentation.


---

[GitHub] drill pull request #1059: DRILL-5851: Empty table during a join operation wi...

[vdiravka]

Github user vdiravka commented on a diff in the pull request:

https://github.com/apache/drill/pull/1059#discussion_r161510835
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/join/TestNestedLoopJoin.java
 ---
@@ -333,4 +339,55 @@ public void testNlJoinWithLargeRightInputSuccess() 
throws Exception {
   test(RESET_JOIN_OPTIMIZATION);
 }
   }
+
+  private static void buildFile(String fileName, String[] data, File 
testDir) throws IOException {
+try(PrintWriter out = new PrintWriter(new FileWriter(new File(testDir, 
fileName {
+  for (String line : data) {
+out.println(line);
+  }
+}
+  }
+
+  public static void testWithEmptyJoin(File testDir, String joinType,
--- End diff --

Please, add java doc for new helper methods.


---

[GitHub] drill pull request #1059: DRILL-5851: Empty table during a join operation wi...

[vdiravka]

Github user vdiravka commented on a diff in the pull request:

https://github.com/apache/drill/pull/1059#discussion_r161512862
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/join/TestHashJoinAdvanced.java
 ---
@@ -160,4 +165,19 @@ public void testJoinWithMapAndDotField() throws 
Exception {
   .baselineValues("1", "2", "1", null, "a")
   .go();
   }
+
+  @Test
+  public void testHashLeftJoinWithEmptyTable() throws Exception {
+TestNestedLoopJoin.testWithEmptyJoin(dirTestWatcher.getRootDir(), 
"left outer", hjpattern, 1155L);
--- End diff --

It is not so good to use helper methods from `TestNestedLoopJoin` in 
`TestHashJoinAdvanced` class. 
Could you factor out the common part of code (helper methods and constants) 
into the Base class, for example `JoinTestBase`?


---