[jira] [Commented] (DRILL-5043) Function that returns a unique id per session/connection similar to MySQL's CONNECTION_ID()

[Nagarajan Chinnasamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15734465#comment-15734465
 ] 

Nagarajan Chinnasamy commented on DRILL-5043:
-

While this solution seems to work in an embedded installation, I have doubt if 
it will work in a distributed installation --- as the SessionId is locally 
generated inside a JVM. We may have to look for zookeeper based solution to get 
a unique SessionId across the DrillBits. Need your guidelines as how to 
approach this problem.

> Function that returns a unique id per session/connection similar to MySQL's 
> CONNECTION_ID()
> ---
>
> Key: DRILL-5043
> URL: https://issues.apache.org/jira/browse/DRILL-5043
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Functions - Drill
>Affects Versions: 1.8.0
>Reporter: Nagarajan Chinnasamy
>Priority: Minor
>  Labels: CONNECTION_ID, SESSION, UDF
> Attachments: 01_session_id_sqlline.png, 
> 02_session_id_webconsole_query.png, 03_session_id_webconsole_result.png
>
>
> Design and implement a function that returns a unique id per 
> session/connection similar to MySQL's CONNECTION_ID().



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5052) Option to debug generated Java code using an IDE

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15734285#comment-15734285
 ] 

ASF GitHub Bot commented on DRILL-5052:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/660#discussion_r91641492
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/ClassGenerator.java ---
@@ -411,6 +417,10 @@ public boolean equals(Object obj) {
 
   }
 
+  /**
+   * Represents a (Nullable)?(Type)Holder instance.
+   */
+
   public static class HoldingContainer{
 private final JVar holder;
--- End diff --

Legacy code, but fixed.


> Option to debug generated Java code using an IDE
> 
>
> Key: DRILL-5052
> URL: https://issues.apache.org/jira/browse/DRILL-5052
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Codegen
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> Drill makes extensive use of Java code generation to implement its operators. 
> Drill uses sophisticated techniques to blend generated code with pre-compiled 
> template code. An unfortunate side-effect of this behavior is that it is very 
> difficult to visualize and debug the generated code.
> As it turns out, Drill's code-merge facility is, in essence, a do-it-yourself 
> version of subclassing. The Drill "template" is the parent class, the 
> generated code is the subclass. But, rather than using plain-old subclassing, 
> Drill combines the code from the two classes into a single "artificial" 
> packet of byte codes for which no source exists.
> Modify the code generation path to optionally allow "plain-old Java" 
> compilation: the generated code is a subclass of the template. Compile the 
> generated code as a plain-old Java class with no byte-code fix-up. Write the 
> code to a known location that the IDE can search when looking for source 
> files.
> With this change, developers can turn on the above feature, set a breakpoint 
> in a template, then step directly into the generated Java code called from 
> the template.
> This feature should be an option, enabled by developers when needed. The 
> existing byte-code technique should be used for production code generation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5052) Option to debug generated Java code using an IDE

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15734265#comment-15734265
 ] 

ASF GitHub Bot commented on DRILL-5052:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/660#discussion_r91633167
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/compile/ClassBuilder.java ---
@@ -0,0 +1,190 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.compile;
+
+import java.io.File;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.util.Map;
+
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.exec.compile.ClassTransformer.ClassNames;
+import org.apache.drill.exec.exception.ClassTransformationException;
+import org.apache.drill.exec.expr.CodeGenerator;
+import org.apache.drill.exec.server.options.OptionManager;
+import org.codehaus.commons.compiler.CompileException;
+
+/**
+ * Implements the "plain-old Java" method of code generation and
+ * compilation. Given a {@link CodeGenerator}, obtains the generated
+ * source code, compiles it with the selected compiler, loads the
+ * byte-codes into a class loader and provides the resulting
+ * class. Compared with the {@link ClassTransformer} mechanism,
+ * this one requires the code generator to have generated a complete
+ * Java class that is capable of direct compilation and loading.
+ * This means the generated class must be a subclass of the template
+ * so that the JVM can use normal Java inheritance to associate the
+ * template and generated methods.
+ * 
+ * Here is how to use the plain-old Java technique to debug
+ * generated code:
+ * 
+ * Set the config option drill.exec.compile.save_source
+ * to true.
+ * Set the config option drill.exec.compile.code_dir
+ * to the location where you want to save the generated source
+ * code.
+ * Where you generate code (using a {@link CodeGenerator}),
+ * set the "plain-old Java" options:
+ * CodeGeneratorFoo> cg = ...
+ * cg.plainOldJavaCapable(true); // Class supports plain-old Java
+ * cg.preferPlainOldJava(true); // Actually generate plain-old Java
+ * ...
+ * In your favorite IDE, add to the code lookup path the
+ * code directory saved earlier. In Eclipse, for example, you do
+ * this in the debug configuration you will use to debug Drill.
+ * Set a breakpoint in template used for the generated code.
+ * Run Drill. The IDE will stop at your breakpoint.
+ * Step into the generated code. Examine class field and
+ * local variables. Have fun!
+ * 
+ * 
+ * Note: not all generated code is ready to be compiled as plain-old
+ * Java. Some classes omit from the template the proper throws
+ * declarations. Other minor problems may also crop up. All are easy
+ * to fix. Once you've done so, add the following to mark that you've
+ * done the clean-up:
--- End diff --

Done. DRILL-5116.


> Option to debug generated Java code using an IDE
> 
>
> Key: DRILL-5052
> URL: https://issues.apache.org/jira/browse/DRILL-5052
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Codegen
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> Drill makes extensive use of Java code generation to implement its operators. 
> Drill uses sophisticated techniques to blend generated code with pre-compiled 
> template code. An unfortunate side-effect of this behavior is that it is very 
> difficult to visualize and debug the generated code.
> As it turns out, Drill's code-merge facility is, in essence, a do-it-yourself 
> version of subclassing. The Drill "template" is the parent class, the 
> generated code is the subclass. 

[jira] [Commented] (DRILL-5052) Option to debug generated Java code using an IDE

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15734271#comment-15734271
 ] 

ASF GitHub Bot commented on DRILL-5052:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/660#discussion_r91638203
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/compile/DrillJavaFileObject.java
 ---
@@ -38,14 +39,18 @@
 
   private Map outputFiles;
 
+  private String className;
--- End diff --

Done.


> Option to debug generated Java code using an IDE
> 
>
> Key: DRILL-5052
> URL: https://issues.apache.org/jira/browse/DRILL-5052
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Codegen
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> Drill makes extensive use of Java code generation to implement its operators. 
> Drill uses sophisticated techniques to blend generated code with pre-compiled 
> template code. An unfortunate side-effect of this behavior is that it is very 
> difficult to visualize and debug the generated code.
> As it turns out, Drill's code-merge facility is, in essence, a do-it-yourself 
> version of subclassing. The Drill "template" is the parent class, the 
> generated code is the subclass. But, rather than using plain-old subclassing, 
> Drill combines the code from the two classes into a single "artificial" 
> packet of byte codes for which no source exists.
> Modify the code generation path to optionally allow "plain-old Java" 
> compilation: the generated code is a subclass of the template. Compile the 
> generated code as a plain-old Java class with no byte-code fix-up. Write the 
> code to a known location that the IDE can search when looking for source 
> files.
> With this change, developers can turn on the above feature, set a breakpoint 
> in a template, then step directly into the generated Java code called from 
> the template.
> This feature should be an option, enabled by developers when needed. The 
> existing byte-code technique should be used for production code generation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5052) Option to debug generated Java code using an IDE

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15734264#comment-15734264
 ] 

ASF GitHub Bot commented on DRILL-5052:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/660#discussion_r91631857
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/compile/CachedClassLoader.java
 ---
@@ -0,0 +1,70 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.compile;
+
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.Map;
+import java.util.concurrent.ConcurrentMap;
+
+import com.google.common.collect.MapMaker;
+
+/**
+ * Class loader for "plain-old Java" generated classes.
+ * Very basic implementation: allows defining a class from
+ * byte codes and finding the loaded classes. Delegates
+ * all other class requests to the thread context class
+ * loader. This structure ensures that a generated class can
+ * find both its own inner classes as well as all the standard
+ * Drill implementation classes.
+ */
+
+public class CachedClassLoader extends URLClassLoader {
+
+  /**
+   * Cache of generated classes. Semantics: a single thread defines
+   * the classes, many threads may access the classes.
+   */
+
+  private ConcurrentMap cache = new 
MapMaker().concurrencyLevel(4).makeMap();
--- End diff --

Done.


> Option to debug generated Java code using an IDE
> 
>
> Key: DRILL-5052
> URL: https://issues.apache.org/jira/browse/DRILL-5052
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Codegen
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> Drill makes extensive use of Java code generation to implement its operators. 
> Drill uses sophisticated techniques to blend generated code with pre-compiled 
> template code. An unfortunate side-effect of this behavior is that it is very 
> difficult to visualize and debug the generated code.
> As it turns out, Drill's code-merge facility is, in essence, a do-it-yourself 
> version of subclassing. The Drill "template" is the parent class, the 
> generated code is the subclass. But, rather than using plain-old subclassing, 
> Drill combines the code from the two classes into a single "artificial" 
> packet of byte codes for which no source exists.
> Modify the code generation path to optionally allow "plain-old Java" 
> compilation: the generated code is a subclass of the template. Compile the 
> generated code as a plain-old Java class with no byte-code fix-up. Write the 
> code to a known location that the IDE can search when looking for source 
> files.
> With this change, developers can turn on the above feature, set a breakpoint 
> in a template, then step directly into the generated Java code called from 
> the template.
> This feature should be an option, enabled by developers when needed. The 
> existing byte-code technique should be used for production code generation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5052) Option to debug generated Java code using an IDE

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15734281#comment-15734281
 ] 

ASF GitHub Bot commented on DRILL-5052:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/660#discussion_r91642538
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/xsort/ExternalSortBatch.java
 ---
@@ -711,18 +711,20 @@ private MSorter createNewMSorter(FragmentContext 
context, List orderin
 g.rotateBlock();
 g.getEvalBlock()._return(JExpr.lit(0));
 
+cg.plainOldJavaCapable(true); // This class can generate plain-old 
Java.
+ // Uncomment out this line to debug the generated code.
+//  cg.preferPlainOldJavaJava(true);
--- End diff --

Fixed.


> Option to debug generated Java code using an IDE
> 
>
> Key: DRILL-5052
> URL: https://issues.apache.org/jira/browse/DRILL-5052
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Codegen
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> Drill makes extensive use of Java code generation to implement its operators. 
> Drill uses sophisticated techniques to blend generated code with pre-compiled 
> template code. An unfortunate side-effect of this behavior is that it is very 
> difficult to visualize and debug the generated code.
> As it turns out, Drill's code-merge facility is, in essence, a do-it-yourself 
> version of subclassing. The Drill "template" is the parent class, the 
> generated code is the subclass. But, rather than using plain-old subclassing, 
> Drill combines the code from the two classes into a single "artificial" 
> packet of byte codes for which no source exists.
> Modify the code generation path to optionally allow "plain-old Java" 
> compilation: the generated code is a subclass of the template. Compile the 
> generated code as a plain-old Java class with no byte-code fix-up. Write the 
> code to a known location that the IDE can search when looking for source 
> files.
> With this change, developers can turn on the above feature, set a breakpoint 
> in a template, then step directly into the generated Java code called from 
> the template.
> This feature should be an option, enabled by developers when needed. The 
> existing byte-code technique should be used for production code generation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5052) Option to debug generated Java code using an IDE

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15734283#comment-15734283
 ] 

ASF GitHub Bot commented on DRILL-5052:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/660#discussion_r91642120
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/DebugStringBuilder.java 
---
@@ -0,0 +1,57 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.expr;
+
+import java.io.PrintWriter;
+import java.io.StringWriter;
+
+import com.sun.codemodel.JFormatter;
+
+/**
+ * Utility class to build a debug string for an object
+ * in a standard format.
+ */
+
+public class DebugStringBuilder {
+
+  private StringWriter strWriter;
+  public PrintWriter writer;
+  public JFormatter fmt;
+
--- End diff --

Done. You are strict, this is just debug code...


> Option to debug generated Java code using an IDE
> 
>
> Key: DRILL-5052
> URL: https://issues.apache.org/jira/browse/DRILL-5052
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Codegen
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> Drill makes extensive use of Java code generation to implement its operators. 
> Drill uses sophisticated techniques to blend generated code with pre-compiled 
> template code. An unfortunate side-effect of this behavior is that it is very 
> difficult to visualize and debug the generated code.
> As it turns out, Drill's code-merge facility is, in essence, a do-it-yourself 
> version of subclassing. The Drill "template" is the parent class, the 
> generated code is the subclass. But, rather than using plain-old subclassing, 
> Drill combines the code from the two classes into a single "artificial" 
> packet of byte codes for which no source exists.
> Modify the code generation path to optionally allow "plain-old Java" 
> compilation: the generated code is a subclass of the template. Compile the 
> generated code as a plain-old Java class with no byte-code fix-up. Write the 
> code to a known location that the IDE can search when looking for source 
> files.
> With this change, developers can turn on the above feature, set a breakpoint 
> in a template, then step directly into the generated Java code called from 
> the template.
> This feature should be an option, enabled by developers when needed. The 
> existing byte-code technique should be used for production code generation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5052) Option to debug generated Java code using an IDE

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15734278#comment-15734278
 ] 

ASF GitHub Bot commented on DRILL-5052:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/660#discussion_r91641345
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/ClassGenerator.java ---
@@ -246,6 +246,12 @@ public void rotateBlock() {
 rotateBlock(BlkCreateMode.TRUE);
   }
 
+  /**
+   * Create a new code block, closing the current block.
+   *
+   * @param mode
+   */
--- End diff --

Oh? Do you use IntelliJ? Eclipse generates these items, but does not 
complain about them. Since I've been cleaning up the yellow warnings that 
appear in Eclipse, I should not create new ones for IntelliJ... Or if Eclipse, 
what setting did you turn on to get the warnings?


> Option to debug generated Java code using an IDE
> 
>
> Key: DRILL-5052
> URL: https://issues.apache.org/jira/browse/DRILL-5052
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Codegen
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> Drill makes extensive use of Java code generation to implement its operators. 
> Drill uses sophisticated techniques to blend generated code with pre-compiled 
> template code. An unfortunate side-effect of this behavior is that it is very 
> difficult to visualize and debug the generated code.
> As it turns out, Drill's code-merge facility is, in essence, a do-it-yourself 
> version of subclassing. The Drill "template" is the parent class, the 
> generated code is the subclass. But, rather than using plain-old subclassing, 
> Drill combines the code from the two classes into a single "artificial" 
> packet of byte codes for which no source exists.
> Modify the code generation path to optionally allow "plain-old Java" 
> compilation: the generated code is a subclass of the template. Compile the 
> generated code as a plain-old Java class with no byte-code fix-up. Write the 
> code to a known location that the IDE can search when looking for source 
> files.
> With this change, developers can turn on the above feature, set a breakpoint 
> in a template, then step directly into the generated Java code called from 
> the template.
> This feature should be an option, enabled by developers when needed. The 
> existing byte-code technique should be used for production code generation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5052) Option to debug generated Java code using an IDE

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15734284#comment-15734284
 ] 

ASF GitHub Bot commented on DRILL-5052:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/660#discussion_r91642556
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/xsort/ExternalSortBatch.java
 ---
@@ -711,18 +711,20 @@ private MSorter createNewMSorter(FragmentContext 
context, List orderin
 g.rotateBlock();
 g.getEvalBlock()._return(JExpr.lit(0));
 
+cg.plainOldJavaCapable(true); // This class can generate plain-old 
Java.
+ // Uncomment out this line to debug the generated code.
+//  cg.preferPlainOldJavaJava(true);
 return context.getImplementationClass(cg);
-
-
   }
 
   public SingleBatchSorter createNewSorter(FragmentContext context, 
VectorAccessible batch)
   throws ClassTransformationException, IOException, 
SchemaChangeException{
 CodeGenerator cg = 
CodeGenerator.get(SingleBatchSorter.TEMPLATE_DEFINITION, 
context.getFunctionRegistry(), context.getOptions());
-ClassGenerator g = cg.getRoot();
-
-generateComparisons(g, batch);
+cg.plainOldJavaCapable(true); // This class can generate plain-old 
Java.
 
+// Uncomment out this line to debug the generated code.
+//cg.preferPlainOldJavaJava(true);
--- End diff --

Fixed.


> Option to debug generated Java code using an IDE
> 
>
> Key: DRILL-5052
> URL: https://issues.apache.org/jira/browse/DRILL-5052
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Codegen
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> Drill makes extensive use of Java code generation to implement its operators. 
> Drill uses sophisticated techniques to blend generated code with pre-compiled 
> template code. An unfortunate side-effect of this behavior is that it is very 
> difficult to visualize and debug the generated code.
> As it turns out, Drill's code-merge facility is, in essence, a do-it-yourself 
> version of subclassing. The Drill "template" is the parent class, the 
> generated code is the subclass. But, rather than using plain-old subclassing, 
> Drill combines the code from the two classes into a single "artificial" 
> packet of byte codes for which no source exists.
> Modify the code generation path to optionally allow "plain-old Java" 
> compilation: the generated code is a subclass of the template. Compile the 
> generated code as a plain-old Java class with no byte-code fix-up. Write the 
> code to a known location that the IDE can search when looking for source 
> files.
> With this change, developers can turn on the above feature, set a breakpoint 
> in a template, then step directly into the generated Java code called from 
> the template.
> This feature should be an option, enabled by developers when needed. The 
> existing byte-code technique should be used for production code generation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5052) Option to debug generated Java code using an IDE

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15734279#comment-15734279
 ] 

ASF GitHub Bot commented on DRILL-5052:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/660#discussion_r91641555
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/CodeGenerator.java ---
@@ -67,26 +96,71 @@
 try {
   this.model = new JCodeModel();
   JDefinedClass clazz = model._package(PACKAGE_NAME)._class(className);
+  if ( isPlainOldJava( ) ) {
+clazz._extends(definition.getTemplateClass( ) );
+  }
   rootGenerator = new ClassGenerator<>(this, mappingSet, 
definition.getSignature(), new EvaluationVisitor(
   funcRegistry), clazz, model, optionManager);
 } catch (JClassAlreadyExistsException e) {
   throw new IllegalStateException(e);
 }
   }
 
+  /**
+   * Indicates that the code for this class can be generated using the
+   * "Plain Old Java" mechanism based on inheritance. The byte-code
+   * method is more lenient, so some code is missing some features such
+   * as proper exception labeling, etc. Set this option to true once
+   * the generation mechanism for a class has been cleaned up to work
+   * via the plain-old Java mechanism.
+   *
+   * @param flag true if the code generated from this instance is
+   * ready to be compiled as a plain-old Java class
+   */
+
+  public void plainOldJavaCapable(boolean flag) {
+plainOldJavaCapable = flag;
+  }
+
+  /**
+   * Identifies that this generated class should be generated via the
+   * plain-old Java mechanism. This flag only has meaning if the
+   * generated class is capable of plain-old Java generation.
+   *
+   * @param flag true if the class should be generated and compiled
+   * as a plain-old Java class (rather than via byte-code manipulations)
+   */
+
+  public void preferPlainOJava(boolean flag) {
+usePlainOldJava = flag;
--- End diff --

Yeah... typo. Fixed.


> Option to debug generated Java code using an IDE
> 
>
> Key: DRILL-5052
> URL: https://issues.apache.org/jira/browse/DRILL-5052
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Codegen
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> Drill makes extensive use of Java code generation to implement its operators. 
> Drill uses sophisticated techniques to blend generated code with pre-compiled 
> template code. An unfortunate side-effect of this behavior is that it is very 
> difficult to visualize and debug the generated code.
> As it turns out, Drill's code-merge facility is, in essence, a do-it-yourself 
> version of subclassing. The Drill "template" is the parent class, the 
> generated code is the subclass. But, rather than using plain-old subclassing, 
> Drill combines the code from the two classes into a single "artificial" 
> packet of byte codes for which no source exists.
> Modify the code generation path to optionally allow "plain-old Java" 
> compilation: the generated code is a subclass of the template. Compile the 
> generated code as a plain-old Java class with no byte-code fix-up. Write the 
> code to a known location that the IDE can search when looking for source 
> files.
> With this change, developers can turn on the above feature, set a breakpoint 
> in a template, then step directly into the generated Java code called from 
> the template.
> This feature should be an option, enabled by developers when needed. The 
> existing byte-code technique should be used for production code generation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5052) Option to debug generated Java code using an IDE

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15734275#comment-15734275
 ] 

ASF GitHub Bot commented on DRILL-5052:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/660#discussion_r91640219
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/compile/MergeAdapter.java ---
@@ -253,7 +257,13 @@ public static MergedClassResult getMergedClass(final 
ClassSet set, final byte[]
   }
 
   // enable when you want all the generated merged class files to also 
be written to disk.
-//  Files.write(outputClass, new 
File(String.format("/src/scratch/drill-generated-classes/%s-output.class", 
set.generated.dot)));
+//  try {
--- End diff --

Bulk writing of class source code would not likely be useful. Just left 
this in place from the previous code, but fixed it so it would work if 
uncommented. Actually, once this PR is in, the need to do this trick will 
disappear and we can simply remove this code.


> Option to debug generated Java code using an IDE
> 
>
> Key: DRILL-5052
> URL: https://issues.apache.org/jira/browse/DRILL-5052
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Codegen
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> Drill makes extensive use of Java code generation to implement its operators. 
> Drill uses sophisticated techniques to blend generated code with pre-compiled 
> template code. An unfortunate side-effect of this behavior is that it is very 
> difficult to visualize and debug the generated code.
> As it turns out, Drill's code-merge facility is, in essence, a do-it-yourself 
> version of subclassing. The Drill "template" is the parent class, the 
> generated code is the subclass. But, rather than using plain-old subclassing, 
> Drill combines the code from the two classes into a single "artificial" 
> packet of byte codes for which no source exists.
> Modify the code generation path to optionally allow "plain-old Java" 
> compilation: the generated code is a subclass of the template. Compile the 
> generated code as a plain-old Java class with no byte-code fix-up. Write the 
> code to a known location that the IDE can search when looking for source 
> files.
> With this change, developers can turn on the above feature, set a breakpoint 
> in a template, then step directly into the generated Java code called from 
> the template.
> This feature should be an option, enabled by developers when needed. The 
> existing byte-code technique should be used for production code generation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5052) Option to debug generated Java code using an IDE

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15734276#comment-15734276
 ] 

ASF GitHub Bot commented on DRILL-5052:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/660#discussion_r91640094
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/compile/JaninoClassCompiler.java
 ---
@@ -58,5 +56,27 @@ public JaninoClassCompiler(ClassLoader 
parentClassLoader, boolean debug) {
   }
 
   @Override
+  public Map compile(final ClassNames className, final 
String sourceCode)
+  throws CompileException, IOException, ClassNotFoundException {
+
+ClassFile[] classFiles = doCompile( className, sourceCode );
+Map results = new HashMap<>( );
+for(int i = 0;  i < classFiles.length;  i++) {
+  ClassFile classFile = classFiles[i];
+  results.put( classFile.getThisClassName(), classFile.toByteArray() );
+}
+return results;
+  }
+
+  public ClassFile[] doCompile(final ClassNames className, final String 
sourceCode)
--- End diff --

Done.


> Option to debug generated Java code using an IDE
> 
>
> Key: DRILL-5052
> URL: https://issues.apache.org/jira/browse/DRILL-5052
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Codegen
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> Drill makes extensive use of Java code generation to implement its operators. 
> Drill uses sophisticated techniques to blend generated code with pre-compiled 
> template code. An unfortunate side-effect of this behavior is that it is very 
> difficult to visualize and debug the generated code.
> As it turns out, Drill's code-merge facility is, in essence, a do-it-yourself 
> version of subclassing. The Drill "template" is the parent class, the 
> generated code is the subclass. But, rather than using plain-old subclassing, 
> Drill combines the code from the two classes into a single "artificial" 
> packet of byte codes for which no source exists.
> Modify the code generation path to optionally allow "plain-old Java" 
> compilation: the generated code is a subclass of the template. Compile the 
> generated code as a plain-old Java class with no byte-code fix-up. Write the 
> code to a known location that the IDE can search when looking for source 
> files.
> With this change, developers can turn on the above feature, set a breakpoint 
> in a template, then step directly into the generated Java code called from 
> the template.
> This feature should be an option, enabled by developers when needed. The 
> existing byte-code technique should be used for production code generation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5052) Option to debug generated Java code using an IDE

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15734263#comment-15734263
 ] 

ASF GitHub Bot commented on DRILL-5052:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/660#discussion_r91638845
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/compile/JaninoClassCompiler.java
 ---
@@ -58,5 +56,27 @@ public JaninoClassCompiler(ClassLoader 
parentClassLoader, boolean debug) {
   }
 
   @Override
+  public Map compile(final ClassNames className, final 
String sourceCode)
+  throws CompileException, IOException, ClassNotFoundException {
+
+ClassFile[] classFiles = doCompile( className, sourceCode );
+Map results = new HashMap<>( );
--- End diff --

See earlier note.


> Option to debug generated Java code using an IDE
> 
>
> Key: DRILL-5052
> URL: https://issues.apache.org/jira/browse/DRILL-5052
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Codegen
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> Drill makes extensive use of Java code generation to implement its operators. 
> Drill uses sophisticated techniques to blend generated code with pre-compiled 
> template code. An unfortunate side-effect of this behavior is that it is very 
> difficult to visualize and debug the generated code.
> As it turns out, Drill's code-merge facility is, in essence, a do-it-yourself 
> version of subclassing. The Drill "template" is the parent class, the 
> generated code is the subclass. But, rather than using plain-old subclassing, 
> Drill combines the code from the two classes into a single "artificial" 
> packet of byte codes for which no source exists.
> Modify the code generation path to optionally allow "plain-old Java" 
> compilation: the generated code is a subclass of the template. Compile the 
> generated code as a plain-old Java class with no byte-code fix-up. Write the 
> code to a known location that the IDE can search when looking for source 
> files.
> With this change, developers can turn on the above feature, set a breakpoint 
> in a template, then step directly into the generated Java code called from 
> the template.
> This feature should be an option, enabled by developers when needed. The 
> existing byte-code technique should be used for production code generation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5052) Option to debug generated Java code using an IDE

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15734269#comment-15734269
 ] 

ASF GitHub Bot commented on DRILL-5052:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/660#discussion_r91634477
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/compile/ClassCompilerSelector.java
 ---
@@ -0,0 +1,146 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.compile;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Map;
+
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.compile.ClassTransformer.ClassNames;
+import org.apache.drill.exec.exception.ClassTransformationException;
+import org.apache.drill.exec.server.options.OptionManager;
+import org.apache.drill.exec.server.options.OptionValidator;
+import org.apache.drill.exec.server.options.OptionValue;
+import 
org.apache.drill.exec.server.options.TypeValidators.BooleanValidator;
+import org.apache.drill.exec.server.options.TypeValidators.LongValidator;
+import org.apache.drill.exec.server.options.TypeValidators.StringValidator;
+import org.codehaus.commons.compiler.CompileException;
+
+/**
+ * Selects between the two supported Java compilers: Janino and
+ * the build-in Java compiler.
+ *
+ * Session Options
+ * 
+ * exec.java_compiler
+ * The compiler to use. Valid options are defined in the
+ * {@link ClassCompilerSelector.CompilerPolicy} enum.
+ * exec.java_compiler_debug
+ * If debug logging is enabled, then {@link AbstractClassCompiler} 
writes the
+ * generated Java code to the log file prior to compilation. This option
+ * adds line numbers to the logged code.
+ * exec.java_compiler_janino_maxsize
+ * The maximum size of code that the Janio compiler can handle. Larger 
code is
+ * handled by the JDK compiler. Defaults to 256K.
+ * 
+ * Configuration Options
+ * Configuration options are used when the above session options are unset.
+ * 
+ * drill.exec.compile.compiler
+ * Default for exec.java_compiler
+ * drill.exec.compile.debug
+ * Default for exec.java_compiler_debug
+ * drill.exec.compile.janino_maxsize
+ * Default for exec.java_compiler_janino_maxsize
+ * 
+ */
+
+public class ClassCompilerSelector {
+  public enum CompilerPolicy {
+DEFAULT, JDK, JANINO;
+  }
+
+  public static final String JAVA_COMPILER_JANINO_MAXSIZE_CONFIG = 
CodeCompiler.COMPILE_BASE + ".janino_maxsize";
+  public static final String JAVA_COMPILER_DEBUG_CONFIG = 
CodeCompiler.COMPILE_BASE + ".debug";
+  public static final String JAVA_COMPILER_CONFIG = 
CodeCompiler.COMPILE_BASE + ".compiler";
+
+  public static final String JAVA_COMPILER_OPTION = "exec.java_compiler";
+  public static final String JAVA_COMPILER_JANINO_MAXSIZE_OPTION = 
"exec.java_compiler_janino_maxsize";
+  public static final OptionValidator JAVA_COMPILER_JANINO_MAXSIZE = new 
LongValidator(JAVA_COMPILER_JANINO_MAXSIZE_OPTION, 256*1024);
+
+  public static final String JAVA_COMPILER_DEBUG_OPTION = 
"exec.java_compiler_debug";
+  public static final OptionValidator JAVA_COMPILER_DEBUG = new 
BooleanValidator(JAVA_COMPILER_DEBUG_OPTION, true);
+
+  public static final StringValidator JAVA_COMPILER_VALIDATOR = new 
StringValidator(JAVA_COMPILER_OPTION, CompilerPolicy.DEFAULT.toString()) {
+@Override
+public void validate(final OptionValue v, final OptionManager manager) 
{
+  super.validate(v, manager);
+  try {
+CompilerPolicy.valueOf(v.string_val.toUpperCase());
+  } catch (IllegalArgumentException e) {
+throw UserException.validationError()
+.message("Invalid value '%s' specified for option '%s'. Valid 
values are 

[jira] [Commented] (DRILL-5052) Option to debug generated Java code using an IDE

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15734268#comment-15734268
 ] 

ASF GitHub Bot commented on DRILL-5052:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/660#discussion_r91638703
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/compile/DrillJavaFileObject.java
 ---
@@ -67,6 +72,14 @@ public boolean isCompiled() {
 }
   }
 
+  public Map getResults( ) {
+Map results = new HashMap<>( );
--- End diff --

There is a note in the Java 8 docs that say that the newFoo() pattern was 
needed in prior Java versions to avoid:
`Map foo = new HashMap();`

But, under Java 7 (the diamond syntax was introduced with type inference, 
so the current preferred idiom is:
`Map foo = new HashMap<>();`

Drill code seems to have been written before the diamond syntax was 
available.


> Option to debug generated Java code using an IDE
> 
>
> Key: DRILL-5052
> URL: https://issues.apache.org/jira/browse/DRILL-5052
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Codegen
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> Drill makes extensive use of Java code generation to implement its operators. 
> Drill uses sophisticated techniques to blend generated code with pre-compiled 
> template code. An unfortunate side-effect of this behavior is that it is very 
> difficult to visualize and debug the generated code.
> As it turns out, Drill's code-merge facility is, in essence, a do-it-yourself 
> version of subclassing. The Drill "template" is the parent class, the 
> generated code is the subclass. But, rather than using plain-old subclassing, 
> Drill combines the code from the two classes into a single "artificial" 
> packet of byte codes for which no source exists.
> Modify the code generation path to optionally allow "plain-old Java" 
> compilation: the generated code is a subclass of the template. Compile the 
> generated code as a plain-old Java class with no byte-code fix-up. Write the 
> code to a known location that the IDE can search when looking for source 
> files.
> With this change, developers can turn on the above feature, set a breakpoint 
> in a template, then step directly into the generated Java code called from 
> the template.
> This feature should be an option, enabled by developers when needed. The 
> existing byte-code technique should be used for production code generation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5052) Option to debug generated Java code using an IDE

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15734259#comment-15734259
 ] 

ASF GitHub Bot commented on DRILL-5052:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/660#discussion_r91638426
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/compile/DrillJavaFileObject.java
 ---
@@ -67,6 +72,14 @@ public boolean isCompiled() {
 }
   }
 
+  public Map getResults( ) {
--- End diff --

Done.


> Option to debug generated Java code using an IDE
> 
>
> Key: DRILL-5052
> URL: https://issues.apache.org/jira/browse/DRILL-5052
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Codegen
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> Drill makes extensive use of Java code generation to implement its operators. 
> Drill uses sophisticated techniques to blend generated code with pre-compiled 
> template code. An unfortunate side-effect of this behavior is that it is very 
> difficult to visualize and debug the generated code.
> As it turns out, Drill's code-merge facility is, in essence, a do-it-yourself 
> version of subclassing. The Drill "template" is the parent class, the 
> generated code is the subclass. But, rather than using plain-old subclassing, 
> Drill combines the code from the two classes into a single "artificial" 
> packet of byte codes for which no source exists.
> Modify the code generation path to optionally allow "plain-old Java" 
> compilation: the generated code is a subclass of the template. Compile the 
> generated code as a plain-old Java class with no byte-code fix-up. Write the 
> code to a known location that the IDE can search when looking for source 
> files.
> With this change, developers can turn on the above feature, set a breakpoint 
> in a template, then step directly into the generated Java code called from 
> the template.
> This feature should be an option, enabled by developers when needed. The 
> existing byte-code technique should be used for production code generation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5052) Option to debug generated Java code using an IDE

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15734274#comment-15734274
 ] 

ASF GitHub Bot commented on DRILL-5052:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/660#discussion_r91632286
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/compile/CachedClassLoader.java
 ---
@@ -0,0 +1,70 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.compile;
+
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.Map;
+import java.util.concurrent.ConcurrentMap;
+
+import com.google.common.collect.MapMaker;
+
+/**
+ * Class loader for "plain-old Java" generated classes.
+ * Very basic implementation: allows defining a class from
+ * byte codes and finding the loaded classes. Delegates
+ * all other class requests to the thread context class
+ * loader. This structure ensures that a generated class can
+ * find both its own inner classes as well as all the standard
+ * Drill implementation classes.
+ */
+
+public class CachedClassLoader extends URLClassLoader {
+
+  /**
+   * Cache of generated classes. Semantics: a single thread defines
+   * the classes, many threads may access the classes.
+   */
+
+  private ConcurrentMap cache = new 
MapMaker().concurrencyLevel(4).makeMap();
+
+  public CachedClassLoader( ) {
+super(new URL[0], Thread.currentThread().getContextClassLoader());
+  }
+
+  public void addClass( String fqcn, byte[] byteCodes ) {
+
+assert ! cache.containsKey( fqcn );
+Class newClass = defineClass(fqcn, byteCodes, 0, byteCodes.length);
+cache.put( fqcn, newClass );
+  }
+
+  @Override
+  public Class findClass(String className) throws 
ClassNotFoundException {
+Class theClass = cache.get( className );
+if ( theClass != null ) {
+  return theClass; }
+return super.findClass(className);
+  }
+
+  public void addClasses(Map results) {
+for ( String key : results.keySet() ) {
+  addClass( key, results.get( key ) );
+}
--- End diff --

Done.


> Option to debug generated Java code using an IDE
> 
>
> Key: DRILL-5052
> URL: https://issues.apache.org/jira/browse/DRILL-5052
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Codegen
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> Drill makes extensive use of Java code generation to implement its operators. 
> Drill uses sophisticated techniques to blend generated code with pre-compiled 
> template code. An unfortunate side-effect of this behavior is that it is very 
> difficult to visualize and debug the generated code.
> As it turns out, Drill's code-merge facility is, in essence, a do-it-yourself 
> version of subclassing. The Drill "template" is the parent class, the 
> generated code is the subclass. But, rather than using plain-old subclassing, 
> Drill combines the code from the two classes into a single "artificial" 
> packet of byte codes for which no source exists.
> Modify the code generation path to optionally allow "plain-old Java" 
> compilation: the generated code is a subclass of the template. Compile the 
> generated code as a plain-old Java class with no byte-code fix-up. Write the 
> code to a known location that the IDE can search when looking for source 
> files.
> With this change, developers can turn on the above feature, set a breakpoint 
> in a template, then step directly into the generated Java code called from 
> the template.
> This feature should be an option, enabled by developers when needed. The 
> existing byte-code technique should be used for production 

[jira] [Commented] (DRILL-5052) Option to debug generated Java code using an IDE

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15734277#comment-15734277
 ] 

ASF GitHub Bot commented on DRILL-5052:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/660#discussion_r91642325
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/DirectExpression.java 
---
@@ -20,6 +20,12 @@
 import com.sun.codemodel.JExpressionImpl;
 import com.sun.codemodel.JFormatter;
 
+/**
+ * Encapsulates a Java expression, defined as anything that is
+ * valid in the following code:
+ * (expr)
+ */
+
 public class DirectExpression extends JExpressionImpl{
   static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(DirectExpression.class);
--- End diff --

Legacy code, but fixed.


> Option to debug generated Java code using an IDE
> 
>
> Key: DRILL-5052
> URL: https://issues.apache.org/jira/browse/DRILL-5052
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Codegen
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> Drill makes extensive use of Java code generation to implement its operators. 
> Drill uses sophisticated techniques to blend generated code with pre-compiled 
> template code. An unfortunate side-effect of this behavior is that it is very 
> difficult to visualize and debug the generated code.
> As it turns out, Drill's code-merge facility is, in essence, a do-it-yourself 
> version of subclassing. The Drill "template" is the parent class, the 
> generated code is the subclass. But, rather than using plain-old subclassing, 
> Drill combines the code from the two classes into a single "artificial" 
> packet of byte codes for which no source exists.
> Modify the code generation path to optionally allow "plain-old Java" 
> compilation: the generated code is a subclass of the template. Compile the 
> generated code as a plain-old Java class with no byte-code fix-up. Write the 
> code to a known location that the IDE can search when looking for source 
> files.
> With this change, developers can turn on the above feature, set a breakpoint 
> in a template, then step directly into the generated Java code called from 
> the template.
> This feature should be an option, enabled by developers when needed. The 
> existing byte-code technique should be used for production code generation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5052) Option to debug generated Java code using an IDE

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15734273#comment-15734273
 ] 

ASF GitHub Bot commented on DRILL-5052:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/660#discussion_r91632144
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/compile/CachedClassLoader.java
 ---
@@ -0,0 +1,70 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.compile;
+
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.Map;
+import java.util.concurrent.ConcurrentMap;
+
+import com.google.common.collect.MapMaker;
+
+/**
+ * Class loader for "plain-old Java" generated classes.
+ * Very basic implementation: allows defining a class from
+ * byte codes and finding the loaded classes. Delegates
+ * all other class requests to the thread context class
+ * loader. This structure ensures that a generated class can
+ * find both its own inner classes as well as all the standard
+ * Drill implementation classes.
+ */
+
+public class CachedClassLoader extends URLClassLoader {
+
+  /**
+   * Cache of generated classes. Semantics: a single thread defines
+   * the classes, many threads may access the classes.
+   */
+
+  private ConcurrentMap cache = new 
MapMaker().concurrencyLevel(4).makeMap();
+
+  public CachedClassLoader( ) {
+super(new URL[0], Thread.currentThread().getContextClassLoader());
+  }
+
+  public void addClass( String fqcn, byte[] byteCodes ) {
+
+assert ! cache.containsKey( fqcn );
--- End diff --

Done. Preconditions is compiled in, meaning it executes in production. 
Asserts fire only in debug. I'll follow the drill standard here, despite the 
cost.


> Option to debug generated Java code using an IDE
> 
>
> Key: DRILL-5052
> URL: https://issues.apache.org/jira/browse/DRILL-5052
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Codegen
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> Drill makes extensive use of Java code generation to implement its operators. 
> Drill uses sophisticated techniques to blend generated code with pre-compiled 
> template code. An unfortunate side-effect of this behavior is that it is very 
> difficult to visualize and debug the generated code.
> As it turns out, Drill's code-merge facility is, in essence, a do-it-yourself 
> version of subclassing. The Drill "template" is the parent class, the 
> generated code is the subclass. But, rather than using plain-old subclassing, 
> Drill combines the code from the two classes into a single "artificial" 
> packet of byte codes for which no source exists.
> Modify the code generation path to optionally allow "plain-old Java" 
> compilation: the generated code is a subclass of the template. Compile the 
> generated code as a plain-old Java class with no byte-code fix-up. Write the 
> code to a known location that the IDE can search when looking for source 
> files.
> With this change, developers can turn on the above feature, set a breakpoint 
> in a template, then step directly into the generated Java code called from 
> the template.
> This feature should be an option, enabled by developers when needed. The 
> existing byte-code technique should be used for production code generation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5052) Option to debug generated Java code using an IDE

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15734266#comment-15734266
 ] 

ASF GitHub Bot commented on DRILL-5052:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/660#discussion_r91632197
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/compile/CachedClassLoader.java
 ---
@@ -0,0 +1,70 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.compile;
+
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.Map;
+import java.util.concurrent.ConcurrentMap;
+
+import com.google.common.collect.MapMaker;
+
+/**
+ * Class loader for "plain-old Java" generated classes.
+ * Very basic implementation: allows defining a class from
+ * byte codes and finding the loaded classes. Delegates
+ * all other class requests to the thread context class
+ * loader. This structure ensures that a generated class can
+ * find both its own inner classes as well as all the standard
+ * Drill implementation classes.
+ */
+
+public class CachedClassLoader extends URLClassLoader {
+
+  /**
+   * Cache of generated classes. Semantics: a single thread defines
+   * the classes, many threads may access the classes.
+   */
+
+  private ConcurrentMap cache = new 
MapMaker().concurrencyLevel(4).makeMap();
+
+  public CachedClassLoader( ) {
+super(new URL[0], Thread.currentThread().getContextClassLoader());
+  }
+
+  public void addClass( String fqcn, byte[] byteCodes ) {
+
+assert ! cache.containsKey( fqcn );
+Class newClass = defineClass(fqcn, byteCodes, 0, byteCodes.length);
+cache.put( fqcn, newClass );
+  }
+
+  @Override
+  public Class findClass(String className) throws 
ClassNotFoundException {
+Class theClass = cache.get( className );
+if ( theClass != null ) {
+  return theClass; }
--- End diff --

Done. Also removed spaces to follow Drill style.


> Option to debug generated Java code using an IDE
> 
>
> Key: DRILL-5052
> URL: https://issues.apache.org/jira/browse/DRILL-5052
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Codegen
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> Drill makes extensive use of Java code generation to implement its operators. 
> Drill uses sophisticated techniques to blend generated code with pre-compiled 
> template code. An unfortunate side-effect of this behavior is that it is very 
> difficult to visualize and debug the generated code.
> As it turns out, Drill's code-merge facility is, in essence, a do-it-yourself 
> version of subclassing. The Drill "template" is the parent class, the 
> generated code is the subclass. But, rather than using plain-old subclassing, 
> Drill combines the code from the two classes into a single "artificial" 
> packet of byte codes for which no source exists.
> Modify the code generation path to optionally allow "plain-old Java" 
> compilation: the generated code is a subclass of the template. Compile the 
> generated code as a plain-old Java class with no byte-code fix-up. Write the 
> code to a known location that the IDE can search when looking for source 
> files.
> With this change, developers can turn on the above feature, set a breakpoint 
> in a template, then step directly into the generated Java code called from 
> the template.
> This feature should be an option, enabled by developers when needed. The 
> existing byte-code technique should be used for production code generation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5052) Option to debug generated Java code using an IDE

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15734272#comment-15734272
 ] 

ASF GitHub Bot commented on DRILL-5052:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/660#discussion_r91638121
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/compile/CodeCompiler.java ---
@@ -33,36 +32,71 @@
 import com.google.common.cache.LoadingCache;
 import com.google.common.collect.Lists;
 
+/**
+ * Global code compiler mechanism shared by all threads and operators.
+ * Holds a single cache of generated code (keyed by code source) to
+ * prevent compiling identical code multiple times. Supports both
+ * the byte-code merging and plain-old Java methods of code
+ * generation and compilation.
+ */
+
 public class CodeCompiler {
-//  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(CodeCompiler.class);
+
+  public static final String COMPILE_BASE = "drill.exec.compile";
+  String MAX_LOADING_CACHE_SIZE_CONFIG = COMPILE_BASE + ".cache_max_size";
 
   private final ClassTransformer transformer;
+  private final ClassBuilder classBuilder;
+
+  /**
+   * Google Guava loading cache that defers creating a cache
+   * entry until first needed. Creation is done in a thread-safe
+   * way: if two threads try to create the same class at the same
+   * time, the first does the work, the second waits for the first
+   * to complete, then grabs the new entry.
+   */
+
   private final LoadingCache cache;
-  private final DrillConfig config;
-  private final OptionManager optionManager;
 
   public CodeCompiler(final DrillConfig config, final OptionManager 
optionManager) {
-transformer = new ClassTransformer(optionManager);
-final int cacheMaxSize = 
config.getInt(ExecConstants.MAX_LOADING_CACHE_SIZE_CONFIG);
+transformer = new ClassTransformer(config, optionManager);
+classBuilder = new ClassBuilder(config, optionManager);
+final int cacheMaxSize = config.getInt(MAX_LOADING_CACHE_SIZE_CONFIG);
 cache = CacheBuilder.newBuilder()
 .maximumSize(cacheMaxSize)
 .build(new Loader());
-this.optionManager = optionManager;
-this.config = config;
   }
 
+  /**
+   * Create a single instance of the generated class.
+   *
+   * @param cg
+   * @return
+   * @throws ClassTransformationException
+   * @throws IOException
+   */
   @SuppressWarnings("unchecked")
-  public  T getImplementationClass(final CodeGenerator cg) throws 
ClassTransformationException, IOException {
-return (T) getImplementationClass(cg, 1).get(0);
+  public  T createInstance(final CodeGenerator cg) throws 
ClassTransformationException, IOException {
+return (T) createInstances(cg, 1).get(0);
   }
 
+  /**
+   * Create multiple instances of the generated class.
+   *
+   * @param cg
+   * @param count
+   * @return
+   * @throws ClassTransformationException
+   * @throws IOException
--- End diff --

Done.


> Option to debug generated Java code using an IDE
> 
>
> Key: DRILL-5052
> URL: https://issues.apache.org/jira/browse/DRILL-5052
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Codegen
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> Drill makes extensive use of Java code generation to implement its operators. 
> Drill uses sophisticated techniques to blend generated code with pre-compiled 
> template code. An unfortunate side-effect of this behavior is that it is very 
> difficult to visualize and debug the generated code.
> As it turns out, Drill's code-merge facility is, in essence, a do-it-yourself 
> version of subclassing. The Drill "template" is the parent class, the 
> generated code is the subclass. But, rather than using plain-old subclassing, 
> Drill combines the code from the two classes into a single "artificial" 
> packet of byte codes for which no source exists.
> Modify the code generation path to optionally allow "plain-old Java" 
> compilation: the generated code is a subclass of the template. Compile the 
> generated code as a plain-old Java class with no byte-code fix-up. Write the 
> code to a known location that the IDE can search when looking for source 
> files.
> With this change, developers can turn on the above feature, set a breakpoint 
> in a template, then step directly into the generated Java code called from 
> the 

[jira] [Commented] (DRILL-5052) Option to debug generated Java code using an IDE

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15734270#comment-15734270
 ] 

ASF GitHub Bot commented on DRILL-5052:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/660#discussion_r91634371
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/compile/ClassCompilerSelector.java
 ---
@@ -0,0 +1,146 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.compile;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Map;
+
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.compile.ClassTransformer.ClassNames;
+import org.apache.drill.exec.exception.ClassTransformationException;
+import org.apache.drill.exec.server.options.OptionManager;
+import org.apache.drill.exec.server.options.OptionValidator;
+import org.apache.drill.exec.server.options.OptionValue;
+import 
org.apache.drill.exec.server.options.TypeValidators.BooleanValidator;
+import org.apache.drill.exec.server.options.TypeValidators.LongValidator;
+import org.apache.drill.exec.server.options.TypeValidators.StringValidator;
+import org.codehaus.commons.compiler.CompileException;
+
+/**
+ * Selects between the two supported Java compilers: Janino and
+ * the build-in Java compiler.
+ *
+ * Session Options
+ * 
+ * exec.java_compiler
+ * The compiler to use. Valid options are defined in the
+ * {@link ClassCompilerSelector.CompilerPolicy} enum.
+ * exec.java_compiler_debug
+ * If debug logging is enabled, then {@link AbstractClassCompiler} 
writes the
+ * generated Java code to the log file prior to compilation. This option
+ * adds line numbers to the logged code.
+ * exec.java_compiler_janino_maxsize
+ * The maximum size of code that the Janio compiler can handle. Larger 
code is
+ * handled by the JDK compiler. Defaults to 256K.
+ * 
+ * Configuration Options
+ * Configuration options are used when the above session options are unset.
+ * 
+ * drill.exec.compile.compiler
+ * Default for exec.java_compiler
+ * drill.exec.compile.debug
+ * Default for exec.java_compiler_debug
+ * drill.exec.compile.janino_maxsize
+ * Default for exec.java_compiler_janino_maxsize
+ * 
+ */
+
+public class ClassCompilerSelector {
+  public enum CompilerPolicy {
+DEFAULT, JDK, JANINO;
+  }
+
+  public static final String JAVA_COMPILER_JANINO_MAXSIZE_CONFIG = 
CodeCompiler.COMPILE_BASE + ".janino_maxsize";
+  public static final String JAVA_COMPILER_DEBUG_CONFIG = 
CodeCompiler.COMPILE_BASE + ".debug";
+  public static final String JAVA_COMPILER_CONFIG = 
CodeCompiler.COMPILE_BASE + ".compiler";
+
+  public static final String JAVA_COMPILER_OPTION = "exec.java_compiler";
+  public static final String JAVA_COMPILER_JANINO_MAXSIZE_OPTION = 
"exec.java_compiler_janino_maxsize";
+  public static final OptionValidator JAVA_COMPILER_JANINO_MAXSIZE = new 
LongValidator(JAVA_COMPILER_JANINO_MAXSIZE_OPTION, 256*1024);
+
+  public static final String JAVA_COMPILER_DEBUG_OPTION = 
"exec.java_compiler_debug";
+  public static final OptionValidator JAVA_COMPILER_DEBUG = new 
BooleanValidator(JAVA_COMPILER_DEBUG_OPTION, true);
+
+  public static final StringValidator JAVA_COMPILER_VALIDATOR = new 
StringValidator(JAVA_COMPILER_OPTION, CompilerPolicy.DEFAULT.toString()) {
+@Override
+public void validate(final OptionValue v, final OptionManager manager) 
{
+  super.validate(v, manager);
+  try {
+CompilerPolicy.valueOf(v.string_val.toUpperCase());
+  } catch (IllegalArgumentException e) {
+throw UserException.validationError()
+.message("Invalid value '%s' specified for option '%s'. Valid 
values are 

[jira] [Commented] (DRILL-5052) Option to debug generated Java code using an IDE

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15734282#comment-15734282
 ] 

ASF GitHub Bot commented on DRILL-5052:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/660#discussion_r91642060
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/DebugStringBuilder.java 
---
@@ -0,0 +1,57 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.expr;
+
+import java.io.PrintWriter;
+import java.io.StringWriter;
+
+import com.sun.codemodel.JFormatter;
+
+/**
+ * Utility class to build a debug string for an object
+ * in a standard format.
+ */
--- End diff --

Done. And, yes, it isn't much of a formatter yet...


> Option to debug generated Java code using an IDE
> 
>
> Key: DRILL-5052
> URL: https://issues.apache.org/jira/browse/DRILL-5052
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Codegen
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> Drill makes extensive use of Java code generation to implement its operators. 
> Drill uses sophisticated techniques to blend generated code with pre-compiled 
> template code. An unfortunate side-effect of this behavior is that it is very 
> difficult to visualize and debug the generated code.
> As it turns out, Drill's code-merge facility is, in essence, a do-it-yourself 
> version of subclassing. The Drill "template" is the parent class, the 
> generated code is the subclass. But, rather than using plain-old subclassing, 
> Drill combines the code from the two classes into a single "artificial" 
> packet of byte codes for which no source exists.
> Modify the code generation path to optionally allow "plain-old Java" 
> compilation: the generated code is a subclass of the template. Compile the 
> generated code as a plain-old Java class with no byte-code fix-up. Write the 
> code to a known location that the IDE can search when looking for source 
> files.
> With this change, developers can turn on the above feature, set a breakpoint 
> in a template, then step directly into the generated Java code called from 
> the template.
> This feature should be an option, enabled by developers when needed. The 
> existing byte-code technique should be used for production code generation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5052) Option to debug generated Java code using an IDE

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15734267#comment-15734267
 ] 

ASF GitHub Bot commented on DRILL-5052:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/660#discussion_r91637977
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/compile/CodeCompiler.java ---
@@ -33,36 +32,71 @@
 import com.google.common.cache.LoadingCache;
 import com.google.common.collect.Lists;
 
+/**
+ * Global code compiler mechanism shared by all threads and operators.
+ * Holds a single cache of generated code (keyed by code source) to
+ * prevent compiling identical code multiple times. Supports both
+ * the byte-code merging and plain-old Java methods of code
+ * generation and compilation.
+ */
+
 public class CodeCompiler {
-//  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(CodeCompiler.class);
+
+  public static final String COMPILE_BASE = "drill.exec.compile";
+  String MAX_LOADING_CACHE_SIZE_CONFIG = COMPILE_BASE + ".cache_max_size";
 
   private final ClassTransformer transformer;
+  private final ClassBuilder classBuilder;
+
+  /**
+   * Google Guava loading cache that defers creating a cache
+   * entry until first needed. Creation is done in a thread-safe
+   * way: if two threads try to create the same class at the same
+   * time, the first does the work, the second waits for the first
+   * to complete, then grabs the new entry.
+   */
+
   private final LoadingCache cache;
-  private final DrillConfig config;
-  private final OptionManager optionManager;
 
   public CodeCompiler(final DrillConfig config, final OptionManager 
optionManager) {
-transformer = new ClassTransformer(optionManager);
-final int cacheMaxSize = 
config.getInt(ExecConstants.MAX_LOADING_CACHE_SIZE_CONFIG);
+transformer = new ClassTransformer(config, optionManager);
+classBuilder = new ClassBuilder(config, optionManager);
+final int cacheMaxSize = config.getInt(MAX_LOADING_CACHE_SIZE_CONFIG);
 cache = CacheBuilder.newBuilder()
 .maximumSize(cacheMaxSize)
 .build(new Loader());
-this.optionManager = optionManager;
-this.config = config;
   }
 
+  /**
+   * Create a single instance of the generated class.
+   *
+   * @param cg
+   * @return
+   * @throws ClassTransformationException
+   * @throws IOException
+   */
--- End diff --

Done.


> Option to debug generated Java code using an IDE
> 
>
> Key: DRILL-5052
> URL: https://issues.apache.org/jira/browse/DRILL-5052
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Codegen
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> Drill makes extensive use of Java code generation to implement its operators. 
> Drill uses sophisticated techniques to blend generated code with pre-compiled 
> template code. An unfortunate side-effect of this behavior is that it is very 
> difficult to visualize and debug the generated code.
> As it turns out, Drill's code-merge facility is, in essence, a do-it-yourself 
> version of subclassing. The Drill "template" is the parent class, the 
> generated code is the subclass. But, rather than using plain-old subclassing, 
> Drill combines the code from the two classes into a single "artificial" 
> packet of byte codes for which no source exists.
> Modify the code generation path to optionally allow "plain-old Java" 
> compilation: the generated code is a subclass of the template. Compile the 
> generated code as a plain-old Java class with no byte-code fix-up. Write the 
> code to a known location that the IDE can search when looking for source 
> files.
> With this change, developers can turn on the above feature, set a breakpoint 
> in a template, then step directly into the generated Java code called from 
> the template.
> This feature should be an option, enabled by developers when needed. The 
> existing byte-code technique should be used for production code generation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-4812) Wildcard queries fail on Windows

[Paul Rogers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paul Rogers updated DRILL-4812:
---
  Labels: easyfix easytest ready-to-commit windows  (was: easyfix easytest 
windows)
Reviewer: Paul Rogers

> Wildcard queries fail on Windows
> 
>
> Key: DRILL-4812
> URL: https://issues.apache.org/jira/browse/DRILL-4812
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Other
>Affects Versions: 1.7.0
> Environment: Windows 7
>Reporter: Mike Lavender
>  Labels: easyfix, easytest, ready-to-commit, windows
>
> Wildcards within the path of a query are not handled on windows and result in 
> a "String index out of range" exception.
> for example:
> {noformat}
> 0: jdbc:drill:zk=local> SELECT SUM(qty) as num FROM 
> dfs.parquet.`/trends/2016/1/*/*/3701`;
> Error: VALIDATION ERROR: String index out of range: -1
> SQL Query null
> {noformat}
> 
> The problem exists within:
> exec\java-exec\src\main\java\org\apache\drill\exec\store\dfs\FileSelection.java
> private static Path handleWildCard(final String root)
> This function is looking for the index of the system specific PATH_SEPARATOR 
> which on windows is '\' (from System.getProperty("file.separator")).  The 
> path passed in to handleWildcard will not ever have those type of path 
> separators as the Path constructor (from org.apache.hadoop.fs.Path) sets all 
> the path separators to '/'.
> NOTE:
> private static String removeLeadingSlash(String path)
> in that same file explicitly looks for '/' and does not use the system 
> specific PATH_SEPARATOR.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-5116) Enable generated code debugging in each Drill operator

[Paul Rogers (JIRA)]

Paul Rogers created DRILL-5116:
--

 Summary: Enable generated code debugging in each Drill operator
 Key: DRILL-5116
 URL: https://issues.apache.org/jira/browse/DRILL-5116
 Project: Apache Drill
  Issue Type: Improvement
Affects Versions: 1.9.0
Reporter: Paul Rogers
Assignee: Paul Rogers
Priority: Minor


DRILL-5052 adds the ability to debug generated code. Some of the code generated 
by Drill's operators has minor problems when compiled directly using the new 
technique. These issues are ignore by the byte-code-merge technique uses in 
production. This ticket asks to try the DRILL-5052 feature in each operator, 
clean up any minor problems, and ensure each operator generates code suitable 
for debugging. Use the new {{CodeGenerator.plainOldJavaCapable()}} method to 
mark each generated class as ready for "plain-old Java" code gen.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5115) Metadata Cache Pruning randomly returns wrong results at higher concurrencies

[Rahul Challapalli (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15733645#comment-15733645
 ] 

Rahul Challapalli commented on DRILL-5115:
--

The same thing can be observed for the below query. Filter is removed and is 
pushed down into the scan. However the scan fails to apply the filter.

Query 
{code}
select num, let, `day`, l_orderkey from ld5 where num=2 and substring(let, 1, 
1)='o' and `day` = 12 and `year`=2015 and `month` in (7,9) order by l_orderkey 
limit 10;
{code}

Wong Plan : 
{code}
00-00Screen : rowType = RecordType(ANY num, ANY let, BIGINT day, ANY 
l_orderkey): rowcount = 10.0, cumulative cost = {1261.0 rows, 8200.19083766033 
cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 7786
00-01  Project(num=[$0], let=[$1], day=[$2], l_orderkey=[$3]) : rowType = 
RecordType(ANY num, ANY let, BIGINT day, ANY l_orderkey): rowcount = 10.0, 
cumulative cost = {1260.0 rows, 8199.19083766033 cpu, 0.0 io, 0.0 network, 0.0 
memory}, id = 7785
00-02SelectionVectorRemover : rowType = RecordType(ANY num, ANY let, 
BIGINT day, ANY l_orderkey): rowcount = 10.0, cumulative cost = {1260.0 rows, 
8199.19083766033 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 7784
00-03  Limit(fetch=[10]) : rowType = RecordType(ANY num, ANY let, 
BIGINT day, ANY l_orderkey): rowcount = 10.0, cumulative cost = {1250.0 rows, 
8189.19083766033 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 7783
00-04SelectionVectorRemover : rowType = RecordType(ANY num, ANY 
let, BIGINT day, ANY l_orderkey): rowcount = 310.0, cumulative cost = {1240.0 
rows, 8149.19083766033 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 7782
00-05  TopN(limit=[10]) : rowType = RecordType(ANY num, ANY let, 
BIGINT day, ANY l_orderkey): rowcount = 310.0, cumulative cost = {930.0 rows, 
7839.19083766033 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 7781
00-06Project(num=[$0], let=[SUBSTR($1, 1, 2)], 
day=[EXTRACT(FLAG(DAY), $2)], l_orderkey=[$3]) : rowType = RecordType(ANY num, 
ANY let, BIGINT day, ANY l_orderkey): rowcount = 310.0, cumulative cost = 
{620.0 rows, 3720.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 7780
00-07  Scan(groupscan=[ParquetGroupScan 
[entries=[ReadEntryWithPath 
[path=maprfs:/drill/testdata/metadata_caching_pp/l_3level/2/one/2015-7-12/50.parquet],
 ReadEntryWithPath 
[path=maprfs:/drill/testdata/metadata_caching_pp/l_3level/2/one/2015-9-12/20.parquet]],
 selectionRoot=maprfs:/drill/testdata/metadata_caching_pp/l_3level, numFiles=2, 
usedMetadataFile=true, 
cacheFileRoot=/drill/testdata/metadata_caching_pp/l_3level, columns=[`dir0`, 
`dir1`, `dir2`, `l_orderkey`]]]) : rowType = RecordType(ANY dir0, ANY dir1, ANY 
dir2, ANY l_orderkey): rowcount = 310.0, cumulative cost = {310.0 rows, 1240.0 
cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 7779
{code}

Correct Plan :
{code}
00-00Screen : rowType = RecordType(ANY num, ANY let, BIGINT day, ANY 
l_orderkey): rowcount = 10.0, cumulative cost = {165.0 rows, 683.7098998795495 
cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 24930
00-01  Project(num=[$0], let=[$1], day=[$2], l_orderkey=[$3]) : rowType = 
RecordType(ANY num, ANY let, BIGINT day, ANY l_orderkey): rowcount = 10.0, 
cumulative cost = {164.0 rows, 682.7098998795495 cpu, 0.0 io, 0.0 network, 0.0 
memory}, id = 24929
00-02SelectionVectorRemover : rowType = RecordType(ANY num, ANY let, 
BIGINT day, ANY l_orderkey): rowcount = 10.0, cumulative cost = {164.0 rows, 
682.7098998795495 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 24928
00-03  Limit(fetch=[10]) : rowType = RecordType(ANY num, ANY let, 
BIGINT day, ANY l_orderkey): rowcount = 10.0, cumulative cost = {154.0 rows, 
672.7098998795495 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 24927
00-04SelectionVectorRemover : rowType = RecordType(ANY num, ANY 
let, BIGINT day, ANY l_orderkey): rowcount = 1.0, cumulative cost = {144.0 
rows, 632.7098998795495 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 24926
00-05  TopN(limit=[10]) : rowType = RecordType(ANY num, ANY let, 
BIGINT day, ANY l_orderkey): rowcount = 1.0, cumulative cost = {143.0 rows, 
631.7098998795495 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 24925
00-06Project(num=[$0], let=[SUBSTR($1, 1, 2)], 
day=[EXTRACT(FLAG(DAY), $2)], l_orderkey=[$3]) : rowType = RecordType(ANY num, 
ANY let, BIGINT day, ANY l_orderkey): rowcount = 1.0, cumulative cost = {142.0 
rows, 618.422187501 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 24924
00-07  SelectionVectorRemover : rowType = RecordType(ANY dir0, 
ANY dir1, ANY dir2, ANY l_orderkey): rowcount = 1.0, cumulative cost = {141.0 
rows, 610.422187501 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 24923
00-08Filter(condition=[AND(=($0, 2), =(SUBSTRING(SUBSTR($1, 
1, 2), 1, 1), 'o'), =(EXTRACT(FLAG(DAY), $2), 12), =(EXTRACT(FLAG(YEAR), $2), 

[jira] [Updated] (DRILL-5115) Metadata Cache Pruning randomly returns wrong results at higher concurrencies

[Rahul Challapalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rahul Challapalli updated DRILL-5115:
-
Attachment: wrongdata_profile.txt
l_3level.tgz
drillbit2.log
drillbit1.log
correctdata_profile.txt

> Metadata Cache Pruning randomly returns wrong results at higher concurrencies
> -
>
> Key: DRILL-5115
> URL: https://issues.apache.org/jira/browse/DRILL-5115
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Metadata, Query Planning & Optimization
>Affects Versions: 1.8.0, 1.9.0, 1.10
>Reporter: Rahul Challapalli
> Attachments: correctdata_profile.txt, drillbit1.log, drillbit2.log, 
> l_3level.tgz, wrongdata_profile.txt
>
>
> git.commit.id.abbrev=4312d65
> When multiple queries are updating the metadata cache simultaneously the 
> below query randomly returns wrong results. 
> A single run includes executing a suite of 90 tests at a concurrency of 50. I 
> encountered a wrong data scenario in my 10th run.
>  
> Query :
> {code}
> select l_orderkey from l_3level where dir0=1 and ((dir1='one' and dir2 IN 
> ('2015-7-12', '2015-7-13')) or (dir1='two' and dir2='2015-8-12'))
> {code}
> Wrong Result Plan (based on the profile) : 
> {code}
> 00-00Screen : rowType = RecordType(ANY l_orderkey): rowcount = 310.0, 
> cumulative cost = {341.0 rows, 341.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, 
> id = 205721
> 00-01  Project(l_orderkey=[$0]) : rowType = RecordType(ANY l_orderkey): 
> rowcount = 310.0, cumulative cost = {310.0 rows, 310.0 cpu, 0.0 io, 0.0 
> network, 0.0 memory}, id = 205720
> 00-02Scan(groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath 
> [path=maprfs:/drill/testdata/metadata_caching_pp/l_3level/1/one/2015-7-13/20.parquet],
>  ReadEntryWithPath 
> [path=maprfs:/drill/testdata/metadata_caching_pp/l_3level/1/two/2015-8-12/30.parquet],
>  ReadEntryWithPath 
> [path=maprfs:/drill/testdata/metadata_caching_pp/l_3level/1/one/2015-7-12/10.parquet]],
>  selectionRoot=maprfs:/drill/testdata/metadata_caching_pp/l_3level, 
> numFiles=3, usedMetadataFile=true, 
> cacheFileRoot=/drill/testdata/metadata_caching_pp/l_3level, 
> columns=[`l_orderkey`]]]) : rowType = RecordType(ANY l_orderkey): rowcount = 
> 310.0, cumulative cost = {310.0 rows, 310.0 cpu, 0.0 io, 0.0 network, 0.0 
> memory}, id = 205719
> {code}
> Correct Result Plan (based on the profile):
> {code}
> 00-00Screen : rowType = RecordType(ANY l_orderkey): rowcount = 2.25, 
> cumulative cost = {122.475 rows, 527.475 cpu, 0.0 io, 0.0 network, 0.0 
> memory}, id = 226849
> 00-01  Project(l_orderkey=[$3]) : rowType = RecordType(ANY l_orderkey): 
> rowcount = 2.25, cumulative cost = {122.25 rows, 527.25 cpu, 0.0 io, 0.0 
> network, 0.0 memory}, id = 226848
> 00-02SelectionVectorRemover : rowType = RecordType(ANY dir0, ANY 
> dir1, ANY dir2, ANY l_orderkey): rowcount = 2.25, cumulative cost = {122.25 
> rows, 527.25 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 226847
> 00-03  Filter(condition=[AND(=($0, 1), OR(AND(=($1, 'one'), OR(=($2, 
> '2015-7-12'), =($2, '2015-7-13'))), AND(=($1, 'two'), =($2, '2015-8-12']) 
> : rowType = RecordType(ANY dir0, ANY dir1, ANY dir2, ANY l_orderkey): 
> rowcount = 2.25, cumulative cost = {120.0 rows, 525.0 cpu, 0.0 io, 0.0 
> network, 0.0 memory}, id = 226846
> 00-04Scan(groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath 
> [path=/drill/testdata/metadata_caching_pp/l_3level/1/one/2015-7-13/20.parquet],
>  ReadEntryWithPath 
> [path=/drill/testdata/metadata_caching_pp/l_3level/1/two/2015-8-12/30.parquet],
>  ReadEntryWithPath 
> [path=/drill/testdata/metadata_caching_pp/l_3level/1/one/2015-7-12/10.parquet]],
>  selectionRoot=/drill/testdata/metadata_caching_pp/l_3level, numFiles=3, 
> usedMetadataFile=true, 
> cacheFileRoot=/drill/testdata/metadata_caching_pp/l_3level/1, 
> columns=[`dir0`, `dir1`, `dir2`, `l_orderkey`]]]) : rowType = RecordType(ANY 
> dir0, ANY dir1, ANY dir2, ANY l_orderkey): rowcount = 60.0, cumulative cost = 
> {60.0 rows, 240.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 226845
> {code}
> I attached the data set, log files and the query profiles. Let me know if you 
> need anything



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-5115) Metadata Cache Pruning randomly returns wrong results at higher concurrencies

[Rahul Challapalli (JIRA)]

Rahul Challapalli created DRILL-5115:


 Summary: Metadata Cache Pruning randomly returns wrong results at 
higher concurrencies
 Key: DRILL-5115
 URL: https://issues.apache.org/jira/browse/DRILL-5115
 Project: Apache Drill
  Issue Type: Bug
  Components: Metadata, Query Planning & Optimization
Affects Versions: 1.8.0, 1.9.0, 1.10
Reporter: Rahul Challapalli


git.commit.id.abbrev=4312d65

When multiple queries are updating the metadata cache simultaneously the below 
query randomly returns wrong results. 

A single run includes executing a suite of 90 tests at a concurrency of 50. I 
encountered a wrong data scenario in my 10th run.
 
Query :
{code}
select l_orderkey from l_3level where dir0=1 and ((dir1='one' and dir2 IN 
('2015-7-12', '2015-7-13')) or (dir1='two' and dir2='2015-8-12'))
{code}

Wrong Result Plan (based on the profile) : 
{code}
00-00Screen : rowType = RecordType(ANY l_orderkey): rowcount = 310.0, 
cumulative cost = {341.0 rows, 341.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id 
= 205721
00-01  Project(l_orderkey=[$0]) : rowType = RecordType(ANY l_orderkey): 
rowcount = 310.0, cumulative cost = {310.0 rows, 310.0 cpu, 0.0 io, 0.0 
network, 0.0 memory}, id = 205720
00-02Scan(groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath 
[path=maprfs:/drill/testdata/metadata_caching_pp/l_3level/1/one/2015-7-13/20.parquet],
 ReadEntryWithPath 
[path=maprfs:/drill/testdata/metadata_caching_pp/l_3level/1/two/2015-8-12/30.parquet],
 ReadEntryWithPath 
[path=maprfs:/drill/testdata/metadata_caching_pp/l_3level/1/one/2015-7-12/10.parquet]],
 selectionRoot=maprfs:/drill/testdata/metadata_caching_pp/l_3level, numFiles=3, 
usedMetadataFile=true, 
cacheFileRoot=/drill/testdata/metadata_caching_pp/l_3level, 
columns=[`l_orderkey`]]]) : rowType = RecordType(ANY l_orderkey): rowcount = 
310.0, cumulative cost = {310.0 rows, 310.0 cpu, 0.0 io, 0.0 network, 0.0 
memory}, id = 205719
{code}

Correct Result Plan (based on the profile):
{code}
00-00Screen : rowType = RecordType(ANY l_orderkey): rowcount = 2.25, 
cumulative cost = {122.475 rows, 527.475 cpu, 0.0 io, 0.0 network, 0.0 memory}, 
id = 226849
00-01  Project(l_orderkey=[$3]) : rowType = RecordType(ANY l_orderkey): 
rowcount = 2.25, cumulative cost = {122.25 rows, 527.25 cpu, 0.0 io, 0.0 
network, 0.0 memory}, id = 226848
00-02SelectionVectorRemover : rowType = RecordType(ANY dir0, ANY dir1, 
ANY dir2, ANY l_orderkey): rowcount = 2.25, cumulative cost = {122.25 rows, 
527.25 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 226847
00-03  Filter(condition=[AND(=($0, 1), OR(AND(=($1, 'one'), OR(=($2, 
'2015-7-12'), =($2, '2015-7-13'))), AND(=($1, 'two'), =($2, '2015-8-12']) : 
rowType = RecordType(ANY dir0, ANY dir1, ANY dir2, ANY l_orderkey): rowcount = 
2.25, cumulative cost = {120.0 rows, 525.0 cpu, 0.0 io, 0.0 network, 0.0 
memory}, id = 226846
00-04Scan(groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath 
[path=/drill/testdata/metadata_caching_pp/l_3level/1/one/2015-7-13/20.parquet], 
ReadEntryWithPath 
[path=/drill/testdata/metadata_caching_pp/l_3level/1/two/2015-8-12/30.parquet], 
ReadEntryWithPath 
[path=/drill/testdata/metadata_caching_pp/l_3level/1/one/2015-7-12/10.parquet]],
 selectionRoot=/drill/testdata/metadata_caching_pp/l_3level, numFiles=3, 
usedMetadataFile=true, 
cacheFileRoot=/drill/testdata/metadata_caching_pp/l_3level/1, columns=[`dir0`, 
`dir1`, `dir2`, `l_orderkey`]]]) : rowType = RecordType(ANY dir0, ANY dir1, ANY 
dir2, ANY l_orderkey): rowcount = 60.0, cumulative cost = {60.0 rows, 240.0 
cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 226845
{code}

I attached the data set, log files and the query profiles. Let me know if you 
need anything



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (DRILL-5041) Make SingleRowListener (and other utilities) published public classes

[Chris Westin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Westin reassigned DRILL-5041:
---

Assignee: Chris Westin

> Make SingleRowListener (and other utilities) published public classes
> -
>
> Key: DRILL-5041
> URL: https://issues.apache.org/jira/browse/DRILL-5041
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.8.0
> Environment: This is actually for the Java Client, but there's no 
> such component.
>Reporter: Chris Westin
>Assignee: Chris Westin
>
> I have an application that uses the DrillClient interface (Specifically, an 
> implementation of OJAI), and it would have been convenient to use things like 
> SingleRowListener in my implementation, but they are not available outside 
> the Drill project. There are many such utilities that are used in unit tests 
> that would be useful to external API users.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-5041) Make SingleRowListener (and other utilities) published public classes

[Chris Westin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Westin updated DRILL-5041:

Component/s: (was: Client - C++)

> Make SingleRowListener (and other utilities) published public classes
> -
>
> Key: DRILL-5041
> URL: https://issues.apache.org/jira/browse/DRILL-5041
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.8.0
> Environment: This is actually for the Java Client, but there's no 
> such component.
>Reporter: Chris Westin
>Assignee: Chris Westin
>
> I have an application that uses the DrillClient interface (Specifically, an 
> implementation of OJAI), and it would have been convenient to use things like 
> SingleRowListener in my implementation, but they are not available outside 
> the Drill project. There are many such utilities that are used in unit tests 
> that would be useful to external API users.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5070) Code gen: create methods in fixed order to allow test verification

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15733047#comment-15733047
 ] 

ASF GitHub Bot commented on DRILL-5070:
---

Github user paul-rogers commented on the issue:

https://github.com/apache/drill/pull/684
  
Golden copies are needed only for tests that, today, do their "testing" by 
dumping generated code to stdout. Such tests do absolutely nothing (other than 
detect crashes) when run under Maven. The golden copy assures that such 
low-level tests are verified.

Other tests verify results by executing code. No need to save a golden copy 
of code for those.

Yes, when code gen changes, the golden copies have to be recaptured. 
However, how else can we verify that the new changes do what we expect if we 
don't actually examine them and save the desired final state? How else do we 
detect unexpected changes to generated code? (For example, if we generated a 
bunch of unneeded boilerplate, we'd not detect that in an execution test.)

This test is also going to be a way to verify DRILL-5052: capture code 
using the current code gen technique, then verify against the code path 
described in DRILL-5052. The two paths should be identical except for an extra 
"extends" in the code from DRILL-5052.

Is it worth testing such cases or is it OK to simply trust that the 
generated code is probably OK?


> Code gen: create methods in fixed order to allow test verification
> --
>
> Key: DRILL-5070
> URL: https://issues.apache.org/jira/browse/DRILL-5070
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> A handy technique in testing is to compare generated code against a "golden" 
> copy that defines the expected results. However, at present, Drill generates 
> code using the method order returned by {{Class.getDeclaredMethods}}, but 
> this method makes no guarantee about the order of the methods. The order 
> varies from one run to the next. There is some evidence [this 
> link|http://stackoverflow.com/questions/28585843/java-reflection-getdeclaredmethods-in-declared-order-strange-behaviour]
>  that order can vary even within a single run, though a quick test was unable 
> to reproduce this case.
> If method order does indeed vary within a single run, then the order can 
> impact the Drill code cache since it compares the sources from two different 
> generation events to detect duplicate code.
> This issue appeared when attempting to modify tests to capture generated code 
> for comparison to future results. Even a simple generated case from 
> {{ExpressionTest.testBasicExpression()}} that generates {{if(true) then 1 
> else 0 end}} (all constants) produced methods in different orders on each 
> test run.
> The fix is simple, in the {{SignatureHolder}} constructor, sort methods by 
> name after retrieving them from the class. The sort ensures that method order 
> is deterministic. Fortunately, the number of methods is small, so the sort 
> step adds little cost.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5070) Code gen: create methods in fixed order to allow test verification

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15733015#comment-15733015
 ] 

ASF GitHub Bot commented on DRILL-5070:
---

Github user jinfengni commented on the issue:

https://github.com/apache/drill/pull/684
  
It's fine to have fixed order for the generated methods. I'm not sure if 
it's appropriate to enforce  "golden" copy in the new tests. The golden copy is 
just one implementation in the current code; people may change the 
implementation in the future, and it's likely these new testcases will fail. 
The person to make the change in the future will have to deal with the overhead.

The question is what benefit we get from enforcing such "golden" copy in 
the testcases.  


> Code gen: create methods in fixed order to allow test verification
> --
>
> Key: DRILL-5070
> URL: https://issues.apache.org/jira/browse/DRILL-5070
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> A handy technique in testing is to compare generated code against a "golden" 
> copy that defines the expected results. However, at present, Drill generates 
> code using the method order returned by {{Class.getDeclaredMethods}}, but 
> this method makes no guarantee about the order of the methods. The order 
> varies from one run to the next. There is some evidence [this 
> link|http://stackoverflow.com/questions/28585843/java-reflection-getdeclaredmethods-in-declared-order-strange-behaviour]
>  that order can vary even within a single run, though a quick test was unable 
> to reproduce this case.
> If method order does indeed vary within a single run, then the order can 
> impact the Drill code cache since it compares the sources from two different 
> generation events to detect duplicate code.
> This issue appeared when attempting to modify tests to capture generated code 
> for comparison to future results. Even a simple generated case from 
> {{ExpressionTest.testBasicExpression()}} that generates {{if(true) then 1 
> else 0 end}} (all constants) produced methods in different orders on each 
> test run.
> The fix is simple, in the {{SignatureHolder}} constructor, sort methods by 
> name after retrieving them from the class. The sort ensures that method order 
> is deterministic. Fortunately, the number of methods is small, so the sort 
> step adds little cost.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4996) Parquet Date auto-correction is not working in auto-partitioned parquet files generated by drill-1.6

[Vitalii Diravka (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15732970#comment-15732970
 ] 

Vitalii Diravka commented on DRILL-4996:


Yes, I meant exactly this. In the result we will completely sure that with 
drill we will always have correct date values from parquet files. 


> Parquet Date auto-correction is not working in auto-partitioned parquet files 
> generated by drill-1.6
> 
>
> Key: DRILL-4996
> URL: https://issues.apache.org/jira/browse/DRILL-4996
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Parquet
>Reporter: Rahul Challapalli
>Assignee: Vitalii Diravka
>Priority: Critical
> Attachments: item.tgz
>
>
> git.commit.id.abbrev=4ee1d4c
> Below are the steps I followed to generate the data :
> {code}
> 1. Generate a parquet file with date column using hive1.2
> 2. Use drill 1.6 to create auto-partitioned parquet files partitioned on the 
> date column
> {code}
> Now the below query returns wrong results :
> {code}
> select i_rec_start_date, i_size from 
> dfs.`/drill/testdata/parquet_date/auto_partition/item_multipart_autorefresh`  
> group by i_rec_start_date, i_size;
> +---+--+
> | i_rec_start_date  |i_size|
> +---+--+
> | null  | large|
> | 366-11-08| extra large  |
> | 366-11-08| medium   |
> | null  | medium   |
> | 366-11-08| petite   |
> | 364-11-07| medium   |
> | null  | petite   |
> | 365-11-07| medium   |
> | 368-11-07| economy  |
> | 365-11-07| large|
> | 365-11-07| small|
> | 366-11-08| small|
> | 365-11-07| extra large  |
> | 364-11-07| N/A  |
> | 366-11-08| economy  |
> | 366-11-08| large|
> | 364-11-07| small|
> | null  | small|
> | 364-11-07| large|
> | 364-11-07| extra large  |
> | 368-11-07| N/A  |
> | 368-11-07| extra large  |
> | 368-11-07| large|
> | 365-11-07| petite   |
> | null  | N/A  |
> | 365-11-07| economy  |
> | 364-11-07| economy  |
> | 364-11-07| petite   |
> | 365-11-07| N/A  |
> | 368-11-07| medium   |
> | null  | extra large  |
> | 368-11-07| small|
> | 368-11-07| petite   |
> | 366-11-08| N/A  |
> +---+--+
> 34 rows selected (0.691 seconds)
> {code}
> However I tried generating the auto-partitioned parquet files using Drill 1.2 
> and then the above query returned the right results.
> I attached the required data sets.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (DRILL-4996) Parquet Date auto-correction is not working in auto-partitioned parquet files generated by drill-1.6

[Vitalii Diravka (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15645729#comment-15645729
 ] 

Vitalii Diravka edited comment on DRILL-4996 at 12/8/16 6:04 PM:
-

Right. Moreover drill-1.6.0 (which generated that file) will show incorrect 
date values too. 
Cause before DRILL-4203 fix drill could not read any correct date values in 
parquet files. 
To see are date values correct, you can use parquet tools.
{code}
vitalii@vitalii-pc:~/parquet-tools/parquet-mr/parquet-tools/target$ java -jar 
parquet-tools-1.6.0rc3-SNAPSHOT.jar meta 
/home/vitalii/Downloads/1.6/0_0_1.parquet
file: file:/home/vitalii/Downloads/1.6/0_0_1.parquet 
creator:  parquet-mr version 1.8.1-drill-r0 (build 
6b605a4ea05b66e1a6bf843353abcb4834a4ced8) 
extra:drill.version = 1.6.0 
file schema:  root 

i_rec_start_date: OPTIONAL INT32 O:DATE R:0 D:1
i_rec_end_date:   OPTIONAL INT32 O:DATE R:0 D:1
...
vitalii@vitalii-pc:~/parquet-tools/parquet-mr/parquet-tools/target$ java -jar 
parquet-tools-1.6.0rc3-SNAPSHOT.jar cat 
/home/vitalii/Downloads/1.6/0_0_1.parquet

i_rec_start_date = 10161
i_rec_end_date = 10891
.
{code}
Incorrect values more than 4881176. 


was (Author: vitalii):
Right. Moreover drill-1.6.0 (which generated that file) will show incorrect 
date values too. 
Cause fefore DRILL-4203 fix drill could not read any correct date values in 
parquet files. 
To see are date values correct, you can use parquet tools.
{code}
vitalii@vitalii-pc:~/parquet-tools/parquet-mr/parquet-tools/target$ java -jar 
parquet-tools-1.6.0rc3-SNAPSHOT.jar meta 
/home/vitalii/Downloads/1.6/0_0_1.parquet
file: file:/home/vitalii/Downloads/1.6/0_0_1.parquet 
creator:  parquet-mr version 1.8.1-drill-r0 (build 
6b605a4ea05b66e1a6bf843353abcb4834a4ced8) 
extra:drill.version = 1.6.0 
file schema:  root 

i_rec_start_date: OPTIONAL INT32 O:DATE R:0 D:1
i_rec_end_date:   OPTIONAL INT32 O:DATE R:0 D:1
...
vitalii@vitalii-pc:~/parquet-tools/parquet-mr/parquet-tools/target$ java -jar 
parquet-tools-1.6.0rc3-SNAPSHOT.jar cat 
/home/vitalii/Downloads/1.6/0_0_1.parquet

i_rec_start_date = 10161
i_rec_end_date = 10891
.
{code}
Incorrect values more on 4881176. 

> Parquet Date auto-correction is not working in auto-partitioned parquet files 
> generated by drill-1.6
> 
>
> Key: DRILL-4996
> URL: https://issues.apache.org/jira/browse/DRILL-4996
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Parquet
>Reporter: Rahul Challapalli
>Assignee: Vitalii Diravka
>Priority: Critical
> Attachments: item.tgz
>
>
> git.commit.id.abbrev=4ee1d4c
> Below are the steps I followed to generate the data :
> {code}
> 1. Generate a parquet file with date column using hive1.2
> 2. Use drill 1.6 to create auto-partitioned parquet files partitioned on the 
> date column
> {code}
> Now the below query returns wrong results :
> {code}
> select i_rec_start_date, i_size from 
> dfs.`/drill/testdata/parquet_date/auto_partition/item_multipart_autorefresh`  
> group by i_rec_start_date, i_size;
> +---+--+
> | i_rec_start_date  |i_size|
> +---+--+
> | null  | large|
> | 366-11-08| extra large  |
> | 366-11-08| medium   |
> | null  | medium   |
> | 366-11-08| petite   |
> | 364-11-07| medium   |
> | null  | petite   |
> | 365-11-07| medium   |
> | 368-11-07| economy  |
> | 365-11-07| large|
> | 365-11-07| small|
> | 366-11-08| small|
> | 365-11-07| extra large  |
> | 364-11-07| N/A  |
> | 366-11-08| economy  |
> | 366-11-08| large|
> | 364-11-07| small|
> | null  | small|
> | 364-11-07| large|
> | 364-11-07| extra large  |
> | 368-11-07| N/A  |
> | 368-11-07| extra large  |
> | 368-11-07| large|
> | 365-11-07| petite   |
> | null  | N/A  |
> | 365-11-07| economy  |
> | 364-11-07| economy  |
> | 364-11-07| petite   |
> | 365-11-07| N/A  |
> | 368-11-07| medium   |
> | null  | extra large  |
> | 368-11-07| small|
> | 368-11-07| petite   |
> | 366-11-08| N/A  |
> 

[jira] [Comment Edited] (DRILL-4996) Parquet Date auto-correction is not working in auto-partitioned parquet files generated by drill-1.6

[Vitalii Diravka (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15645729#comment-15645729
 ] 

Vitalii Diravka edited comment on DRILL-4996 at 12/8/16 6:04 PM:
-

Right. Moreover drill-1.6.0 (which generated that file) will show incorrect 
date values too. 
Cause before DRILL-4203 fix drill could not read any correct date values in 
parquet files. 
To see are date values correct, you can use parquet tools.
{code}
vitalii@vitalii-pc:~/parquet-tools/parquet-mr/parquet-tools/target$ java -jar 
parquet-tools-1.6.0rc3-SNAPSHOT.jar meta 
/home/vitalii/Downloads/1.6/0_0_1.parquet
file: file:/home/vitalii/Downloads/1.6/0_0_1.parquet 
creator:  parquet-mr version 1.8.1-drill-r0 (build 
6b605a4ea05b66e1a6bf843353abcb4834a4ced8) 
extra:drill.version = 1.6.0 
file schema:  root 

i_rec_start_date: OPTIONAL INT32 O:DATE R:0 D:1
i_rec_end_date:   OPTIONAL INT32 O:DATE R:0 D:1
...
vitalii@vitalii-pc:~/parquet-tools/parquet-mr/parquet-tools/target$ java -jar 
parquet-tools-1.6.0rc3-SNAPSHOT.jar cat 
/home/vitalii/Downloads/1.6/0_0_1.parquet

i_rec_start_date = 10161
i_rec_end_date = 10891
.
{code}
Incorrect values are more than 4881176. 


was (Author: vitalii):
Right. Moreover drill-1.6.0 (which generated that file) will show incorrect 
date values too. 
Cause before DRILL-4203 fix drill could not read any correct date values in 
parquet files. 
To see are date values correct, you can use parquet tools.
{code}
vitalii@vitalii-pc:~/parquet-tools/parquet-mr/parquet-tools/target$ java -jar 
parquet-tools-1.6.0rc3-SNAPSHOT.jar meta 
/home/vitalii/Downloads/1.6/0_0_1.parquet
file: file:/home/vitalii/Downloads/1.6/0_0_1.parquet 
creator:  parquet-mr version 1.8.1-drill-r0 (build 
6b605a4ea05b66e1a6bf843353abcb4834a4ced8) 
extra:drill.version = 1.6.0 
file schema:  root 

i_rec_start_date: OPTIONAL INT32 O:DATE R:0 D:1
i_rec_end_date:   OPTIONAL INT32 O:DATE R:0 D:1
...
vitalii@vitalii-pc:~/parquet-tools/parquet-mr/parquet-tools/target$ java -jar 
parquet-tools-1.6.0rc3-SNAPSHOT.jar cat 
/home/vitalii/Downloads/1.6/0_0_1.parquet

i_rec_start_date = 10161
i_rec_end_date = 10891
.
{code}
Incorrect values more than 4881176. 

> Parquet Date auto-correction is not working in auto-partitioned parquet files 
> generated by drill-1.6
> 
>
> Key: DRILL-4996
> URL: https://issues.apache.org/jira/browse/DRILL-4996
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Parquet
>Reporter: Rahul Challapalli
>Assignee: Vitalii Diravka
>Priority: Critical
> Attachments: item.tgz
>
>
> git.commit.id.abbrev=4ee1d4c
> Below are the steps I followed to generate the data :
> {code}
> 1. Generate a parquet file with date column using hive1.2
> 2. Use drill 1.6 to create auto-partitioned parquet files partitioned on the 
> date column
> {code}
> Now the below query returns wrong results :
> {code}
> select i_rec_start_date, i_size from 
> dfs.`/drill/testdata/parquet_date/auto_partition/item_multipart_autorefresh`  
> group by i_rec_start_date, i_size;
> +---+--+
> | i_rec_start_date  |i_size|
> +---+--+
> | null  | large|
> | 366-11-08| extra large  |
> | 366-11-08| medium   |
> | null  | medium   |
> | 366-11-08| petite   |
> | 364-11-07| medium   |
> | null  | petite   |
> | 365-11-07| medium   |
> | 368-11-07| economy  |
> | 365-11-07| large|
> | 365-11-07| small|
> | 366-11-08| small|
> | 365-11-07| extra large  |
> | 364-11-07| N/A  |
> | 366-11-08| economy  |
> | 366-11-08| large|
> | 364-11-07| small|
> | null  | small|
> | 364-11-07| large|
> | 364-11-07| extra large  |
> | 368-11-07| N/A  |
> | 368-11-07| extra large  |
> | 368-11-07| large|
> | 365-11-07| petite   |
> | null  | N/A  |
> | 365-11-07| economy  |
> | 364-11-07| economy  |
> | 364-11-07| petite   |
> | 365-11-07| N/A  |
> | 368-11-07| medium   |
> | null  | extra large  |
> | 368-11-07| small|
> | 368-11-07| petite   |
> | 366-11-08| N/A  |
> 

[jira] [Comment Edited] (DRILL-4996) Parquet Date auto-correction is not working in auto-partitioned parquet files generated by drill-1.6

[Vitalii Diravka (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15645729#comment-15645729
 ] 

Vitalii Diravka edited comment on DRILL-4996 at 12/8/16 6:04 PM:
-

Right. Moreover drill-1.6.0 (which generated that file) will show incorrect 
date values too. 
Cause before DRILL-4203 fix drill could not read any correct date values in 
parquet files. 
To see are date values correct, you can use parquet tools.
{code}
vitalii@vitalii-pc:~/parquet-tools/parquet-mr/parquet-tools/target$ java -jar 
parquet-tools-1.6.0rc3-SNAPSHOT.jar meta 
/home/vitalii/Downloads/1.6/0_0_1.parquet
file: file:/home/vitalii/Downloads/1.6/0_0_1.parquet 
creator:  parquet-mr version 1.8.1-drill-r0 (build 
6b605a4ea05b66e1a6bf843353abcb4834a4ced8) 
extra:drill.version = 1.6.0 
file schema:  root 

i_rec_start_date: OPTIONAL INT32 O:DATE R:0 D:1
i_rec_end_date:   OPTIONAL INT32 O:DATE R:0 D:1
...
vitalii@vitalii-pc:~/parquet-tools/parquet-mr/parquet-tools/target$ java -jar 
parquet-tools-1.6.0rc3-SNAPSHOT.jar cat 
/home/vitalii/Downloads/1.6/0_0_1.parquet

i_rec_start_date = 10161
i_rec_end_date = 10891
.
{code}
Incorrect values are more than 4881176. 


was (Author: vitalii):
Right. Moreover drill-1.6.0 (which generated that file) will show incorrect 
date values too. 
Cause before DRILL-4203 fix drill could not read any correct date values in 
parquet files. 
To see are date values correct, you can use parquet tools.
{code}
vitalii@vitalii-pc:~/parquet-tools/parquet-mr/parquet-tools/target$ java -jar 
parquet-tools-1.6.0rc3-SNAPSHOT.jar meta 
/home/vitalii/Downloads/1.6/0_0_1.parquet
file: file:/home/vitalii/Downloads/1.6/0_0_1.parquet 
creator:  parquet-mr version 1.8.1-drill-r0 (build 
6b605a4ea05b66e1a6bf843353abcb4834a4ced8) 
extra:drill.version = 1.6.0 
file schema:  root 

i_rec_start_date: OPTIONAL INT32 O:DATE R:0 D:1
i_rec_end_date:   OPTIONAL INT32 O:DATE R:0 D:1
...
vitalii@vitalii-pc:~/parquet-tools/parquet-mr/parquet-tools/target$ java -jar 
parquet-tools-1.6.0rc3-SNAPSHOT.jar cat 
/home/vitalii/Downloads/1.6/0_0_1.parquet

i_rec_start_date = 10161
i_rec_end_date = 10891
.
{code}
Incorrect values are more than 4881176. 

> Parquet Date auto-correction is not working in auto-partitioned parquet files 
> generated by drill-1.6
> 
>
> Key: DRILL-4996
> URL: https://issues.apache.org/jira/browse/DRILL-4996
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Parquet
>Reporter: Rahul Challapalli
>Assignee: Vitalii Diravka
>Priority: Critical
> Attachments: item.tgz
>
>
> git.commit.id.abbrev=4ee1d4c
> Below are the steps I followed to generate the data :
> {code}
> 1. Generate a parquet file with date column using hive1.2
> 2. Use drill 1.6 to create auto-partitioned parquet files partitioned on the 
> date column
> {code}
> Now the below query returns wrong results :
> {code}
> select i_rec_start_date, i_size from 
> dfs.`/drill/testdata/parquet_date/auto_partition/item_multipart_autorefresh`  
> group by i_rec_start_date, i_size;
> +---+--+
> | i_rec_start_date  |i_size|
> +---+--+
> | null  | large|
> | 366-11-08| extra large  |
> | 366-11-08| medium   |
> | null  | medium   |
> | 366-11-08| petite   |
> | 364-11-07| medium   |
> | null  | petite   |
> | 365-11-07| medium   |
> | 368-11-07| economy  |
> | 365-11-07| large|
> | 365-11-07| small|
> | 366-11-08| small|
> | 365-11-07| extra large  |
> | 364-11-07| N/A  |
> | 366-11-08| economy  |
> | 366-11-08| large|
> | 364-11-07| small|
> | null  | small|
> | 364-11-07| large|
> | 364-11-07| extra large  |
> | 368-11-07| N/A  |
> | 368-11-07| extra large  |
> | 368-11-07| large|
> | 365-11-07| petite   |
> | null  | N/A  |
> | 365-11-07| economy  |
> | 364-11-07| economy  |
> | 364-11-07| petite   |
> | 365-11-07| N/A  |
> | 368-11-07| medium   |
> | null  | extra large  |
> | 368-11-07| small|
> | 368-11-07| petite   |
> | 366-11-08| N/A  |
> 

[jira] [Assigned] (DRILL-5002) Using hive's date functions on top of date column in parquet gives wrong results

[Zelaine Fong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zelaine Fong reassigned DRILL-5002:
---

Assignee: Vitalii Diravka

> Using hive's date functions on top of date column in parquet gives wrong 
> results
> 
>
> Key: DRILL-5002
> URL: https://issues.apache.org/jira/browse/DRILL-5002
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Hive, Storage - Parquet
>Reporter: Rahul Challapalli
>Assignee: Vitalii Diravka
>Priority: Critical
>
> git.commit.id.abbrev=190d5d4
> Wrong Result 1 :
> {code}
> select l_shipdate, `month`(l_shipdate) from cp.`tpch/lineitem.parquet` where 
> l_shipdate = date '1994-02-01' limit 2;
> +-+-+
> | l_shipdate  | EXPR$1  |
> +-+-+
> | 1994-02-01  | 1   |
> | 1994-02-01  | 1   |
> +-+-+
> {code}
> Wrong Result 2 : 
> {code}
> select l_shipdate, `day`(l_shipdate) from cp.`tpch/lineitem.parquet` where 
> l_shipdate = date '1998-06-02' limit 2;
> +-+-+
> | l_shipdate  | EXPR$1  |
> +-+-+
> | 1998-06-02  | 1   |
> | 1998-06-02  | 1   |
> +-+-+
> {code}
> Correct Result :
> {code}
> select l_shipdate, `month`(l_shipdate) from cp.`tpch/lineitem.parquet` where 
> l_shipdate = date '1998-06-02' limit 2;
> +-+-+
> | l_shipdate  | EXPR$1  |
> +-+-+
> | 1998-06-02  | 6   |
> | 1998-06-02  | 6   |
> +-+-+
> {code}
> It looks like we are getting wrong results when the 'day' is '01'. I only 
> tried month and day hive functionsbut wouldn't be surprised if they have 
> similar issues too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (DRILL-4996) Parquet Date auto-correction is not working in auto-partitioned parquet files generated by drill-1.6

[Zelaine Fong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zelaine Fong reassigned DRILL-4996:
---

Assignee: Vitalii Diravka

> Parquet Date auto-correction is not working in auto-partitioned parquet files 
> generated by drill-1.6
> 
>
> Key: DRILL-4996
> URL: https://issues.apache.org/jira/browse/DRILL-4996
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Parquet
>Reporter: Rahul Challapalli
>Assignee: Vitalii Diravka
>Priority: Critical
> Attachments: item.tgz
>
>
> git.commit.id.abbrev=4ee1d4c
> Below are the steps I followed to generate the data :
> {code}
> 1. Generate a parquet file with date column using hive1.2
> 2. Use drill 1.6 to create auto-partitioned parquet files partitioned on the 
> date column
> {code}
> Now the below query returns wrong results :
> {code}
> select i_rec_start_date, i_size from 
> dfs.`/drill/testdata/parquet_date/auto_partition/item_multipart_autorefresh`  
> group by i_rec_start_date, i_size;
> +---+--+
> | i_rec_start_date  |i_size|
> +---+--+
> | null  | large|
> | 366-11-08| extra large  |
> | 366-11-08| medium   |
> | null  | medium   |
> | 366-11-08| petite   |
> | 364-11-07| medium   |
> | null  | petite   |
> | 365-11-07| medium   |
> | 368-11-07| economy  |
> | 365-11-07| large|
> | 365-11-07| small|
> | 366-11-08| small|
> | 365-11-07| extra large  |
> | 364-11-07| N/A  |
> | 366-11-08| economy  |
> | 366-11-08| large|
> | 364-11-07| small|
> | null  | small|
> | 364-11-07| large|
> | 364-11-07| extra large  |
> | 368-11-07| N/A  |
> | 368-11-07| extra large  |
> | 368-11-07| large|
> | 365-11-07| petite   |
> | null  | N/A  |
> | 365-11-07| economy  |
> | 364-11-07| economy  |
> | 364-11-07| petite   |
> | 365-11-07| N/A  |
> | 368-11-07| medium   |
> | null  | extra large  |
> | 368-11-07| small|
> | 368-11-07| petite   |
> | 366-11-08| N/A  |
> +---+--+
> 34 rows selected (0.691 seconds)
> {code}
> However I tried generating the auto-partitioned parquet files using Drill 1.2 
> and then the above query returned the right results.
> I attached the required data sets.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-5070) Code gen: create methods in fixed order to allow test verification

[Zelaine Fong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zelaine Fong updated DRILL-5070:

Assignee: Paul Rogers
Reviewer: Karthikeyan Manivannan

Assigned Reviewer to [~karthikm]

> Code gen: create methods in fixed order to allow test verification
> --
>
> Key: DRILL-5070
> URL: https://issues.apache.org/jira/browse/DRILL-5070
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> A handy technique in testing is to compare generated code against a "golden" 
> copy that defines the expected results. However, at present, Drill generates 
> code using the method order returned by {{Class.getDeclaredMethods}}, but 
> this method makes no guarantee about the order of the methods. The order 
> varies from one run to the next. There is some evidence [this 
> link|http://stackoverflow.com/questions/28585843/java-reflection-getdeclaredmethods-in-declared-order-strange-behaviour]
>  that order can vary even within a single run, though a quick test was unable 
> to reproduce this case.
> If method order does indeed vary within a single run, then the order can 
> impact the Drill code cache since it compares the sources from two different 
> generation events to detect duplicate code.
> This issue appeared when attempting to modify tests to capture generated code 
> for comparison to future results. Even a simple generated case from 
> {{ExpressionTest.testBasicExpression()}} that generates {{if(true) then 1 
> else 0 end}} (all constants) produced methods in different orders on each 
> test run.
> The fix is simple, in the {{SignatureHolder}} constructor, sort methods by 
> name after retrieving them from the class. The sort ensures that method order 
> is deterministic. Fortunately, the number of methods is small, so the sort 
> step adds little cost.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5052) Option to debug generated Java code using an IDE

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15732512#comment-15732512
 ] 

ASF GitHub Bot commented on DRILL-5052:
---

Github user arina-ielchiieva commented on the issue:

https://github.com/apache/drill/pull/660
  
Paul, it's really good feature! I have left some comments in PR.
Other than that:
1. Please format the code to remove unnecessary spaces.
Examples:
addClass( key, results.get( key ) );
public CachedClassLoader( ) {
codeDir = new File( config.getString( CODE_DIR_OPTION ) );

2. May we can add plain-old Java support for Projector in this PR, it's the 
easiest template to start with.


> Option to debug generated Java code using an IDE
> 
>
> Key: DRILL-5052
> URL: https://issues.apache.org/jira/browse/DRILL-5052
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Codegen
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> Drill makes extensive use of Java code generation to implement its operators. 
> Drill uses sophisticated techniques to blend generated code with pre-compiled 
> template code. An unfortunate side-effect of this behavior is that it is very 
> difficult to visualize and debug the generated code.
> As it turns out, Drill's code-merge facility is, in essence, a do-it-yourself 
> version of subclassing. The Drill "template" is the parent class, the 
> generated code is the subclass. But, rather than using plain-old subclassing, 
> Drill combines the code from the two classes into a single "artificial" 
> packet of byte codes for which no source exists.
> Modify the code generation path to optionally allow "plain-old Java" 
> compilation: the generated code is a subclass of the template. Compile the 
> generated code as a plain-old Java class with no byte-code fix-up. Write the 
> code to a known location that the IDE can search when looking for source 
> files.
> With this change, developers can turn on the above feature, set a breakpoint 
> in a template, then step directly into the generated Java code called from 
> the template.
> This feature should be an option, enabled by developers when needed. The 
> existing byte-code technique should be used for production code generation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5052) Option to debug generated Java code using an IDE

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15732506#comment-15732506
 ] 

ASF GitHub Bot commented on DRILL-5052:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/660#discussion_r91533256
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/compile/ClassBuilder.java ---
@@ -0,0 +1,190 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.compile;
+
+import java.io.File;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.util.Map;
+
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.exec.compile.ClassTransformer.ClassNames;
+import org.apache.drill.exec.exception.ClassTransformationException;
+import org.apache.drill.exec.expr.CodeGenerator;
+import org.apache.drill.exec.server.options.OptionManager;
+import org.codehaus.commons.compiler.CompileException;
+
+/**
+ * Implements the "plain-old Java" method of code generation and
+ * compilation. Given a {@link CodeGenerator}, obtains the generated
+ * source code, compiles it with the selected compiler, loads the
+ * byte-codes into a class loader and provides the resulting
+ * class. Compared with the {@link ClassTransformer} mechanism,
+ * this one requires the code generator to have generated a complete
+ * Java class that is capable of direct compilation and loading.
+ * This means the generated class must be a subclass of the template
+ * so that the JVM can use normal Java inheritance to associate the
+ * template and generated methods.
+ * 
+ * Here is how to use the plain-old Java technique to debug
+ * generated code:
+ * 
+ * Set the config option drill.exec.compile.save_source
+ * to true.
+ * Set the config option drill.exec.compile.code_dir
+ * to the location where you want to save the generated source
+ * code.
+ * Where you generate code (using a {@link CodeGenerator}),
+ * set the "plain-old Java" options:
+ * CodeGeneratorFoo> cg = ...
+ * cg.plainOldJavaCapable(true); // Class supports plain-old Java
+ * cg.preferPlainOldJava(true); // Actually generate plain-old Java
+ * ...
+ * In your favorite IDE, add to the code lookup path the
+ * code directory saved earlier. In Eclipse, for example, you do
+ * this in the debug configuration you will use to debug Drill.
+ * Set a breakpoint in template used for the generated code.
+ * Run Drill. The IDE will stop at your breakpoint.
+ * Step into the generated code. Examine class field and
+ * local variables. Have fun!
+ * 
+ * 
+ * Note: not all generated code is ready to be compiled as plain-old
+ * Java. Some classes omit from the template the proper throws
+ * declarations. Other minor problems may also crop up. All are easy
+ * to fix. Once you've done so, add the following to mark that you've
+ * done the clean-up:
--- End diff --

May be to create Jira to add this support?


> Option to debug generated Java code using an IDE
> 
>
> Key: DRILL-5052
> URL: https://issues.apache.org/jira/browse/DRILL-5052
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Codegen
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> Drill makes extensive use of Java code generation to implement its operators. 
> Drill uses sophisticated techniques to blend generated code with pre-compiled 
> template code. An unfortunate side-effect of this behavior is that it is very 
> difficult to visualize and debug the generated code.
> As it turns out, Drill's code-merge facility is, in essence, a do-it-yourself 
> version of subclassing. The Drill "template" is the parent class, the 
> 

[jira] [Commented] (DRILL-5052) Option to debug generated Java code using an IDE

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15732483#comment-15732483
 ] 

ASF GitHub Bot commented on DRILL-5052:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/660#discussion_r91273747
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/compile/CachedClassLoader.java
 ---
@@ -0,0 +1,70 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.compile;
+
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.Map;
+import java.util.concurrent.ConcurrentMap;
+
+import com.google.common.collect.MapMaker;
+
+/**
+ * Class loader for "plain-old Java" generated classes.
+ * Very basic implementation: allows defining a class from
+ * byte codes and finding the loaded classes. Delegates
+ * all other class requests to the thread context class
+ * loader. This structure ensures that a generated class can
+ * find both its own inner classes as well as all the standard
+ * Drill implementation classes.
+ */
+
+public class CachedClassLoader extends URLClassLoader {
+
+  /**
+   * Cache of generated classes. Semantics: a single thread defines
+   * the classes, many threads may access the classes.
+   */
+
+  private ConcurrentMap cache = new 
MapMaker().concurrencyLevel(4).makeMap();
--- End diff --

1. Please add final
2. Please replace new MapMaker().concurrencyLevel(4).makeMap() -> 
Maps.newConcurrentMap();
newConcurrentMap() contains the same code.


> Option to debug generated Java code using an IDE
> 
>
> Key: DRILL-5052
> URL: https://issues.apache.org/jira/browse/DRILL-5052
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Codegen
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> Drill makes extensive use of Java code generation to implement its operators. 
> Drill uses sophisticated techniques to blend generated code with pre-compiled 
> template code. An unfortunate side-effect of this behavior is that it is very 
> difficult to visualize and debug the generated code.
> As it turns out, Drill's code-merge facility is, in essence, a do-it-yourself 
> version of subclassing. The Drill "template" is the parent class, the 
> generated code is the subclass. But, rather than using plain-old subclassing, 
> Drill combines the code from the two classes into a single "artificial" 
> packet of byte codes for which no source exists.
> Modify the code generation path to optionally allow "plain-old Java" 
> compilation: the generated code is a subclass of the template. Compile the 
> generated code as a plain-old Java class with no byte-code fix-up. Write the 
> code to a known location that the IDE can search when looking for source 
> files.
> With this change, developers can turn on the above feature, set a breakpoint 
> in a template, then step directly into the generated Java code called from 
> the template.
> This feature should be an option, enabled by developers when needed. The 
> existing byte-code technique should be used for production code generation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5052) Option to debug generated Java code using an IDE

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15732505#comment-15732505
 ] 

ASF GitHub Bot commented on DRILL-5052:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/660#discussion_r91531972
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/compile/CodeCompiler.java ---
@@ -33,36 +32,71 @@
 import com.google.common.cache.LoadingCache;
 import com.google.common.collect.Lists;
 
+/**
+ * Global code compiler mechanism shared by all threads and operators.
+ * Holds a single cache of generated code (keyed by code source) to
+ * prevent compiling identical code multiple times. Supports both
+ * the byte-code merging and plain-old Java methods of code
+ * generation and compilation.
+ */
+
 public class CodeCompiler {
-//  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(CodeCompiler.class);
+
+  public static final String COMPILE_BASE = "drill.exec.compile";
+  String MAX_LOADING_CACHE_SIZE_CONFIG = COMPILE_BASE + ".cache_max_size";
 
   private final ClassTransformer transformer;
+  private final ClassBuilder classBuilder;
+
+  /**
+   * Google Guava loading cache that defers creating a cache
+   * entry until first needed. Creation is done in a thread-safe
+   * way: if two threads try to create the same class at the same
+   * time, the first does the work, the second waits for the first
+   * to complete, then grabs the new entry.
+   */
+
   private final LoadingCache cache;
-  private final DrillConfig config;
-  private final OptionManager optionManager;
 
   public CodeCompiler(final DrillConfig config, final OptionManager 
optionManager) {
-transformer = new ClassTransformer(optionManager);
-final int cacheMaxSize = 
config.getInt(ExecConstants.MAX_LOADING_CACHE_SIZE_CONFIG);
+transformer = new ClassTransformer(config, optionManager);
+classBuilder = new ClassBuilder(config, optionManager);
+final int cacheMaxSize = config.getInt(MAX_LOADING_CACHE_SIZE_CONFIG);
 cache = CacheBuilder.newBuilder()
 .maximumSize(cacheMaxSize)
 .build(new Loader());
-this.optionManager = optionManager;
-this.config = config;
   }
 
+  /**
+   * Create a single instance of the generated class.
+   *
+   * @param cg
+   * @return
+   * @throws ClassTransformationException
+   * @throws IOException
+   */
--- End diff --

Description for param, return and exceptions?


> Option to debug generated Java code using an IDE
> 
>
> Key: DRILL-5052
> URL: https://issues.apache.org/jira/browse/DRILL-5052
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Codegen
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> Drill makes extensive use of Java code generation to implement its operators. 
> Drill uses sophisticated techniques to blend generated code with pre-compiled 
> template code. An unfortunate side-effect of this behavior is that it is very 
> difficult to visualize and debug the generated code.
> As it turns out, Drill's code-merge facility is, in essence, a do-it-yourself 
> version of subclassing. The Drill "template" is the parent class, the 
> generated code is the subclass. But, rather than using plain-old subclassing, 
> Drill combines the code from the two classes into a single "artificial" 
> packet of byte codes for which no source exists.
> Modify the code generation path to optionally allow "plain-old Java" 
> compilation: the generated code is a subclass of the template. Compile the 
> generated code as a plain-old Java class with no byte-code fix-up. Write the 
> code to a known location that the IDE can search when looking for source 
> files.
> With this change, developers can turn on the above feature, set a breakpoint 
> in a template, then step directly into the generated Java code called from 
> the template.
> This feature should be an option, enabled by developers when needed. The 
> existing byte-code technique should be used for production code generation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5052) Option to debug generated Java code using an IDE

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15732492#comment-15732492
 ] 

ASF GitHub Bot commented on DRILL-5052:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/660#discussion_r91322821
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/compile/TemplateClassDefinition.java
 ---
@@ -63,6 +78,14 @@ public SignatureHolder getSignature(){
 
   @Override
   public String toString() {
-return "TemplateClassDefinition [template=" + template + ", 
signature=" + signature + "]";
+StringBuilder buf = new StringBuilder( );
+buf.append( "TemplateClassDefinition [interface=" );
+buf.append( (iface == null) ? "null" : iface.getName() );
+buf.append( ", template=" );
+buf.append( (template == null) ? "null" : template.getName() );
+buf.append( ", signature=\n" );
--- End diff --

May be it's better to start new line with variable name rather then with 
its value?


> Option to debug generated Java code using an IDE
> 
>
> Key: DRILL-5052
> URL: https://issues.apache.org/jira/browse/DRILL-5052
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Codegen
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> Drill makes extensive use of Java code generation to implement its operators. 
> Drill uses sophisticated techniques to blend generated code with pre-compiled 
> template code. An unfortunate side-effect of this behavior is that it is very 
> difficult to visualize and debug the generated code.
> As it turns out, Drill's code-merge facility is, in essence, a do-it-yourself 
> version of subclassing. The Drill "template" is the parent class, the 
> generated code is the subclass. But, rather than using plain-old subclassing, 
> Drill combines the code from the two classes into a single "artificial" 
> packet of byte codes for which no source exists.
> Modify the code generation path to optionally allow "plain-old Java" 
> compilation: the generated code is a subclass of the template. Compile the 
> generated code as a plain-old Java class with no byte-code fix-up. Write the 
> code to a known location that the IDE can search when looking for source 
> files.
> With this change, developers can turn on the above feature, set a breakpoint 
> in a template, then step directly into the generated Java code called from 
> the template.
> This feature should be an option, enabled by developers when needed. The 
> existing byte-code technique should be used for production code generation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5052) Option to debug generated Java code using an IDE

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15732488#comment-15732488
 ] 

ASF GitHub Bot commented on DRILL-5052:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/660#discussion_r91300674
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/xsort/ExternalSortBatch.java
 ---
@@ -711,18 +711,20 @@ private MSorter createNewMSorter(FragmentContext 
context, List orderin
 g.rotateBlock();
 g.getEvalBlock()._return(JExpr.lit(0));
 
+cg.plainOldJavaCapable(true); // This class can generate plain-old 
Java.
+ // Uncomment out this line to debug the generated code.
+//  cg.preferPlainOldJavaJava(true);
--- End diff --

preferPlainOldJavaJava -> incorrect method name


> Option to debug generated Java code using an IDE
> 
>
> Key: DRILL-5052
> URL: https://issues.apache.org/jira/browse/DRILL-5052
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Codegen
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> Drill makes extensive use of Java code generation to implement its operators. 
> Drill uses sophisticated techniques to blend generated code with pre-compiled 
> template code. An unfortunate side-effect of this behavior is that it is very 
> difficult to visualize and debug the generated code.
> As it turns out, Drill's code-merge facility is, in essence, a do-it-yourself 
> version of subclassing. The Drill "template" is the parent class, the 
> generated code is the subclass. But, rather than using plain-old subclassing, 
> Drill combines the code from the two classes into a single "artificial" 
> packet of byte codes for which no source exists.
> Modify the code generation path to optionally allow "plain-old Java" 
> compilation: the generated code is a subclass of the template. Compile the 
> generated code as a plain-old Java class with no byte-code fix-up. Write the 
> code to a known location that the IDE can search when looking for source 
> files.
> With this change, developers can turn on the above feature, set a breakpoint 
> in a template, then step directly into the generated Java code called from 
> the template.
> This feature should be an option, enabled by developers when needed. The 
> existing byte-code technique should be used for production code generation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5052) Option to debug generated Java code using an IDE

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15732498#comment-15732498
 ] 

ASF GitHub Bot commented on DRILL-5052:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/660#discussion_r91275047
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/compile/CachedClassLoader.java
 ---
@@ -0,0 +1,70 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.compile;
+
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.Map;
+import java.util.concurrent.ConcurrentMap;
+
+import com.google.common.collect.MapMaker;
+
+/**
+ * Class loader for "plain-old Java" generated classes.
+ * Very basic implementation: allows defining a class from
+ * byte codes and finding the loaded classes. Delegates
+ * all other class requests to the thread context class
+ * loader. This structure ensures that a generated class can
+ * find both its own inner classes as well as all the standard
+ * Drill implementation classes.
+ */
+
+public class CachedClassLoader extends URLClassLoader {
+
+  /**
+   * Cache of generated classes. Semantics: a single thread defines
+   * the classes, many threads may access the classes.
+   */
+
+  private ConcurrentMap cache = new 
MapMaker().concurrencyLevel(4).makeMap();
+
+  public CachedClassLoader( ) {
+super(new URL[0], Thread.currentThread().getContextClassLoader());
+  }
+
+  public void addClass( String fqcn, byte[] byteCodes ) {
+
+assert ! cache.containsKey( fqcn );
+Class newClass = defineClass(fqcn, byteCodes, 0, byteCodes.length);
+cache.put( fqcn, newClass );
+  }
+
+  @Override
+  public Class findClass(String className) throws 
ClassNotFoundException {
+Class theClass = cache.get( className );
+if ( theClass != null ) {
+  return theClass; }
--- End diff --

Move curly brace to the next line.


> Option to debug generated Java code using an IDE
> 
>
> Key: DRILL-5052
> URL: https://issues.apache.org/jira/browse/DRILL-5052
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Codegen
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> Drill makes extensive use of Java code generation to implement its operators. 
> Drill uses sophisticated techniques to blend generated code with pre-compiled 
> template code. An unfortunate side-effect of this behavior is that it is very 
> difficult to visualize and debug the generated code.
> As it turns out, Drill's code-merge facility is, in essence, a do-it-yourself 
> version of subclassing. The Drill "template" is the parent class, the 
> generated code is the subclass. But, rather than using plain-old subclassing, 
> Drill combines the code from the two classes into a single "artificial" 
> packet of byte codes for which no source exists.
> Modify the code generation path to optionally allow "plain-old Java" 
> compilation: the generated code is a subclass of the template. Compile the 
> generated code as a plain-old Java class with no byte-code fix-up. Write the 
> code to a known location that the IDE can search when looking for source 
> files.
> With this change, developers can turn on the above feature, set a breakpoint 
> in a template, then step directly into the generated Java code called from 
> the template.
> This feature should be an option, enabled by developers when needed. The 
> existing byte-code technique should be used for production code generation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5052) Option to debug generated Java code using an IDE

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15732481#comment-15732481
 ] 

ASF GitHub Bot commented on DRILL-5052:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/660#discussion_r91278330
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/DebugStringBuilder.java 
---
@@ -0,0 +1,57 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.expr;
+
+import java.io.PrintWriter;
+import java.io.StringWriter;
+
+import com.sun.codemodel.JFormatter;
+
+/**
+ * Utility class to build a debug string for an object
+ * in a standard format.
+ */
+
+public class DebugStringBuilder {
+
+  private StringWriter strWriter;
+  public PrintWriter writer;
+  public JFormatter fmt;
+
--- End diff --

1. Make variables private, and use getter when access is needed.
2.  All variables can be made final.


> Option to debug generated Java code using an IDE
> 
>
> Key: DRILL-5052
> URL: https://issues.apache.org/jira/browse/DRILL-5052
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Codegen
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> Drill makes extensive use of Java code generation to implement its operators. 
> Drill uses sophisticated techniques to blend generated code with pre-compiled 
> template code. An unfortunate side-effect of this behavior is that it is very 
> difficult to visualize and debug the generated code.
> As it turns out, Drill's code-merge facility is, in essence, a do-it-yourself 
> version of subclassing. The Drill "template" is the parent class, the 
> generated code is the subclass. But, rather than using plain-old subclassing, 
> Drill combines the code from the two classes into a single "artificial" 
> packet of byte codes for which no source exists.
> Modify the code generation path to optionally allow "plain-old Java" 
> compilation: the generated code is a subclass of the template. Compile the 
> generated code as a plain-old Java class with no byte-code fix-up. Write the 
> code to a known location that the IDE can search when looking for source 
> files.
> With this change, developers can turn on the above feature, set a breakpoint 
> in a template, then step directly into the generated Java code called from 
> the template.
> This feature should be an option, enabled by developers when needed. The 
> existing byte-code technique should be used for production code generation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5052) Option to debug generated Java code using an IDE

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15732487#comment-15732487
 ] 

ASF GitHub Bot commented on DRILL-5052:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/660#discussion_r91276697
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/compile/CachedClassLoader.java
 ---
@@ -0,0 +1,70 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.compile;
+
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.Map;
+import java.util.concurrent.ConcurrentMap;
+
+import com.google.common.collect.MapMaker;
+
+/**
+ * Class loader for "plain-old Java" generated classes.
+ * Very basic implementation: allows defining a class from
+ * byte codes and finding the loaded classes. Delegates
+ * all other class requests to the thread context class
+ * loader. This structure ensures that a generated class can
+ * find both its own inner classes as well as all the standard
+ * Drill implementation classes.
+ */
+
+public class CachedClassLoader extends URLClassLoader {
+
+  /**
+   * Cache of generated classes. Semantics: a single thread defines
+   * the classes, many threads may access the classes.
+   */
+
+  private ConcurrentMap cache = new 
MapMaker().concurrencyLevel(4).makeMap();
+
+  public CachedClassLoader( ) {
+super(new URL[0], Thread.currentThread().getContextClassLoader());
+  }
+
+  public void addClass( String fqcn, byte[] byteCodes ) {
+
+assert ! cache.containsKey( fqcn );
+Class newClass = defineClass(fqcn, byteCodes, 0, byteCodes.length);
+cache.put( fqcn, newClass );
+  }
+
+  @Override
+  public Class findClass(String className) throws 
ClassNotFoundException {
+Class theClass = cache.get( className );
+if ( theClass != null ) {
+  return theClass; }
+return super.findClass(className);
+  }
+
+  public void addClasses(Map results) {
+for ( String key : results.keySet() ) {
+  addClass( key, results.get( key ) );
+}
--- End diff --

Using iteration over map via entry set is more efficient than via key set:
for (Map.Entry result : results.entrySet()) {
  addClass(result.getKey(), result.getValue());
}


> Option to debug generated Java code using an IDE
> 
>
> Key: DRILL-5052
> URL: https://issues.apache.org/jira/browse/DRILL-5052
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Codegen
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> Drill makes extensive use of Java code generation to implement its operators. 
> Drill uses sophisticated techniques to blend generated code with pre-compiled 
> template code. An unfortunate side-effect of this behavior is that it is very 
> difficult to visualize and debug the generated code.
> As it turns out, Drill's code-merge facility is, in essence, a do-it-yourself 
> version of subclassing. The Drill "template" is the parent class, the 
> generated code is the subclass. But, rather than using plain-old subclassing, 
> Drill combines the code from the two classes into a single "artificial" 
> packet of byte codes for which no source exists.
> Modify the code generation path to optionally allow "plain-old Java" 
> compilation: the generated code is a subclass of the template. Compile the 
> generated code as a plain-old Java class with no byte-code fix-up. Write the 
> code to a known location that the IDE can search when looking for source 
> files.
> With this change, developers can turn on the above feature, set a breakpoint 
> in a template, then 

[jira] [Commented] (DRILL-5052) Option to debug generated Java code using an IDE

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15732482#comment-15732482
 ] 

ASF GitHub Bot commented on DRILL-5052:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/660#discussion_r91509351
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/compile/MergeAdapter.java ---
@@ -253,7 +257,13 @@ public static MergedClassResult getMergedClass(final 
ClassSet set, final byte[]
   }
 
   // enable when you want all the generated merged class files to also 
be written to disk.
-//  Files.write(outputClass, new 
File(String.format("/src/scratch/drill-generated-classes/%s-output.class", 
set.generated.dot)));
+//  try {
--- End diff --

1. May also to add some configuration option to avoid uncommenting?
2. May be to avoid hard-coding "/tmp/scratch/drill-generated-classes" and 
make it configurable?


> Option to debug generated Java code using an IDE
> 
>
> Key: DRILL-5052
> URL: https://issues.apache.org/jira/browse/DRILL-5052
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Codegen
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> Drill makes extensive use of Java code generation to implement its operators. 
> Drill uses sophisticated techniques to blend generated code with pre-compiled 
> template code. An unfortunate side-effect of this behavior is that it is very 
> difficult to visualize and debug the generated code.
> As it turns out, Drill's code-merge facility is, in essence, a do-it-yourself 
> version of subclassing. The Drill "template" is the parent class, the 
> generated code is the subclass. But, rather than using plain-old subclassing, 
> Drill combines the code from the two classes into a single "artificial" 
> packet of byte codes for which no source exists.
> Modify the code generation path to optionally allow "plain-old Java" 
> compilation: the generated code is a subclass of the template. Compile the 
> generated code as a plain-old Java class with no byte-code fix-up. Write the 
> code to a known location that the IDE can search when looking for source 
> files.
> With this change, developers can turn on the above feature, set a breakpoint 
> in a template, then step directly into the generated Java code called from 
> the template.
> This feature should be an option, enabled by developers when needed. The 
> existing byte-code technique should be used for production code generation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5052) Option to debug generated Java code using an IDE

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15732503#comment-15732503
 ] 

ASF GitHub Bot commented on DRILL-5052:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/660#discussion_r91521257
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/compile/DrillJavaFileObject.java
 ---
@@ -67,6 +72,14 @@ public boolean isCompiled() {
 }
   }
 
+  public Map getResults( ) {
--- End diff --

May be more user-friendly method name? Results seems to be a little vague.


> Option to debug generated Java code using an IDE
> 
>
> Key: DRILL-5052
> URL: https://issues.apache.org/jira/browse/DRILL-5052
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Codegen
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> Drill makes extensive use of Java code generation to implement its operators. 
> Drill uses sophisticated techniques to blend generated code with pre-compiled 
> template code. An unfortunate side-effect of this behavior is that it is very 
> difficult to visualize and debug the generated code.
> As it turns out, Drill's code-merge facility is, in essence, a do-it-yourself 
> version of subclassing. The Drill "template" is the parent class, the 
> generated code is the subclass. But, rather than using plain-old subclassing, 
> Drill combines the code from the two classes into a single "artificial" 
> packet of byte codes for which no source exists.
> Modify the code generation path to optionally allow "plain-old Java" 
> compilation: the generated code is a subclass of the template. Compile the 
> generated code as a plain-old Java class with no byte-code fix-up. Write the 
> code to a known location that the IDE can search when looking for source 
> files.
> With this change, developers can turn on the above feature, set a breakpoint 
> in a template, then step directly into the generated Java code called from 
> the template.
> This feature should be an option, enabled by developers when needed. The 
> existing byte-code technique should be used for production code generation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5052) Option to debug generated Java code using an IDE

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15732500#comment-15732500
 ] 

ASF GitHub Bot commented on DRILL-5052:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/660#discussion_r91513389
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/compile/DrillJavaFileObject.java
 ---
@@ -67,6 +72,14 @@ public boolean isCompiled() {
 }
   }
 
+  public Map getResults( ) {
+Map results = new HashMap<>( );
--- End diff --

I've noticed that mostly in Drill Guava lib is used for collection 
creation: Maps.newHashMap();


> Option to debug generated Java code using an IDE
> 
>
> Key: DRILL-5052
> URL: https://issues.apache.org/jira/browse/DRILL-5052
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Codegen
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> Drill makes extensive use of Java code generation to implement its operators. 
> Drill uses sophisticated techniques to blend generated code with pre-compiled 
> template code. An unfortunate side-effect of this behavior is that it is very 
> difficult to visualize and debug the generated code.
> As it turns out, Drill's code-merge facility is, in essence, a do-it-yourself 
> version of subclassing. The Drill "template" is the parent class, the 
> generated code is the subclass. But, rather than using plain-old subclassing, 
> Drill combines the code from the two classes into a single "artificial" 
> packet of byte codes for which no source exists.
> Modify the code generation path to optionally allow "plain-old Java" 
> compilation: the generated code is a subclass of the template. Compile the 
> generated code as a plain-old Java class with no byte-code fix-up. Write the 
> code to a known location that the IDE can search when looking for source 
> files.
> With this change, developers can turn on the above feature, set a breakpoint 
> in a template, then step directly into the generated Java code called from 
> the template.
> This feature should be an option, enabled by developers when needed. The 
> existing byte-code technique should be used for production code generation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5052) Option to debug generated Java code using an IDE

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15732504#comment-15732504
 ] 

ASF GitHub Bot commented on DRILL-5052:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/660#discussion_r91531809
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/compile/ClassBuilder.java ---
@@ -0,0 +1,190 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.compile;
+
+import java.io.File;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.util.Map;
+
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.exec.compile.ClassTransformer.ClassNames;
+import org.apache.drill.exec.exception.ClassTransformationException;
+import org.apache.drill.exec.expr.CodeGenerator;
+import org.apache.drill.exec.server.options.OptionManager;
+import org.codehaus.commons.compiler.CompileException;
+
+/**
+ * Implements the "plain-old Java" method of code generation and
+ * compilation. Given a {@link CodeGenerator}, obtains the generated
+ * source code, compiles it with the selected compiler, loads the
+ * byte-codes into a class loader and provides the resulting
+ * class. Compared with the {@link ClassTransformer} mechanism,
+ * this one requires the code generator to have generated a complete
+ * Java class that is capable of direct compilation and loading.
+ * This means the generated class must be a subclass of the template
+ * so that the JVM can use normal Java inheritance to associate the
+ * template and generated methods.
+ * 
+ * Here is how to use the plain-old Java technique to debug
+ * generated code:
+ * 
+ * Set the config option drill.exec.compile.save_source
+ * to true.
+ * Set the config option drill.exec.compile.code_dir
+ * to the location where you want to save the generated source
+ * code.
+ * Where you generate code (using a {@link CodeGenerator}),
+ * set the "plain-old Java" options:
+ * CodeGeneratorFoo> cg = ...
+ * cg.plainOldJavaCapable(true); // Class supports plain-old Java
+ * cg.preferPlainOldJava(true); // Actually generate plain-old Java
+ * ...
+ * In your favorite IDE, add to the code lookup path the
+ * code directory saved earlier. In Eclipse, for example, you do
+ * this in the debug configuration you will use to debug Drill.
+ * Set a breakpoint in template used for the generated code.
+ * Run Drill. The IDE will stop at your breakpoint.
+ * Step into the generated code. Examine class field and
+ * local variables. Have fun!
+ * 
+ * 
+ * Note: not all generated code is ready to be compiled as plain-old
+ * Java. Some classes omit from the template the proper throws
+ * declarations. Other minor problems may also crop up. All are easy
+ * to fix. Once you've done so, add the following to mark that you've
+ * done the clean-up:
+ * cg.plainOldJavaCapable(true); // Class supports plain-old Java
+ * 
+ * The setting to prefer plain-old Java is ignored for generated
+ * classes not marked as plain-old Java capable.
+ */
+
+public class ClassBuilder {
+
+  public static final String SAVE_CODE_OPTION = CodeCompiler.COMPILE_BASE 
+ ".save_source";
+  public static final String CODE_DIR_OPTION = CodeCompiler.COMPILE_BASE + 
".code_dir";
+
+  private final DrillConfig config;
+  private final OptionManager options;
+  private final boolean saveCode;
+  private final File codeDir;
+
+  public ClassBuilder(DrillConfig config, OptionManager optionManager) {
+this.config = config;
+options = optionManager;
+
+// The option to save code is a boot-time option because
+// it is used selectively during debugging, but can cause
+// excessive I/O in a running server if used to save all code.
+

[jira] [Commented] (DRILL-5052) Option to debug generated Java code using an IDE

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15732491#comment-15732491
 ] 

ASF GitHub Bot commented on DRILL-5052:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/660#discussion_r91277152
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/compile/CodeCompiler.java ---
@@ -33,36 +32,71 @@
 import com.google.common.cache.LoadingCache;
 import com.google.common.collect.Lists;
 
+/**
+ * Global code compiler mechanism shared by all threads and operators.
+ * Holds a single cache of generated code (keyed by code source) to
+ * prevent compiling identical code multiple times. Supports both
+ * the byte-code merging and plain-old Java methods of code
+ * generation and compilation.
+ */
+
 public class CodeCompiler {
-//  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(CodeCompiler.class);
+
+  public static final String COMPILE_BASE = "drill.exec.compile";
+  String MAX_LOADING_CACHE_SIZE_CONFIG = COMPILE_BASE + ".cache_max_size";
--- End diff --

public static final?


> Option to debug generated Java code using an IDE
> 
>
> Key: DRILL-5052
> URL: https://issues.apache.org/jira/browse/DRILL-5052
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Codegen
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> Drill makes extensive use of Java code generation to implement its operators. 
> Drill uses sophisticated techniques to blend generated code with pre-compiled 
> template code. An unfortunate side-effect of this behavior is that it is very 
> difficult to visualize and debug the generated code.
> As it turns out, Drill's code-merge facility is, in essence, a do-it-yourself 
> version of subclassing. The Drill "template" is the parent class, the 
> generated code is the subclass. But, rather than using plain-old subclassing, 
> Drill combines the code from the two classes into a single "artificial" 
> packet of byte codes for which no source exists.
> Modify the code generation path to optionally allow "plain-old Java" 
> compilation: the generated code is a subclass of the template. Compile the 
> generated code as a plain-old Java class with no byte-code fix-up. Write the 
> code to a known location that the IDE can search when looking for source 
> files.
> With this change, developers can turn on the above feature, set a breakpoint 
> in a template, then step directly into the generated Java code called from 
> the template.
> This feature should be an option, enabled by developers when needed. The 
> existing byte-code technique should be used for production code generation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5052) Option to debug generated Java code using an IDE

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15732494#comment-15732494
 ] 

ASF GitHub Bot commented on DRILL-5052:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/660#discussion_r91510444
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/compile/ClassBuilder.java ---
@@ -0,0 +1,190 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.compile;
+
+import java.io.File;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.util.Map;
+
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.exec.compile.ClassTransformer.ClassNames;
+import org.apache.drill.exec.exception.ClassTransformationException;
+import org.apache.drill.exec.expr.CodeGenerator;
+import org.apache.drill.exec.server.options.OptionManager;
+import org.codehaus.commons.compiler.CompileException;
+
+/**
+ * Implements the "plain-old Java" method of code generation and
+ * compilation. Given a {@link CodeGenerator}, obtains the generated
+ * source code, compiles it with the selected compiler, loads the
+ * byte-codes into a class loader and provides the resulting
+ * class. Compared with the {@link ClassTransformer} mechanism,
+ * this one requires the code generator to have generated a complete
+ * Java class that is capable of direct compilation and loading.
+ * This means the generated class must be a subclass of the template
+ * so that the JVM can use normal Java inheritance to associate the
+ * template and generated methods.
+ * 
+ * Here is how to use the plain-old Java technique to debug
+ * generated code:
+ * 
+ * Set the config option drill.exec.compile.save_source
+ * to true.
+ * Set the config option drill.exec.compile.code_dir
+ * to the location where you want to save the generated source
+ * code.
+ * Where you generate code (using a {@link CodeGenerator}),
+ * set the "plain-old Java" options:
+ * CodeGeneratorFoo> cg = ...
+ * cg.plainOldJavaCapable(true); // Class supports plain-old Java
+ * cg.preferPlainOldJava(true); // Actually generate plain-old Java
+ * ...
+ * In your favorite IDE, add to the code lookup path the
+ * code directory saved earlier. In Eclipse, for example, you do
+ * this in the debug configuration you will use to debug Drill.
+ * Set a breakpoint in template used for the generated code.
+ * Run Drill. The IDE will stop at your breakpoint.
+ * Step into the generated code. Examine class field and
+ * local variables. Have fun!
+ * 
+ * 
+ * Note: not all generated code is ready to be compiled as plain-old
+ * Java. Some classes omit from the template the proper throws
+ * declarations. Other minor problems may also crop up. All are easy
+ * to fix. Once you've done so, add the following to mark that you've
+ * done the clean-up:
+ * cg.plainOldJavaCapable(true); // Class supports plain-old Java
+ * 
+ * The setting to prefer plain-old Java is ignored for generated
+ * classes not marked as plain-old Java capable.
+ */
+
+public class ClassBuilder {
+
+  public static final String SAVE_CODE_OPTION = CodeCompiler.COMPILE_BASE 
+ ".save_source";
+  public static final String CODE_DIR_OPTION = CodeCompiler.COMPILE_BASE + 
".code_dir";
+
+  private final DrillConfig config;
+  private final OptionManager options;
+  private final boolean saveCode;
+  private final File codeDir;
+
+  public ClassBuilder(DrillConfig config, OptionManager optionManager) {
+this.config = config;
+options = optionManager;
+
+// The option to save code is a boot-time option because
+// it is used selectively during debugging, but can cause
+// excessive I/O in a running server if used to save all code.
+

[jira] [Commented] (DRILL-5052) Option to debug generated Java code using an IDE

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15732497#comment-15732497
 ] 

ASF GitHub Bot commented on DRILL-5052:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/660#discussion_r91278968
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/DebugStringBuilder.java 
---
@@ -0,0 +1,57 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.expr;
+
+import java.io.PrintWriter;
+import java.io.StringWriter;
+
+import com.sun.codemodel.JFormatter;
+
+/**
+ * Utility class to build a debug string for an object
+ * in a standard format.
+ */
--- End diff --

Example of the standard format?


> Option to debug generated Java code using an IDE
> 
>
> Key: DRILL-5052
> URL: https://issues.apache.org/jira/browse/DRILL-5052
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Codegen
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> Drill makes extensive use of Java code generation to implement its operators. 
> Drill uses sophisticated techniques to blend generated code with pre-compiled 
> template code. An unfortunate side-effect of this behavior is that it is very 
> difficult to visualize and debug the generated code.
> As it turns out, Drill's code-merge facility is, in essence, a do-it-yourself 
> version of subclassing. The Drill "template" is the parent class, the 
> generated code is the subclass. But, rather than using plain-old subclassing, 
> Drill combines the code from the two classes into a single "artificial" 
> packet of byte codes for which no source exists.
> Modify the code generation path to optionally allow "plain-old Java" 
> compilation: the generated code is a subclass of the template. Compile the 
> generated code as a plain-old Java class with no byte-code fix-up. Write the 
> code to a known location that the IDE can search when looking for source 
> files.
> With this change, developers can turn on the above feature, set a breakpoint 
> in a template, then step directly into the generated Java code called from 
> the template.
> This feature should be an option, enabled by developers when needed. The 
> existing byte-code technique should be used for production code generation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5052) Option to debug generated Java code using an IDE

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15732493#comment-15732493
 ] 

ASF GitHub Bot commented on DRILL-5052:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/660#discussion_r91274791
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/compile/CachedClassLoader.java
 ---
@@ -0,0 +1,70 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.compile;
+
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.Map;
+import java.util.concurrent.ConcurrentMap;
+
+import com.google.common.collect.MapMaker;
+
+/**
+ * Class loader for "plain-old Java" generated classes.
+ * Very basic implementation: allows defining a class from
+ * byte codes and finding the loaded classes. Delegates
+ * all other class requests to the thread context class
+ * loader. This structure ensures that a generated class can
+ * find both its own inner classes as well as all the standard
+ * Drill implementation classes.
+ */
+
+public class CachedClassLoader extends URLClassLoader {
+
+  /**
+   * Cache of generated classes. Semantics: a single thread defines
+   * the classes, many threads may access the classes.
+   */
+
+  private ConcurrentMap cache = new 
MapMaker().concurrencyLevel(4).makeMap();
+
+  public CachedClassLoader( ) {
+super(new URL[0], Thread.currentThread().getContextClassLoader());
+  }
+
+  public void addClass( String fqcn, byte[] byteCodes ) {
+
+assert ! cache.containsKey( fqcn );
--- End diff --

Please use Guava Preconditions.checkState(cache.containsKey(fqcn), "your 
error message");


> Option to debug generated Java code using an IDE
> 
>
> Key: DRILL-5052
> URL: https://issues.apache.org/jira/browse/DRILL-5052
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Codegen
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> Drill makes extensive use of Java code generation to implement its operators. 
> Drill uses sophisticated techniques to blend generated code with pre-compiled 
> template code. An unfortunate side-effect of this behavior is that it is very 
> difficult to visualize and debug the generated code.
> As it turns out, Drill's code-merge facility is, in essence, a do-it-yourself 
> version of subclassing. The Drill "template" is the parent class, the 
> generated code is the subclass. But, rather than using plain-old subclassing, 
> Drill combines the code from the two classes into a single "artificial" 
> packet of byte codes for which no source exists.
> Modify the code generation path to optionally allow "plain-old Java" 
> compilation: the generated code is a subclass of the template. Compile the 
> generated code as a plain-old Java class with no byte-code fix-up. Write the 
> code to a known location that the IDE can search when looking for source 
> files.
> With this change, developers can turn on the above feature, set a breakpoint 
> in a template, then step directly into the generated Java code called from 
> the template.
> This feature should be an option, enabled by developers when needed. The 
> existing byte-code technique should be used for production code generation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5052) Option to debug generated Java code using an IDE

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15732496#comment-15732496
 ] 

ASF GitHub Bot commented on DRILL-5052:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/660#discussion_r91509053
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/compile/ClassCompilerSelector.java
 ---
@@ -0,0 +1,146 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.compile;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Map;
+
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.compile.ClassTransformer.ClassNames;
+import org.apache.drill.exec.exception.ClassTransformationException;
+import org.apache.drill.exec.server.options.OptionManager;
+import org.apache.drill.exec.server.options.OptionValidator;
+import org.apache.drill.exec.server.options.OptionValue;
+import 
org.apache.drill.exec.server.options.TypeValidators.BooleanValidator;
+import org.apache.drill.exec.server.options.TypeValidators.LongValidator;
+import org.apache.drill.exec.server.options.TypeValidators.StringValidator;
+import org.codehaus.commons.compiler.CompileException;
+
+/**
+ * Selects between the two supported Java compilers: Janino and
+ * the build-in Java compiler.
+ *
+ * Session Options
+ * 
+ * exec.java_compiler
+ * The compiler to use. Valid options are defined in the
+ * {@link ClassCompilerSelector.CompilerPolicy} enum.
+ * exec.java_compiler_debug
+ * If debug logging is enabled, then {@link AbstractClassCompiler} 
writes the
+ * generated Java code to the log file prior to compilation. This option
+ * adds line numbers to the logged code.
+ * exec.java_compiler_janino_maxsize
+ * The maximum size of code that the Janio compiler can handle. Larger 
code is
+ * handled by the JDK compiler. Defaults to 256K.
+ * 
+ * Configuration Options
+ * Configuration options are used when the above session options are unset.
+ * 
+ * drill.exec.compile.compiler
+ * Default for exec.java_compiler
+ * drill.exec.compile.debug
+ * Default for exec.java_compiler_debug
+ * drill.exec.compile.janino_maxsize
+ * Default for exec.java_compiler_janino_maxsize
+ * 
+ */
+
+public class ClassCompilerSelector {
+  public enum CompilerPolicy {
+DEFAULT, JDK, JANINO;
+  }
+
+  public static final String JAVA_COMPILER_JANINO_MAXSIZE_CONFIG = 
CodeCompiler.COMPILE_BASE + ".janino_maxsize";
+  public static final String JAVA_COMPILER_DEBUG_CONFIG = 
CodeCompiler.COMPILE_BASE + ".debug";
+  public static final String JAVA_COMPILER_CONFIG = 
CodeCompiler.COMPILE_BASE + ".compiler";
+
+  public static final String JAVA_COMPILER_OPTION = "exec.java_compiler";
+  public static final String JAVA_COMPILER_JANINO_MAXSIZE_OPTION = 
"exec.java_compiler_janino_maxsize";
+  public static final OptionValidator JAVA_COMPILER_JANINO_MAXSIZE = new 
LongValidator(JAVA_COMPILER_JANINO_MAXSIZE_OPTION, 256*1024);
+
+  public static final String JAVA_COMPILER_DEBUG_OPTION = 
"exec.java_compiler_debug";
+  public static final OptionValidator JAVA_COMPILER_DEBUG = new 
BooleanValidator(JAVA_COMPILER_DEBUG_OPTION, true);
+
+  public static final StringValidator JAVA_COMPILER_VALIDATOR = new 
StringValidator(JAVA_COMPILER_OPTION, CompilerPolicy.DEFAULT.toString()) {
+@Override
+public void validate(final OptionValue v, final OptionManager manager) 
{
+  super.validate(v, manager);
+  try {
+CompilerPolicy.valueOf(v.string_val.toUpperCase());
+  } catch (IllegalArgumentException e) {
+throw UserException.validationError()
+.message("Invalid value '%s' specified for option '%s'. Valid 

[jira] [Commented] (DRILL-5052) Option to debug generated Java code using an IDE

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15732508#comment-15732508
 ] 

ASF GitHub Bot commented on DRILL-5052:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/660#discussion_r91531999
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/compile/CodeCompiler.java ---
@@ -33,36 +32,71 @@
 import com.google.common.cache.LoadingCache;
 import com.google.common.collect.Lists;
 
+/**
+ * Global code compiler mechanism shared by all threads and operators.
+ * Holds a single cache of generated code (keyed by code source) to
+ * prevent compiling identical code multiple times. Supports both
+ * the byte-code merging and plain-old Java methods of code
+ * generation and compilation.
+ */
+
 public class CodeCompiler {
-//  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(CodeCompiler.class);
+
+  public static final String COMPILE_BASE = "drill.exec.compile";
+  String MAX_LOADING_CACHE_SIZE_CONFIG = COMPILE_BASE + ".cache_max_size";
 
   private final ClassTransformer transformer;
+  private final ClassBuilder classBuilder;
+
+  /**
+   * Google Guava loading cache that defers creating a cache
+   * entry until first needed. Creation is done in a thread-safe
+   * way: if two threads try to create the same class at the same
+   * time, the first does the work, the second waits for the first
+   * to complete, then grabs the new entry.
+   */
+
   private final LoadingCache cache;
-  private final DrillConfig config;
-  private final OptionManager optionManager;
 
   public CodeCompiler(final DrillConfig config, final OptionManager 
optionManager) {
-transformer = new ClassTransformer(optionManager);
-final int cacheMaxSize = 
config.getInt(ExecConstants.MAX_LOADING_CACHE_SIZE_CONFIG);
+transformer = new ClassTransformer(config, optionManager);
+classBuilder = new ClassBuilder(config, optionManager);
+final int cacheMaxSize = config.getInt(MAX_LOADING_CACHE_SIZE_CONFIG);
 cache = CacheBuilder.newBuilder()
 .maximumSize(cacheMaxSize)
 .build(new Loader());
-this.optionManager = optionManager;
-this.config = config;
   }
 
+  /**
+   * Create a single instance of the generated class.
+   *
+   * @param cg
+   * @return
+   * @throws ClassTransformationException
+   * @throws IOException
+   */
   @SuppressWarnings("unchecked")
-  public  T getImplementationClass(final CodeGenerator cg) throws 
ClassTransformationException, IOException {
-return (T) getImplementationClass(cg, 1).get(0);
+  public  T createInstance(final CodeGenerator cg) throws 
ClassTransformationException, IOException {
+return (T) createInstances(cg, 1).get(0);
   }
 
+  /**
+   * Create multiple instances of the generated class.
+   *
+   * @param cg
+   * @param count
+   * @return
+   * @throws ClassTransformationException
+   * @throws IOException
--- End diff --

Description for param, return and exceptions?


> Option to debug generated Java code using an IDE
> 
>
> Key: DRILL-5052
> URL: https://issues.apache.org/jira/browse/DRILL-5052
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Codegen
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> Drill makes extensive use of Java code generation to implement its operators. 
> Drill uses sophisticated techniques to blend generated code with pre-compiled 
> template code. An unfortunate side-effect of this behavior is that it is very 
> difficult to visualize and debug the generated code.
> As it turns out, Drill's code-merge facility is, in essence, a do-it-yourself 
> version of subclassing. The Drill "template" is the parent class, the 
> generated code is the subclass. But, rather than using plain-old subclassing, 
> Drill combines the code from the two classes into a single "artificial" 
> packet of byte codes for which no source exists.
> Modify the code generation path to optionally allow "plain-old Java" 
> compilation: the generated code is a subclass of the template. Compile the 
> generated code as a plain-old Java class with no byte-code fix-up. Write the 
> code to a known location that the IDE can search when looking for source 
> files.
> With this change, developers can turn on the above feature, set a breakpoint 
> in a template, then step directly into the 

[jira] [Commented] (DRILL-5052) Option to debug generated Java code using an IDE

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15732485#comment-15732485
 ] 

ASF GitHub Bot commented on DRILL-5052:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/660#discussion_r91285978
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/compile/ClassCompilerSelector.java
 ---
@@ -0,0 +1,146 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.compile;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Map;
+
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.compile.ClassTransformer.ClassNames;
+import org.apache.drill.exec.exception.ClassTransformationException;
+import org.apache.drill.exec.server.options.OptionManager;
+import org.apache.drill.exec.server.options.OptionValidator;
+import org.apache.drill.exec.server.options.OptionValue;
+import 
org.apache.drill.exec.server.options.TypeValidators.BooleanValidator;
+import org.apache.drill.exec.server.options.TypeValidators.LongValidator;
+import org.apache.drill.exec.server.options.TypeValidators.StringValidator;
+import org.codehaus.commons.compiler.CompileException;
+
+/**
+ * Selects between the two supported Java compilers: Janino and
+ * the build-in Java compiler.
+ *
+ * Session Options
+ * 
+ * exec.java_compiler
+ * The compiler to use. Valid options are defined in the
+ * {@link ClassCompilerSelector.CompilerPolicy} enum.
+ * exec.java_compiler_debug
+ * If debug logging is enabled, then {@link AbstractClassCompiler} 
writes the
+ * generated Java code to the log file prior to compilation. This option
+ * adds line numbers to the logged code.
+ * exec.java_compiler_janino_maxsize
+ * The maximum size of code that the Janio compiler can handle. Larger 
code is
+ * handled by the JDK compiler. Defaults to 256K.
+ * 
+ * Configuration Options
+ * Configuration options are used when the above session options are unset.
+ * 
+ * drill.exec.compile.compiler
+ * Default for exec.java_compiler
+ * drill.exec.compile.debug
+ * Default for exec.java_compiler_debug
+ * drill.exec.compile.janino_maxsize
+ * Default for exec.java_compiler_janino_maxsize
+ * 
+ */
+
+public class ClassCompilerSelector {
+  public enum CompilerPolicy {
+DEFAULT, JDK, JANINO;
+  }
+
+  public static final String JAVA_COMPILER_JANINO_MAXSIZE_CONFIG = 
CodeCompiler.COMPILE_BASE + ".janino_maxsize";
+  public static final String JAVA_COMPILER_DEBUG_CONFIG = 
CodeCompiler.COMPILE_BASE + ".debug";
+  public static final String JAVA_COMPILER_CONFIG = 
CodeCompiler.COMPILE_BASE + ".compiler";
+
+  public static final String JAVA_COMPILER_OPTION = "exec.java_compiler";
+  public static final String JAVA_COMPILER_JANINO_MAXSIZE_OPTION = 
"exec.java_compiler_janino_maxsize";
+  public static final OptionValidator JAVA_COMPILER_JANINO_MAXSIZE = new 
LongValidator(JAVA_COMPILER_JANINO_MAXSIZE_OPTION, 256*1024);
+
+  public static final String JAVA_COMPILER_DEBUG_OPTION = 
"exec.java_compiler_debug";
+  public static final OptionValidator JAVA_COMPILER_DEBUG = new 
BooleanValidator(JAVA_COMPILER_DEBUG_OPTION, true);
+
+  public static final StringValidator JAVA_COMPILER_VALIDATOR = new 
StringValidator(JAVA_COMPILER_OPTION, CompilerPolicy.DEFAULT.toString()) {
+@Override
+public void validate(final OptionValue v, final OptionManager manager) 
{
+  super.validate(v, manager);
+  try {
+CompilerPolicy.valueOf(v.string_val.toUpperCase());
+  } catch (IllegalArgumentException e) {
+throw UserException.validationError()
+.message("Invalid value '%s' specified for option '%s'. Valid 

[jira] [Commented] (DRILL-5052) Option to debug generated Java code using an IDE

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15732484#comment-15732484
 ] 

ASF GitHub Bot commented on DRILL-5052:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/660#discussion_r91279238
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/ClassGenerator.java ---
@@ -411,6 +417,10 @@ public boolean equals(Object obj) {
 
   }
 
+  /**
+   * Represents a (Nullable)?(Type)Holder instance.
+   */
+
   public static class HoldingContainer{
 private final JVar holder;
--- End diff --

Space before curly brace.


> Option to debug generated Java code using an IDE
> 
>
> Key: DRILL-5052
> URL: https://issues.apache.org/jira/browse/DRILL-5052
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Codegen
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> Drill makes extensive use of Java code generation to implement its operators. 
> Drill uses sophisticated techniques to blend generated code with pre-compiled 
> template code. An unfortunate side-effect of this behavior is that it is very 
> difficult to visualize and debug the generated code.
> As it turns out, Drill's code-merge facility is, in essence, a do-it-yourself 
> version of subclassing. The Drill "template" is the parent class, the 
> generated code is the subclass. But, rather than using plain-old subclassing, 
> Drill combines the code from the two classes into a single "artificial" 
> packet of byte codes for which no source exists.
> Modify the code generation path to optionally allow "plain-old Java" 
> compilation: the generated code is a subclass of the template. Compile the 
> generated code as a plain-old Java class with no byte-code fix-up. Write the 
> code to a known location that the IDE can search when looking for source 
> files.
> With this change, developers can turn on the above feature, set a breakpoint 
> in a template, then step directly into the generated Java code called from 
> the template.
> This feature should be an option, enabled by developers when needed. The 
> existing byte-code technique should be used for production code generation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5052) Option to debug generated Java code using an IDE

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15732486#comment-15732486
 ] 

ASF GitHub Bot commented on DRILL-5052:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/660#discussion_r91300800
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/xsort/ExternalSortBatch.java
 ---
@@ -711,18 +711,20 @@ private MSorter createNewMSorter(FragmentContext 
context, List orderin
 g.rotateBlock();
 g.getEvalBlock()._return(JExpr.lit(0));
 
+cg.plainOldJavaCapable(true); // This class can generate plain-old 
Java.
+ // Uncomment out this line to debug the generated code.
+//  cg.preferPlainOldJavaJava(true);
 return context.getImplementationClass(cg);
-
-
   }
 
   public SingleBatchSorter createNewSorter(FragmentContext context, 
VectorAccessible batch)
   throws ClassTransformationException, IOException, 
SchemaChangeException{
 CodeGenerator cg = 
CodeGenerator.get(SingleBatchSorter.TEMPLATE_DEFINITION, 
context.getFunctionRegistry(), context.getOptions());
-ClassGenerator g = cg.getRoot();
-
-generateComparisons(g, batch);
+cg.plainOldJavaCapable(true); // This class can generate plain-old 
Java.
 
+// Uncomment out this line to debug the generated code.
+//cg.preferPlainOldJavaJava(true);
--- End diff --

preferPlainOldJavaJava -> incorrect method name


> Option to debug generated Java code using an IDE
> 
>
> Key: DRILL-5052
> URL: https://issues.apache.org/jira/browse/DRILL-5052
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Codegen
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> Drill makes extensive use of Java code generation to implement its operators. 
> Drill uses sophisticated techniques to blend generated code with pre-compiled 
> template code. An unfortunate side-effect of this behavior is that it is very 
> difficult to visualize and debug the generated code.
> As it turns out, Drill's code-merge facility is, in essence, a do-it-yourself 
> version of subclassing. The Drill "template" is the parent class, the 
> generated code is the subclass. But, rather than using plain-old subclassing, 
> Drill combines the code from the two classes into a single "artificial" 
> packet of byte codes for which no source exists.
> Modify the code generation path to optionally allow "plain-old Java" 
> compilation: the generated code is a subclass of the template. Compile the 
> generated code as a plain-old Java class with no byte-code fix-up. Write the 
> code to a known location that the IDE can search when looking for source 
> files.
> With this change, developers can turn on the above feature, set a breakpoint 
> in a template, then step directly into the generated Java code called from 
> the template.
> This feature should be an option, enabled by developers when needed. The 
> existing byte-code technique should be used for production code generation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5052) Option to debug generated Java code using an IDE

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15732489#comment-15732489
 ] 

ASF GitHub Bot commented on DRILL-5052:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/660#discussion_r91300485
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/CodeGenerator.java ---
@@ -67,26 +96,71 @@
 try {
   this.model = new JCodeModel();
   JDefinedClass clazz = model._package(PACKAGE_NAME)._class(className);
+  if ( isPlainOldJava( ) ) {
+clazz._extends(definition.getTemplateClass( ) );
+  }
   rootGenerator = new ClassGenerator<>(this, mappingSet, 
definition.getSignature(), new EvaluationVisitor(
   funcRegistry), clazz, model, optionManager);
 } catch (JClassAlreadyExistsException e) {
   throw new IllegalStateException(e);
 }
   }
 
+  /**
+   * Indicates that the code for this class can be generated using the
+   * "Plain Old Java" mechanism based on inheritance. The byte-code
+   * method is more lenient, so some code is missing some features such
+   * as proper exception labeling, etc. Set this option to true once
+   * the generation mechanism for a class has been cleaned up to work
+   * via the plain-old Java mechanism.
+   *
+   * @param flag true if the code generated from this instance is
+   * ready to be compiled as a plain-old Java class
+   */
+
+  public void plainOldJavaCapable(boolean flag) {
+plainOldJavaCapable = flag;
+  }
+
+  /**
+   * Identifies that this generated class should be generated via the
+   * plain-old Java mechanism. This flag only has meaning if the
+   * generated class is capable of plain-old Java generation.
+   *
+   * @param flag true if the class should be generated and compiled
+   * as a plain-old Java class (rather than via byte-code manipulations)
+   */
+
+  public void preferPlainOJava(boolean flag) {
+usePlainOldJava = flag;
--- End diff --

preferPlainOJava -> preferPlainOldJava


> Option to debug generated Java code using an IDE
> 
>
> Key: DRILL-5052
> URL: https://issues.apache.org/jira/browse/DRILL-5052
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Codegen
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> Drill makes extensive use of Java code generation to implement its operators. 
> Drill uses sophisticated techniques to blend generated code with pre-compiled 
> template code. An unfortunate side-effect of this behavior is that it is very 
> difficult to visualize and debug the generated code.
> As it turns out, Drill's code-merge facility is, in essence, a do-it-yourself 
> version of subclassing. The Drill "template" is the parent class, the 
> generated code is the subclass. But, rather than using plain-old subclassing, 
> Drill combines the code from the two classes into a single "artificial" 
> packet of byte codes for which no source exists.
> Modify the code generation path to optionally allow "plain-old Java" 
> compilation: the generated code is a subclass of the template. Compile the 
> generated code as a plain-old Java class with no byte-code fix-up. Write the 
> code to a known location that the IDE can search when looking for source 
> files.
> With this change, developers can turn on the above feature, set a breakpoint 
> in a template, then step directly into the generated Java code called from 
> the template.
> This feature should be an option, enabled by developers when needed. The 
> existing byte-code technique should be used for production code generation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5052) Option to debug generated Java code using an IDE

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15732502#comment-15732502
 ] 

ASF GitHub Bot commented on DRILL-5052:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/660#discussion_r91520598
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/compile/DrillJavaFileObject.java
 ---
@@ -38,14 +39,18 @@
 
   private Map outputFiles;
 
+  private String className;
--- End diff --

final?


> Option to debug generated Java code using an IDE
> 
>
> Key: DRILL-5052
> URL: https://issues.apache.org/jira/browse/DRILL-5052
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Codegen
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> Drill makes extensive use of Java code generation to implement its operators. 
> Drill uses sophisticated techniques to blend generated code with pre-compiled 
> template code. An unfortunate side-effect of this behavior is that it is very 
> difficult to visualize and debug the generated code.
> As it turns out, Drill's code-merge facility is, in essence, a do-it-yourself 
> version of subclassing. The Drill "template" is the parent class, the 
> generated code is the subclass. But, rather than using plain-old subclassing, 
> Drill combines the code from the two classes into a single "artificial" 
> packet of byte codes for which no source exists.
> Modify the code generation path to optionally allow "plain-old Java" 
> compilation: the generated code is a subclass of the template. Compile the 
> generated code as a plain-old Java class with no byte-code fix-up. Write the 
> code to a known location that the IDE can search when looking for source 
> files.
> With this change, developers can turn on the above feature, set a breakpoint 
> in a template, then step directly into the generated Java code called from 
> the template.
> This feature should be an option, enabled by developers when needed. The 
> existing byte-code technique should be used for production code generation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5052) Option to debug generated Java code using an IDE

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15732490#comment-15732490
 ] 

ASF GitHub Bot commented on DRILL-5052:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/660#discussion_r91278889
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/DirectExpression.java 
---
@@ -20,6 +20,12 @@
 import com.sun.codemodel.JExpressionImpl;
 import com.sun.codemodel.JFormatter;
 
+/**
+ * Encapsulates a Java expression, defined as anything that is
+ * valid in the following code:
+ * (expr)
+ */
+
 public class DirectExpression extends JExpressionImpl{
   static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(DirectExpression.class);
--- End diff --

Space before curly brace.


> Option to debug generated Java code using an IDE
> 
>
> Key: DRILL-5052
> URL: https://issues.apache.org/jira/browse/DRILL-5052
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Codegen
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> Drill makes extensive use of Java code generation to implement its operators. 
> Drill uses sophisticated techniques to blend generated code with pre-compiled 
> template code. An unfortunate side-effect of this behavior is that it is very 
> difficult to visualize and debug the generated code.
> As it turns out, Drill's code-merge facility is, in essence, a do-it-yourself 
> version of subclassing. The Drill "template" is the parent class, the 
> generated code is the subclass. But, rather than using plain-old subclassing, 
> Drill combines the code from the two classes into a single "artificial" 
> packet of byte codes for which no source exists.
> Modify the code generation path to optionally allow "plain-old Java" 
> compilation: the generated code is a subclass of the template. Compile the 
> generated code as a plain-old Java class with no byte-code fix-up. Write the 
> code to a known location that the IDE can search when looking for source 
> files.
> With this change, developers can turn on the above feature, set a breakpoint 
> in a template, then step directly into the generated Java code called from 
> the template.
> This feature should be an option, enabled by developers when needed. The 
> existing byte-code technique should be used for production code generation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5052) Option to debug generated Java code using an IDE

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15732499#comment-15732499
 ] 

ASF GitHub Bot commented on DRILL-5052:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/660#discussion_r91511795
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/compile/JaninoClassCompiler.java
 ---
@@ -58,5 +56,27 @@ public JaninoClassCompiler(ClassLoader 
parentClassLoader, boolean debug) {
   }
 
   @Override
+  public Map compile(final ClassNames className, final 
String sourceCode)
+  throws CompileException, IOException, ClassNotFoundException {
+
+ClassFile[] classFiles = doCompile( className, sourceCode );
+Map results = new HashMap<>( );
--- End diff --

I've noticed that mostly in Drill Guava lib is used for collection 
creation: Maps.newHashMap();


> Option to debug generated Java code using an IDE
> 
>
> Key: DRILL-5052
> URL: https://issues.apache.org/jira/browse/DRILL-5052
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Codegen
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> Drill makes extensive use of Java code generation to implement its operators. 
> Drill uses sophisticated techniques to blend generated code with pre-compiled 
> template code. An unfortunate side-effect of this behavior is that it is very 
> difficult to visualize and debug the generated code.
> As it turns out, Drill's code-merge facility is, in essence, a do-it-yourself 
> version of subclassing. The Drill "template" is the parent class, the 
> generated code is the subclass. But, rather than using plain-old subclassing, 
> Drill combines the code from the two classes into a single "artificial" 
> packet of byte codes for which no source exists.
> Modify the code generation path to optionally allow "plain-old Java" 
> compilation: the generated code is a subclass of the template. Compile the 
> generated code as a plain-old Java class with no byte-code fix-up. Write the 
> code to a known location that the IDE can search when looking for source 
> files.
> With this change, developers can turn on the above feature, set a breakpoint 
> in a template, then step directly into the generated Java code called from 
> the template.
> This feature should be an option, enabled by developers when needed. The 
> existing byte-code technique should be used for production code generation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5052) Option to debug generated Java code using an IDE

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15732495#comment-15732495
 ] 

ASF GitHub Bot commented on DRILL-5052:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/660#discussion_r91279454
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/ClassGenerator.java ---
@@ -246,6 +246,12 @@ public void rotateBlock() {
 rotateBlock(BlkCreateMode.TRUE);
   }
 
+  /**
+   * Create a new code block, closing the current block.
+   *
+   * @param mode
+   */
--- End diff --

Please add description about mode param, the less we have these yellow 
warnings from IDE the better.


> Option to debug generated Java code using an IDE
> 
>
> Key: DRILL-5052
> URL: https://issues.apache.org/jira/browse/DRILL-5052
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Codegen
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> Drill makes extensive use of Java code generation to implement its operators. 
> Drill uses sophisticated techniques to blend generated code with pre-compiled 
> template code. An unfortunate side-effect of this behavior is that it is very 
> difficult to visualize and debug the generated code.
> As it turns out, Drill's code-merge facility is, in essence, a do-it-yourself 
> version of subclassing. The Drill "template" is the parent class, the 
> generated code is the subclass. But, rather than using plain-old subclassing, 
> Drill combines the code from the two classes into a single "artificial" 
> packet of byte codes for which no source exists.
> Modify the code generation path to optionally allow "plain-old Java" 
> compilation: the generated code is a subclass of the template. Compile the 
> generated code as a plain-old Java class with no byte-code fix-up. Write the 
> code to a known location that the IDE can search when looking for source 
> files.
> With this change, developers can turn on the above feature, set a breakpoint 
> in a template, then step directly into the generated Java code called from 
> the template.
> This feature should be an option, enabled by developers when needed. The 
> existing byte-code technique should be used for production code generation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4995) Allow lazy init when dynamic UDF support is disabled

[Roman (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15732362#comment-15732362
 ] 

Roman commented on DRILL-4995:
--

I have tested this issue on 2-nodes cluster and it was not reproduced. Verified 
and closed.

> Allow lazy init when dynamic UDF support is disabled
> 
>
> Key: DRILL-4995
> URL: https://issues.apache.org/jira/browse/DRILL-4995
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Drill
>Affects Versions: 1.9.0
>Reporter: Roman
>Assignee: Arina Ielchiieva
>  Labels: ready-to-commit
> Fix For: 1.10.0
>
>
> Steps in 2 nodes cluster:
> In 1st node:
> 1. Register jar
> 2. Run function (success)
> 3. Disable dynamic UDF support 
> 4. Run function again (success)
> In 2nd node:
> 5. Try to run function (failed).
> In 1st node the function was initialized before disabling dynamic UDF 
> support. But in 2nd node the function was not initialized. So It seems we 
> need to allow lazy initialization when dynamic UDF support is disabled.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (DRILL-4995) Allow lazy init when dynamic UDF support is disabled

[Roman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Roman closed DRILL-4995.


> Allow lazy init when dynamic UDF support is disabled
> 
>
> Key: DRILL-4995
> URL: https://issues.apache.org/jira/browse/DRILL-4995
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Drill
>Affects Versions: 1.9.0
>Reporter: Roman
>Assignee: Arina Ielchiieva
>  Labels: ready-to-commit
> Fix For: 1.10.0
>
>
> Steps in 2 nodes cluster:
> In 1st node:
> 1. Register jar
> 2. Run function (success)
> 3. Disable dynamic UDF support 
> 4. Run function again (success)
> In 2nd node:
> 5. Try to run function (failed).
> In 1st node the function was initialized before disabling dynamic UDF 
> support. But in 2nd node the function was not initialized. So It seems we 
> need to allow lazy initialization when dynamic UDF support is disabled.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5043) Function that returns a unique id per session/connection similar to MySQL's CONNECTION_ID()

[Nagarajan Chinnasamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15732216#comment-15732216
 ] 

Nagarajan Chinnasamy commented on DRILL-5043:
-

Hi Gautam,

I have submitted the pull request. The link is:


https://github.com/apache/drill/pull/685



> Function that returns a unique id per session/connection similar to MySQL's 
> CONNECTION_ID()
> ---
>
> Key: DRILL-5043
> URL: https://issues.apache.org/jira/browse/DRILL-5043
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Functions - Drill
>Affects Versions: 1.8.0
>Reporter: Nagarajan Chinnasamy
>Priority: Minor
>  Labels: CONNECTION_ID, SESSION, UDF
> Attachments: 01_session_id_sqlline.png, 
> 02_session_id_webconsole_query.png, 03_session_id_webconsole_result.png
>
>
> Design and implement a function that returns a unique id per 
> session/connection similar to MySQL's CONNECTION_ID().



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5043) Function that returns a unique id per session/connection similar to MySQL's CONNECTION_ID()

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15732207#comment-15732207
 ] 

ASF GitHub Bot commented on DRILL-5043:
---

GitHub user nagarajanchinnasamy opened a pull request:

https://github.com/apache/drill/pull/685

Drill 5043: Function that returns a unique id per session/connection 
similar to MySQL's CONNECTION_ID()

Please check [DRILL-5043](https://issues.apache.org/jira/browse/DRILL-5043) 
for the details on changes made and other details.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/nagarajanchinnasamy/drill DRILL-5043

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/685.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #685


commit a579e1beaa5ea65c30a08f55abe41a689d8d6ead
Author: Nagarajan Chinnasamy 
Date:   2016-11-19T15:23:34Z

[DRILL-5043] Added session_id() to ContextFunctions - Initial changes

commit a29269b7368de022eb7360b22315cf9471bf8f62
Author: Nagarajan Chinnasamy 
Date:   2016-11-19T15:38:46Z

Merge https://github.com/apache/drill




> Function that returns a unique id per session/connection similar to MySQL's 
> CONNECTION_ID()
> ---
>
> Key: DRILL-5043
> URL: https://issues.apache.org/jira/browse/DRILL-5043
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Functions - Drill
>Affects Versions: 1.8.0
>Reporter: Nagarajan Chinnasamy
>Priority: Minor
>  Labels: CONNECTION_ID, SESSION, UDF
> Attachments: 01_session_id_sqlline.png, 
> 02_session_id_webconsole_query.png, 03_session_id_webconsole_result.png
>
>
> Design and implement a function that returns a unique id per 
> session/connection similar to MySQL's CONNECTION_ID().



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4935) Allow drillbits to advertise a configurable host address to Zookeeper

[Uwe L. Korn (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15731567#comment-15731567
 ] 

Uwe L. Korn commented on DRILL-4935:


To add another use case where this is very helpful:

If you run Drill on nodes with multiple hostnames, you can use this to tell 
them the correct hostname to use. For example host often have a management and 
a service hostname. Depending on your setup {{/etc/hostname}} may be the 
management hostname. In this case the drillbits would by default try to connect 
to eachother using their management interfaces (which will be firewalled). With 
the new configuration option, you can force them to advertise their hostnames 
and connect using the correct hostnames/interfaces.

> Allow drillbits to advertise a configurable host address to Zookeeper
> -
>
> Key: DRILL-4935
> URL: https://issues.apache.org/jira/browse/DRILL-4935
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Execution - RPC
>Affects Versions: 1.8.0
>Reporter: Harrison Mebane
>Priority: Minor
>  Labels: ready-to-commit
> Fix For: 1.10.0
>
>
> There are certain situations, such as running Drill in distributed Docker 
> containers, in which it is desirable to advertise a different hostname to 
> Zookeeper than would be output by INetAddress.getLocalHost().  I propose 
> adding a configuration variable 'drill.exec.rpc.bit.advertised.host' and 
> passing this address to Zookeeper when the configuration variable is 
> populated, otherwise falling back to the present behavior.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)