[GitHub] drill pull request #716: DRILL-5116: Enable generated code debugging in each...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/716


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill issue #716: DRILL-5116: Enable generated code debugging in each Drill ...

[jinfengni]

Github user jinfengni commented on the issue:

https://github.com/apache/drill/pull/716
  
+1

The patch looks good to me. Thanks for the PR, @paul-rogers !

I'll squash and re-run the regression, before merge it to master branch. 



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #716: DRILL-5116: Enable generated code debugging in each...

[jinfengni]

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/716#discussion_r95490516
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/compile/ClassBuilder.java ---
@@ -64,36 +67,33 @@
  * local variables. Have fun!
  * 
  * 
- * Note: not all generated code is ready to be compiled as plain-old
- * Java. Some classes omit from the template the proper throws
- * declarations. Other minor problems may also crop up. All are easy
- * to fix. Once you've done so, add the following to mark that you've
- * done the clean-up:
- * cg.plainOldJavaCapable(true); // Class supports plain-old Java
+ * Most generated classes have been upgraded to support Plain-old Java
+ * compilation. Once this work is complete, the calls to
+ * plainOldJavaCapable can be removed as all generated classes
+ * will be capable.
  * 
- * The setting to prefer plain-old Java is ignored for generated
+ * The setting to prefer plain-old Java is ignored for any remaining 
generated
  * classes not marked as plain-old Java capable.
  */
 
 public class ClassBuilder {
 
-  public static final String SAVE_CODE_OPTION = CodeCompiler.COMPILE_BASE 
+ ".save_source";
   public static final String CODE_DIR_OPTION = CodeCompiler.COMPILE_BASE + 
".code_dir";
 
   private final DrillConfig config;
   private final OptionManager options;
-  private final boolean saveCode;
   private final File codeDir;
 
   public ClassBuilder(DrillConfig config, OptionManager optionManager) {
 this.config = config;
 options = optionManager;
 
-// The option to save code is a boot-time option because
-// it is used selectively during debugging, but can cause
-// excessive I/O in a running server if used to save all code.
+// Code can be saved per-class to enable debugging.
+// Just mark the code generator as to be persisted,
+// point your debugger to the directory set below, and you
--- End diff --

It's probably fine to leave as it is now with proper document usage. If 
more people uses this feature and run into the class name difference issue, we 
can do that at that time. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #716: DRILL-5116: Enable generated code debugging in each...

[paul-rogers]

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/716#discussion_r95472935
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TestConvertFunctions.java
 ---
@@ -76,6 +75,22 @@
 
   String textFileContent;
 
+  @BeforeClass
+  public static void setup( ) {
+// Tests here rely on the byte-code merge approach to code
+// generation and will fail if using plain-old Java.
+// Actually, some queries succeed with plain-old Java that
+// fail with scalar replacement, but the tests check for the
+// scalar replacement failure and, not finding it, fail the
+// test.
+//
+// The setting here forces byte-code merge even if the
+// config file asks for plain-old Java.
+// TODO: Fix the tests to handle both cases.
+
+System.setProperty("drill.compile.prefer_plain_java", "false");
--- End diff --

Fixed as above.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #716: DRILL-5116: Enable generated code debugging in each...

[paul-rogers]

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/716#discussion_r95468443
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/compile/ClassBuilder.java ---
@@ -64,36 +67,33 @@
  * local variables. Have fun!
  * 
  * 
- * Note: not all generated code is ready to be compiled as plain-old
- * Java. Some classes omit from the template the proper throws
- * declarations. Other minor problems may also crop up. All are easy
- * to fix. Once you've done so, add the following to mark that you've
- * done the clean-up:
- * cg.plainOldJavaCapable(true); // Class supports plain-old Java
+ * Most generated classes have been upgraded to support Plain-old Java
+ * compilation. Once this work is complete, the calls to
+ * plainOldJavaCapable can be removed as all generated classes
+ * will be capable.
  * 
- * The setting to prefer plain-old Java is ignored for generated
+ * The setting to prefer plain-old Java is ignored for any remaining 
generated
  * classes not marked as plain-old Java capable.
  */
 
 public class ClassBuilder {
 
-  public static final String SAVE_CODE_OPTION = CodeCompiler.COMPILE_BASE 
+ ".save_source";
   public static final String CODE_DIR_OPTION = CodeCompiler.COMPILE_BASE + 
".code_dir";
 
   private final DrillConfig config;
   private final OptionManager options;
-  private final boolean saveCode;
   private final File codeDir;
 
   public ClassBuilder(DrillConfig config, OptionManager optionManager) {
 this.config = config;
 options = optionManager;
 
-// The option to save code is a boot-time option because
-// it is used selectively during debugging, but can cause
-// excessive I/O in a running server if used to save all code.
+// Code can be saved per-class to enable debugging.
+// Just mark the code generator as to be persisted,
+// point your debugger to the directory set below, and you
--- End diff --

Thanks. The key thing to remember here is that the generated code does not 
exist until the code generator runs, so it is somewhat hard to set a breakpoint 
ahead of time given the current structure.

You are right that we can *add* another feature that causes the code to 
reuse class names so that we can set breakpoints inside the generated code. I 
worked around this in my sort performance stuff by creating my own copy of the 
code which I then hand modified to try out various changes. That was a quick 
way to avoid the name issue and to try changes without having to muck with the 
code generator just to do a prototype.

Do you feel we need to add the consistent-class-name feature in this PR, or 
can that be a refinement in a later PR?

And, I agree we should document usage as a way to encourage folks to try it 
and help us figure out what else we can improve.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #716: DRILL-5116: Enable generated code debugging in each...

[paul-rogers]

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/716#discussion_r95479214
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/compile/ClassBuilder.java ---
@@ -64,14 +67,12 @@
  * local variables. Have fun!
  * 
  * 
- * Note: not all generated code is ready to be compiled as plain-old
- * Java. Some classes omit from the template the proper throws
- * declarations. Other minor problems may also crop up. All are easy
- * to fix. Once you've done so, add the following to mark that you've
- * done the clean-up:
- * cg.plainOldJavaCapable(true); // Class supports plain-old Java
+ * Most generated classes have been upgraded to support Plain-old Java
+ * compilation. Once this work is complete, the calls to
+ * plainOldJavaCapable can be removed as all generated classes
+ * will be capable.
  * 
- * The setting to prefer plain-old Java is ignored for generated
+ * The setting to prefer plain-old Java is ignored for any remaining 
generated
--- End diff --

Since I had to rename other method; went ahead and changed references to 
"plain-old Java" to just "plain Java."


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #716: DRILL-5116: Enable generated code debugging in each...

[paul-rogers]

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/716#discussion_r95472493
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/compile/TestClassTransformation.java
 ---
@@ -44,6 +44,9 @@
 
   @BeforeClass
   public static void beforeTestClassTransformation() throws Exception {
+// Tests here require the byte-code merge technique and are meaningless
+// if the plain-old Java technique is selected.
+System.setProperty("drill.compile.prefer_plain_java", "false");
--- End diff --

You are right. Replaced the name with the constant defined in 
`CodeCompiler` as I should have done in the first place.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #716: DRILL-5116: Enable generated code debugging in each...

[paul-rogers]

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/716#discussion_r95469122
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/compile/ClassBuilder.java ---
@@ -141,7 +142,9 @@ public ClassBuilder(DrillConfig config, OptionManager 
optionManager) {
 // A key advantage of this method is that the code can be
 // saved and debugged, if needed.
 
-saveCode(code, name);
+if (cg.persistCode()) {
--- End diff --

Method names changed to `saveCodeForDebugging` and `isCodeToBeSaved`. More 
of  a mouthful, but hopefully this will be clearer -- also clearer that code is 
saved only for the purpose of debugging (not, for example, as a code cache that 
spans Drillbit runs, etc.) I'd kind of hoped the Javadoc would explain all 
that...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #719: DRILL-5172: Display elapsed time for queries in the...

[kkhatua]

Github user kkhatua commented on a diff in the pull request:

https://github.com/apache/drill/pull/719#discussion_r95475935
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/profile/ProfileResources.java
 ---
@@ -105,6 +109,25 @@ public String getTime() {
   return format.format(time);
 }
 
+public long getStartTime() {
+  return startTime;
+}
+
+public long getEndTime() {
+  return endTime;
+}
+
+public String getDuration() {
--- End diff --

Done. This is a much simpler and cleaner solution. Thank you, 
@arina-ielchiieva !


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #719: DRILL-5172: Display elapsed time for queries in the...

[kkhatua]

Github user kkhatua commented on a diff in the pull request:

https://github.com/apache/drill/pull/719#discussion_r95470389
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/profile/ProfileResources.java
 ---
@@ -105,6 +109,25 @@ public String getTime() {
   return format.format(time);
 }
 
+public long getStartTime() {
+  return startTime;
+}
+
+public long getEndTime() {
+  return endTime;
+}
+
+public String getDuration() {
--- End diff --

That could be one way to work around it. It would be good to have the 
human-readable formatting done through only one static method. I'm still trying 
to familiarize myself with the way these classes and the freemarker templates 
interact, so let me see if I can do with a wrapper. :) 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #719: DRILL-5172: Display elapsed time for queries in the...

[kkhatua]

Github user kkhatua commented on a diff in the pull request:

https://github.com/apache/drill/pull/719#discussion_r95469066
  
--- Diff: 
protocol/src/main/java/org/apache/drill/exec/proto/UserBitShared.java ---
@@ -13595,6 +13597,17 @@ public long getEnd() {
   return end_;
 }
 
+public String getDuration() {
--- End diff --

The ProfileInfo class is actually where the change was made to retrieve the 
value from the model. We don't create any duration field, as it is computed 
based on the other 2 fields (start,end) and simply provide a getter for that. 
However, the ProfileInfo class is used only by the profiles listing freemarker 
template (list.ftl). 
For the individual query profile's template (profile.ftl), the model from 
which we retrieve the duration value is auto-generated by the protobuf, making 
this a lot more complicated to implement. Hence, we took the JavaScript 
approach. 
I'll probably refine it to strictly follow the MVC model when I'm 
attempting a fix for DRILL-5190 
https://issues.apache.org/jira/browse/DRILL-5190  [Display planning time for a 
query in its profile page ] , since that would require me to modify the 
protobuf.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (DRILL-5190) Display planning time for a query in its profile page

[Kunal Khatua (JIRA)]

Kunal Khatua created DRILL-5190:
---

 Summary: Display planning time for a query in its profile page
 Key: DRILL-5190
 URL: https://issues.apache.org/jira/browse/DRILL-5190
 Project: Apache Drill
  Issue Type: Bug
  Components: Web Server
Affects Versions: 1.9.0
Reporter: Kunal Khatua


Currently, the Web UI does not display the time spent in planning for a query 
in its profile page. The estimate needs to be done by seeing how late did the 
earliest major fragment start. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] drill pull request #716: DRILL-5116: Enable generated code debugging in each...

[jinfengni]

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/716#discussion_r95461369
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/compile/ClassBuilder.java ---
@@ -64,36 +67,33 @@
  * local variables. Have fun!
  * 
  * 
- * Note: not all generated code is ready to be compiled as plain-old
- * Java. Some classes omit from the template the proper throws
- * declarations. Other minor problems may also crop up. All are easy
- * to fix. Once you've done so, add the following to mark that you've
- * done the clean-up:
- * cg.plainOldJavaCapable(true); // Class supports plain-old Java
+ * Most generated classes have been upgraded to support Plain-old Java
+ * compilation. Once this work is complete, the calls to
+ * plainOldJavaCapable can be removed as all generated classes
+ * will be capable.
  * 
- * The setting to prefer plain-old Java is ignored for generated
+ * The setting to prefer plain-old Java is ignored for any remaining 
generated
  * classes not marked as plain-old Java capable.
  */
 
 public class ClassBuilder {
 
-  public static final String SAVE_CODE_OPTION = CodeCompiler.COMPILE_BASE 
+ ".save_source";
   public static final String CODE_DIR_OPTION = CodeCompiler.COMPILE_BASE + 
".code_dir";
 
   private final DrillConfig config;
   private final OptionManager options;
-  private final boolean saveCode;
   private final File codeDir;
 
   public ClassBuilder(DrillConfig config, OptionManager optionManager) {
 this.config = config;
 options = optionManager;
 
-// The option to save code is a boot-time option because
-// it is used selectively during debugging, but can cause
-// excessive I/O in a running server if used to save all code.
+// Code can be saved per-class to enable debugging.
+// Just mark the code generator as to be persisted,
+// point your debugger to the directory set below, and you
--- End diff --

What we talked about probably is slightly different. What you mean is to 
step into the generated code, without setting a breaking point in it.  What I 
mean is first run the query and have the code generated, then I setup some 
breakpoint in the code generated, re-run the query and start debugs. In such 
cases, we need make sure the two runs of query get the same class names, 
otherwise the breakpoint will not be effective. 
  
Either way, it would be good to doc the typical ways of using the generated 
code in debug mode. 



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill issue #716: DRILL-5116: Enable generated code debugging in each Drill ...

[jinfengni]

Github user jinfengni commented on the issue:

https://github.com/apache/drill/pull/716
  
In the current code generator, we log both the source code and the size of 
source / byte code, the time spent on compiling, and the complier (Janino vs 
JDK) at debug level. It's probably debatable to log the source code, but the 
other information is quite useful for analyzing customer issue. In the 
plain-old style generator, seems we do not put those information in debug log. 
For now, the patch is not enabled. But in the future, if we are going to use 
this new code generator in production, we had better log more information( 
either in the log, or in operator profile), as that will help improve service 
workload. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (DRILL-5189) There's no documentation for the typeof() function

[Chris Westin (JIRA)]

Chris Westin created DRILL-5189:
---

 Summary: There's no documentation for the typeof() function
 Key: DRILL-5189
 URL: https://issues.apache.org/jira/browse/DRILL-5189
 Project: Apache Drill
  Issue Type: Bug
  Components: Documentation
Reporter: Chris Westin


I looked through the documentation at https://drill.apache.org/docs/ under SQL 
Reference > SQL Functions > ... and could not find any reference to typeof(). 
Google searches only turned up a reference to DRILL-4204.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] drill pull request #716: DRILL-5116: Enable generated code debugging in each...

[jinfengni]

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/716#discussion_r95442410
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/compile/TestClassTransformation.java
 ---
@@ -44,6 +44,9 @@
 
   @BeforeClass
   public static void beforeTestClassTransformation() throws Exception {
+// Tests here require the byte-code merge technique and are meaningless
+// if the plain-old Java technique is selected.
+System.setProperty("drill.compile.prefer_plain_java", "false");
--- End diff --

Any comment regarding this one? I thought the name of the options has a 
typo?



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #716: DRILL-5116: Enable generated code debugging in each...

[jinfengni]

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/716#discussion_r95442239
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/compile/ClassBuilder.java ---
@@ -141,7 +142,9 @@ public ClassBuilder(DrillConfig config, OptionManager 
optionManager) {
 // A key advantage of this method is that the code can be
 // saved and debugged, if needed.
 
-saveCode(code, name);
+if (cg.persistCode()) {
--- End diff --

Sounds to me that `persistCode(boolean)` is to set the persist boolean flag 
in code generator if it's allowed, while `persistCode()` is to check if the 
boolean flag is turned on or not. In such case, can we call the former as 
`setPersistMode(boolean)`, and call the later as `isPersistable`? 



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Apache Drill Hangout Minutes (1/10/2017)

[Aman Sinha]

Attendees:  Arina, Aman, Khurram, Karthik, Paul, Roman



1.   Khurram wanted to know the status of Calcite rebasing. Roman indicated 
it was in progress but currently on temporary hold due to couple of higher 
priority issues.

2.   Lazy initialization for UDFs:  Paul wanted more background on this for 
the review.  Arina provided some details and will sync up with Paul.

3.   Case-sensitivity of temporary tables names based on review comments: 
after some discussion, we concluded that it should be case-insensitive, similar 
to the alias names for columns and tables.

4.   Function names without parenthesis:  suggest examining how 
current_date, current_time are implemented.  However, it will be easier to 
implement using parenthesis since there is no confusion with column names.

[GitHub] drill pull request #719: DRILL-5172: Display elapsed time for queries in the...

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/719#discussion_r95436174
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/profile/ProfileResources.java
 ---
@@ -105,6 +109,25 @@ public String getTime() {
   return format.format(time);
 }
 
+public long getStartTime() {
+  return startTime;
+}
+
+public long getEndTime() {
+  return endTime;
+}
+
+public String getDuration() {
--- End diff --

How about we create wrapper class around `QueryProfile` (ex: 
`QueryProfileWrapper`) which will contain `getDuration` method along with 
needed `QueryProfile` methods. Logic for converting two long values (start and 
end) into human-readable format will be factored out into static method in 
`ProfileResources` (ex: `convertDurationIntoString(long start, long end)`). 
`QueryProfileWrapper` and `ProfileResources` will call this utility method from 
their `getDuration` methods.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #719: DRILL-5172: Display elapsed time for queries in the...

[kkhatua]

Github user kkhatua commented on a diff in the pull request:

https://github.com/apache/drill/pull/719#discussion_r95432224
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/profile/ProfileResources.java
 ---
@@ -105,6 +109,25 @@ public String getTime() {
   return format.format(time);
 }
 
+public long getStartTime() {
+  return startTime;
+}
+
+public long getEndTime() {
+  return endTime;
+}
+
+public String getDuration() {
--- End diff --

The logic in this Java class provides the duration for the listing 
generated in the profile listing Freemarker file (list.ftl). For the actual 
query profile page, this gets trickier, since the freemarker template refers to 
methods from the auto-generated UserBitShared class in the protocol package. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Suggestion for topics for Drill hangout tomorrow

[Nagarajan Chinnasamy]

Drill way of invoking UDFs without parenthesis


Best Regards,
Nagu.

On Tue, Jan 10, 2017 at 11:41 PM, Aman Sinha  wrote:

> Hangout starting .. link: https://plus.google.com/hangouts/_/event/
> ci4rdiju8bv04a64efj5fedd0lc
>
> On 1/9/17, 5:24 PM, "Aman Sinha"  wrote:
>
> Hi all,
>
> The bi-weekly hangout is tomorrow (01/10/17, 10 AM PST). If you have
> any suggestions for topics for tomorrow, please add to this thread. We
> will also ask for
> topics at the beginning of the hangout.
>
> Thanks.
>
>
>

Re: Suggestion for topics for Drill hangout tomorrow

[Aman Sinha]

Hangout starting .. link: 
https://plus.google.com/hangouts/_/event/ci4rdiju8bv04a64efj5fedd0lc

On 1/9/17, 5:24 PM, "Aman Sinha"  wrote:

Hi all,

The bi-weekly hangout is tomorrow (01/10/17, 10 AM PST). If you have
any suggestions for topics for tomorrow, please add to this thread. We will 
also ask for
topics at the beginning of the hangout.

Thanks.

[GitHub] drill pull request #719: DRILL-5172: Display elapsed time for queries in the...

[paul-rogers]

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/719#discussion_r95411447
  
--- Diff: 
protocol/src/main/java/org/apache/drill/exec/proto/UserBitShared.java ---
@@ -13595,6 +13597,17 @@ public long getEnd() {
   return end_;
 }
 
+public String getDuration() {
--- End diff --

Actually, the right place to do this work is in the Java code that creates 
the "model" that we pass to Freemarker. (Yeah, it is kind of complicated.)

Freemarker uses the classic MVC model. The template is the view. Java 
builds the model. (Let's ignore the controller for now.)

For an example, look in `ProfileResources.java`, which is in this PR. Look 
at the `ProfileInfo` class. Here we gather up the bits of information we want 
to display on the web page. Any calcs needed to prepare the data are done here. 
Then, Freemarker simply pulls out the pre-created data using the getter methods.

So, the `ProfileInfo` class could be extended to have, say, the raw 
duration in ms along with methods to return the computed values: 
`getDisplayDuration()` and `getDurationUnits()`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (DRILL-5188) TPC-DS query16 fails - IllegalArgumentException: Target must be less than target count

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-5188:
-

 Summary: TPC-DS query16 fails - IllegalArgumentException: Target 
must be less than target count
 Key: DRILL-5188
 URL: https://issues.apache.org/jira/browse/DRILL-5188
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Flow
Affects Versions: 1.10.0
 Environment: 4 node cluster
Reporter: Khurram Faraaz


TPC-DS query 16 fails with below Exception. Query 32, Query 92, query 95 also 
fail due to same Exception, on Drill 1.10.0 (commit ID: bbcf4b76) against SF1 
data.

Details from drillbit.log and stack trace.
{noformat}
2017-01-10 11:42:54,769 [278b3740-fa7b-0f27-5c42-92f2f63f7f3d:foreman] INFO  
o.a.drill.exec.work.foreman.Foreman - Query text for query id 
278b3740-fa7b-0f27-5c42-92f2f63f7f3d: SELECT
Count(DISTINCT cs_order_number) AS `order count` ,
Sum(cs_ext_ship_cost)   AS `total shipping cost` ,
Sum(cs_net_profit)  AS `total net profit`
FROM catalog_sales cs1 ,
date_dim ,
customer_address ,
call_center
WHEREd_date BETWEEN '2002-3-01' AND  (
Cast('2002-3-01' AS DATE) + INTERVAL '60' day)
AND  cs1.cs_ship_date_sk = d_date_sk
AND  cs1.cs_ship_addr_sk = ca_address_sk
AND  ca_state = 'IA'
AND  cs1.cs_call_center_sk = cc_call_center_sk
AND  cc_county IN ('Williamson County',
'Williamson County',
'Williamson County',
'Williamson County',
'Williamson County' )
AND  EXISTS
(
SELECT *
FROM   catalog_sales cs2
WHERE  cs1.cs_order_number = cs2.cs_order_number
ANDcs1.cs_warehouse_sk <> cs2.cs_warehouse_sk)
AND  NOT EXISTS
(
SELECT *
FROM   catalog_returns cr1
WHERE  cs1.cs_order_number = cr1.cr_order_number)
ORDER BY count(DISTINCT cs_order_number)
LIMIT 100
2017-01-10 11:42:54,887 [278b3740-fa7b-0f27-5c42-92f2f63f7f3d:foreman] INFO  
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
numFiles: 1
2017-01-10 11:42:54,888 [278b3740-fa7b-0f27-5c42-92f2f63f7f3d:foreman] INFO  
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
numFiles: 1
2017-01-10 11:42:54,895 [278b3740-fa7b-0f27-5c42-92f2f63f7f3d:foreman] INFO  
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
numFiles: 1
2017-01-10 11:42:54,895 [278b3740-fa7b-0f27-5c42-92f2f63f7f3d:foreman] INFO  
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
numFiles: 1
2017-01-10 11:42:54,902 [278b3740-fa7b-0f27-5c42-92f2f63f7f3d:foreman] INFO  
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
numFiles: 1
2017-01-10 11:42:54,903 [278b3740-fa7b-0f27-5c42-92f2f63f7f3d:foreman] INFO  
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
numFiles: 1
2017-01-10 11:42:54,908 [278b3740-fa7b-0f27-5c42-92f2f63f7f3d:foreman] INFO  
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
numFiles: 1
2017-01-10 11:42:54,909 [278b3740-fa7b-0f27-5c42-92f2f63f7f3d:foreman] INFO  
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
numFiles: 1
2017-01-10 11:42:54,919 [278b3740-fa7b-0f27-5c42-92f2f63f7f3d:foreman] INFO  
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
numFiles: 1
2017-01-10 11:42:54,920 [278b3740-fa7b-0f27-5c42-92f2f63f7f3d:foreman] INFO  
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
numFiles: 1
2017-01-10 11:42:54,946 [278b3740-fa7b-0f27-5c42-92f2f63f7f3d:foreman] ERROR 
o.a.drill.exec.work.foreman.Foreman - SYSTEM ERROR: IllegalArgumentException: 
Target must be less than target count, 3


[Error Id: 61befda9-3479-48a4-a6ff-181703aea1b7 on centos-01.qa.lab:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
IllegalArgumentException: Target must be less than target count, 3


[Error Id: 61befda9-3479-48a4-a6ff-181703aea1b7 on centos-01.qa.lab:31010]
at 
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544)
 ~[drill-common-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at 
org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:824)
 [drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at 
org.apache.drill.exec.work.foreman.Foreman.moveToState(Foreman.java:944) 
[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:281) 
[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[na:1.8.0_91]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[na:1.8.0_91]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
Caused by: org.apache.drill.exec.work.foreman.ForemanException: Unexpected 
exception during fragment initialization: Target must be less than target 
count, 3

[GitHub] drill pull request #719: DRILL-5172: Display elapsed time for queries in the...

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/719#discussion_r95344665
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/profile/ProfileResources.java
 ---
@@ -105,6 +109,25 @@ public String getTime() {
   return format.format(time);
 }
 
+public long getStartTime() {
+  return startTime;
+}
+
+public long getEndTime() {
+  return endTime;
+}
+
+public String getDuration() {
--- End diff --

It seems we have similar logic here and in freemarker template for 
duration. May it can be removed in one of them?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (DRILL-5187) TPC-DS query 72 takes 32 minutes to complete

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-5187:
-

 Summary: TPC-DS query 72 takes 32 minutes to complete
 Key: DRILL-5187
 URL: https://issues.apache.org/jira/browse/DRILL-5187
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Flow
Affects Versions: 1.10.0
 Environment: 4 node cluster
Reporter: Khurram Faraaz
Priority: Critical


TPC-DS query takes close to 32 minutes to complete execution on a 4 node CentOS 
cluster, on Drill 1.10.0 against SF1 data.

Attached here are,
 - drillbit.log (has only query 72 information)
 - JSON profile for query 72
 - screen shot of fragment information from profiles tab on UI
 
HASH_JOIN operator takes 21minutes (fragment : 04-xx-53).
HASH_JOIN operator seems to be taking longer, it can be verified by looking at 
the profile information.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] drill pull request #719: DRILL-5172: Display elapsed time for queries in the...

[kkhatua]

Github user kkhatua commented on a diff in the pull request:

https://github.com/apache/drill/pull/719#discussion_r95327930
  
--- Diff: 
protocol/src/main/java/org/apache/drill/exec/proto/UserBitShared.java ---
@@ -13595,6 +13597,17 @@ public long getEnd() {
   return end_;
 }
 
+public String getDuration() {
--- End diff --

Removed the logic. We'll do this within the Webpage using Javascript. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (DRILL-4872) NPE from CTAS partitioned by a projected casted null

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva resolved DRILL-4872.
-
Resolution: Fixed

> NPE from CTAS partitioned by a projected casted null
> 
>
> Key: DRILL-4872
> URL: https://issues.apache.org/jira/browse/DRILL-4872
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Drill
>Affects Versions: 1.7.0
>Reporter: Boaz Ben-Zvi
>Assignee: Arina Ielchiieva
>  Labels: NPE
> Fix For: 1.10.0
>
>
> Extracted from DRILL-3898 : Running the same test case on a smaller table ( 
> store_sales.dat from TPCDS SF 1) has no space issues, but there is a Null 
> Pointer Exception from the projection:
> Caused by: java.lang.NullPointerException: null
>   at 
> org.apache.drill.exec.expr.fn.impl.ByteFunctionHelpers.compare(ByteFunctionHelpers.java:100)
>  ~[vector-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
>   at 
> org.apache.drill.exec.test.generated.ProjectorGen1.doEval(ProjectorTemplate.java:49)
>  ~[na:na]
>   at 
> org.apache.drill.exec.test.generated.ProjectorGen1.projectRecords(ProjectorTemplate.java:62)
>  ~[na:na]
>   at 
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.doWork(ProjectRecordBatch.java:199)
>  ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> A simplified version of the test case:
> 0: jdbc:drill:zk=local> create table dfs.tmp.ttt partition by ( x ) as select 
> case when columns[8] = '' then cast(null as varchar(10)) else cast(columns[8] 
> as varchar(10)) end as x FROM 
> dfs.`/Users/boazben-zvi/data/store_sales/store_sales.dat`;
> Error: SYSTEM ERROR: NullPointerException
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)