[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472737#comment-16472737
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

amansinha100 closed pull request #1251: DRILL-6249: Adding more unit testing 
documentation.
URL: https://github.com/apache/drill/pull/1251
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/README.md b/README.md
index a30cf07f16..f2b5b9d5d9 100644
--- a/README.md
+++ b/README.md
@@ -20,7 +20,10 @@ Please see the [Apache Drill 
Website](http://drill.apache.org/) or the [Apache D
 
 
 ## Join the community!
-Apache Drill is an Apache Foundation project and is seeking all types of 
contributions.  Please say hello on the Apache Drill mailing list or join our 
Google Hangouts for more information.  (More information can be found at the 
Apache Drill website).
+Apache Drill is an Apache Foundation project and is seeking all types of 
contributions.
+Please say hello on the [Apache Drill mailing 
list](http://drill.apache.org/mailinglists/)
+or join our [Google Hangouts](http://drill.apache.org/community-resources/) 
for more information.
+(More information can be found at the [Apache Drill 
website](http://drill.apache.org/)).
 
 ## Export Control
 This distribution includes cryptographic software. The country in which you 
currently reside may have restrictions on the import, possession, use, and/or 
re-export to another country, of encryption software. BEFORE using any 
encryption software, please check your country's laws, regulations and policies 
concerning the import, possession, or use, and re-export of encryption 
software, to see if this is permitted. See  for more 
information.  
diff --git a/common/src/test/java/org/apache/drill/test/DirTestWatcher.java 
b/common/src/test/java/org/apache/drill/test/DirTestWatcher.java
index 271ac940d1..6bdc9519f9 100644
--- a/common/src/test/java/org/apache/drill/test/DirTestWatcher.java
+++ b/common/src/test/java/org/apache/drill/test/DirTestWatcher.java
@@ -31,10 +31,10 @@
  * 
  *
  * 
- * A {@link DirTestWatcher} is added to a test by declaring it as a JUnit 
{@link org.junit.Rule}. A {@link org.junit.Rule} is
- * a piece of code that is run before and after every JUnit test marked with 
the {@link org.junit.Test} annotation. When the
+ * A {@link DirTestWatcher} is added to a test by declaring it as a JUnit 
{@link org.junit.Rule}. A {@link org.junit.Rule Rule} is
+ * a piece of code that is run before and after every JUnit test marked with 
the {@link org.junit.Test Test} annotation. When the
  * {@link DirTestWatcher} is added to a test class the {@link DirTestWatcher} 
will create a temp directory before each of your
- * {@link org.junit.Test}s and optionally delete the temp directory after each 
of your {@link org.junit.Test}s. The temp directory
+ * {@link org.junit.Test Test}s and optionally delete the temp directory after 
each of your {@link org.junit.Test Test}s. The base temp directory
  * created by the {@link DirTestWatcher} is in the target folder of the 
maven project and has the form
  * (my test class fully qualified name)/(my test method name). So in 
the context of the code example below, the temp directory created for
  * each test in target will be my.proj.MyTestClass/myTestMethod1 
and my.proj.MyTestClass/myTestMethod2 respectively.
@@ -46,7 +46,7 @@
  * 
  *
  * 
- * By default, the {@link DirTestWatcher} deletes the temp directory it 
creates at the end of each {@link org.junit.Test}. However, you can create a 
{@link DirTestWatcher}
+ * By default, the {@link DirTestWatcher} deletes the temp directory it 
creates at the end of each {@link org.junit.Test Test}. However, you can create 
a {@link DirTestWatcher}
  * by doing {@code new DirTestWatcher(false)} to disable the deletion of temp 
directories after a test. This is useful if you want to examine files after a 
test runs.
  * 
  *
@@ -54,16 +54,16 @@
  * package my.proj;
  *
  * public class MyTestClass {
- *   org.junit.Rule
+ *   Rule
  *   public final DirTestWatcher dirTestWatcher = new DirTestWatcher();
  *
- *   org.junit.Test
+ *   Test
  *   public void myTestMethod1() {
  * File dir = dirTestWatcher.getDir();
  * // Do stuff in the temp directory
  *   }
  *
- *   org.junit.Test
+ *   Test
  *   public void myTestMethod2() {
  * File dir = dirTestWatcher.getDir();
  * // Do stuff in the temp directory
diff --git a/common/src/test/java/org/apache/drill/test/TestTools.java 
b/common/src/test/java/org/apache/drill/test/TestTools.java
index 661b978d65..8cf7ca7c42 100644
--- a/common/src/test/java/org/apache/drill/test/TestTools.java
+++ 

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471011#comment-16471011
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

ilooner commented on issue #1251: DRILL-6249: Adding more unit testing 
documentation.
URL: https://github.com/apache/drill/pull/1251#issuecomment-388165401
 
 
   @paul-rogers Incorporated your feedback. Thanks for all the useful info!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16470977#comment-16470977
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

ilooner commented on a change in pull request #1251: DRILL-6249: Adding more 
unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r187431211
 
 

 ##
 File path: docs/dev/Testing.md
 ##
 @@ -2,27 +2,108 @@
 
 Drill makes extensive use of [JUnit](http://junit.org/junit4/) and other 
libraries for testing. This page provides pointers to the information you need 
to work with Drill tests. We don't repeat that information; you will want to 
follow the links and read the original material to get a complete understanding 
of the libraries that Drill uses.
 
-Caveat: information here about Drill is "reverse engineered" from the code; 
this page has not yet had the benefit of insight from the developers who 
created Drill's test structure.
-
-# Topics
-
-"Classic" Drill testing techniques
+# Writing Tests
 
+* [Test Data Sets](TestDataSets.md)
+* [Temp Directory Utilities](TempDirectories.md)
 * [Testing with JUnit](JUnit.md)
 * [Test Logging](TestLogging.md)
+
+## Deprecated Drill Testing Techniques
+
+This is a list of old Drill testing machinery that we have cleaner machinery 
for now.
+
+* [BaseTestQuery](BaseTestQuery.md): Deprecated, use 
[ClusterTest](ClusterTest.md) instead.
+
+## Legacy Drill Testing Techniques
+
+This is a list of old Drill testing machinery for which there is no other 
alternative at the moment.
+
 * [Testing with Physical Plans and Mock Data](LegacyTestingFrameworks.md)
 
-"Updated" Drill testing techniques
+## Latest Drill Testing Techniques
 
+* [RowSet Framework](RowSetFramework.md)
 * [Cluster Fixture Framework](ClusterFixture.md)
 * [Operator Fixture Framework](OperatorFixture.md)
-* [The Mock Record Reader](MockRecordReader.md)
+* [ClusterTest](ClusterTest.md)
+* [Single Operator Unit Test](PhysicalOpUnitTestBase.md)
+* [Mocking Components](MockingComponents.md)
+* [Generated Code](GeneratedCode.md)
 
 Review comment:
   Added a reference to MiniPlanUnitTestBase


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16470967#comment-16470967
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

ilooner commented on a change in pull request #1251: DRILL-6249: Adding more 
unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r187429699
 
 

 ##
 File path: docs/dev/Testing.md
 ##
 @@ -2,27 +2,108 @@
 
 Drill makes extensive use of [JUnit](http://junit.org/junit4/) and other 
libraries for testing. This page provides pointers to the information you need 
to work with Drill tests. We don't repeat that information; you will want to 
follow the links and read the original material to get a complete understanding 
of the libraries that Drill uses.
 
-Caveat: information here about Drill is "reverse engineered" from the code; 
this page has not yet had the benefit of insight from the developers who 
created Drill's test structure.
-
-# Topics
-
-"Classic" Drill testing techniques
+# Writing Tests
 
 Review comment:
   Adapted your description here into a Testing Guidelines section of Testing.md


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16470933#comment-16470933
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

ilooner commented on a change in pull request #1251: DRILL-6249: Adding more 
unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r187422140
 
 

 ##
 File path: docs/dev/RowSetFramework.md
 ##
 @@ -0,0 +1,62 @@
+# RowSet Framework
+
+The RowSet Framework allows you to create custom instances of:
+
+ * 
[BatchSchema](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/BatchSchema.java)
+ * 
[VectorContainer](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java)
+ * 
[RecordBatch](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/RecordBatch.java)
+ 
+It also allows the comparison of data container in 
[VectorContainer](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java)s
 through the use
+of the 
[RowSetComparison](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSetComparison.java)
 and 
+[RowSetUtilities](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSetUtilities.java).
+
+## Creating A 
[BatchSchema](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/BatchSchema.java)
+
+The 
[SchemaBuilder](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/schema/SchemaBuilder.java)
 class can be used
+to create an instance 
[BatchSchema](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/BatchSchema.java).
 An example 
+of how to to this can be found the **secondTest()** method of 
[ExampleTest](../../exec/java-exec/src/test/java/org/apache/drill/test/ExampleTest.java).
+
+## Creating Test 
[VectorContainer](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java)
+
+[VectorContainer](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java)s
 populated with data can be created with the 
+[RowSetBuilder](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSetBuilder.java).
 In order to use it do the following:
+
+ 1. Create an allocator
+```
+RootAllocator allocator = new RootAllocator(Long.MAX_VALUE);
 
 Review comment:
   Updated


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16470923#comment-16470923
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

ilooner commented on a change in pull request #1251: DRILL-6249: Adding more 
unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r187419005
 
 

 ##
 File path: docs/dev/RowSetFramework.md
 ##
 @@ -0,0 +1,62 @@
+# RowSet Framework
+
+The RowSet Framework allows you to create custom instances of:
+
+ * 
[BatchSchema](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/BatchSchema.java)
+ * 
[VectorContainer](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java)
+ * 
[RecordBatch](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/RecordBatch.java)
+ 
+It also allows the comparison of data container in 
[VectorContainer](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java)s
 through the use
+of the 
[RowSetComparison](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSetComparison.java)
 and 
+[RowSetUtilities](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSetUtilities.java).
+
+## Creating A 
[BatchSchema](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/BatchSchema.java)
+
+The 
[SchemaBuilder](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/schema/SchemaBuilder.java)
 class can be used
+to create an instance 
[BatchSchema](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/BatchSchema.java).
 An example 
+of how to to this can be found the **secondTest()** method of 
[ExampleTest](../../exec/java-exec/src/test/java/org/apache/drill/test/ExampleTest.java).
+
+## Creating Test 
[VectorContainer](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java)
+
+[VectorContainer](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java)s
 populated with data can be created with the 
+[RowSetBuilder](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSetBuilder.java).
 In order to use it do the following:
+
+ 1. Create an allocator
+```
+RootAllocator allocator = new RootAllocator(Long.MAX_VALUE);
+```
+ 1. Create the desired BatchSchema using the 
[SchemaBuilder](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/schema/SchemaBuilder.java).
+```
+BatchSchema schema = new SchemaBuilder()
+.add(...)
+.add(...)
+.build();
+```
+ 1. Create a 
[RowSetBuilder](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSetBuilder.java)
 and add
 
 Review comment:
   Added this info to javadocs and this doc as well.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16470919#comment-16470919
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

ilooner commented on a change in pull request #1251: DRILL-6249: Adding more 
unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r187418818
 
 

 ##
 File path: docs/dev/RowSetFramework.md
 ##
 @@ -0,0 +1,62 @@
+# RowSet Framework
+
+The RowSet Framework allows you to create custom instances of:
+
+ * 
[BatchSchema](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/BatchSchema.java)
+ * 
[VectorContainer](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java)
+ * 
[RecordBatch](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/RecordBatch.java)
+ 
+It also allows the comparison of data container in 
[VectorContainer](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java)s
 through the use
+of the 
[RowSetComparison](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSetComparison.java)
 and 
+[RowSetUtilities](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSetUtilities.java).
+
+## Creating A 
[BatchSchema](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/BatchSchema.java)
+
+The 
[SchemaBuilder](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/schema/SchemaBuilder.java)
 class can be used
+to create an instance 
[BatchSchema](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/BatchSchema.java).
 An example 
+of how to to this can be found the **secondTest()** method of 
[ExampleTest](../../exec/java-exec/src/test/java/org/apache/drill/test/ExampleTest.java).
+
+## Creating Test 
[VectorContainer](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java)
 
 Review comment:
   Updated


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469742#comment-16469742
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

ilooner commented on a change in pull request #1251: DRILL-6249: Adding more 
unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r187210825
 
 

 ##
 File path: docs/dev/MockingComponents.md
 ##
 @@ -0,0 +1,96 @@
+# Mocking Components
+
+There are a few techniques for mocking classes in unit tests:
+
+* Use a mocking library
+* Provide a simple impementation of an interface
+* Build a real instance of class using the class's builders / constructors
+* 
+
+## Mocking Libraries
+
+Drill uses two mocking libraries in order to mock classes.
+
+* [Mockito](http://site.mockito.org)
+* [JMockit](http://jmockit.github.io/tutorial.html) (Deprecated)
+
+While some tests use [JMockit](http://jmockit.github.io/tutorial.html), its 
usage has been deprecated since it doesn't work well with
+eclipse. Specifically any test that uses JMockit will fail when run with 
eclipse. So if you
+need to use a mocking library use [Mockito](http://site.mockito.org).
+
+## Mocking Contexts
+
+There are several contexts used throughout Drill, for a complete description 
of each and how
+they are used please see 
[FragmentContextImpl](../../exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContextImpl.java).
+
+When doing tests you can use the following mock contexts:
+
+  * 
[MockFragmentContext](../../exec/java-exec/src/test/java/org/apache/drill/test/OperatorFixture.java)
 is a simple mock implementation of
+  the 
[FragmentContext](../../exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java).
+
+## Creating An Instance of 
[QueryId](../../protocol/src/main/java/org/apache/drill/exec/proto/beans/QueryId.java)
+
+```
+UserBitShared.QueryId queryId = UserBitShared.QueryId.newBuilder()
+  .setPart1(1L)
+  .setPart2(2L)
+  .build();
+```
+
+## Creating 
[FragmentHandle](../../protocol/src/main/java/org/apache/drill/exec/proto/beans/FragmentHandle.java)
+
+```
+ExecProtos.FragmentHandle fragmentHandle = 
ExecProtos.FragmentHandle.newBuilder()
+  .setQueryId(queryId)
+  .setMinorFragmentId(1)
+  .setMajorFragmentId(2)
+  .build();
+```
+
+## Creating A 
[DrillConfig](../../common/src/main/java/org/apache/drill/common/config/DrillConfig.java)
+
+There are a few ways to create a 
[DrillConfig](../../common/src/main/java/org/apache/drill/common/config/DrillConfig.java).
 See the class for all of them. The simplest way is to
+ create a default config.
+
+```
+DrillConfig config = DrillConfig.create();
 
 Review comment:
   Added a reference to ConfigBuilder.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469738#comment-16469738
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

ilooner commented on a change in pull request #1251: DRILL-6249: Adding more 
unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r187210241
 
 

 ##
 File path: docs/dev/MockingComponents.md
 ##
 @@ -0,0 +1,96 @@
+# Mocking Components
+
+There are a few techniques for mocking classes in unit tests:
+
+* Use a mocking library
+* Provide a simple impementation of an interface
+* Build a real instance of class using the class's builders / constructors
+* 
+
+## Mocking Libraries
+
+Drill uses two mocking libraries in order to mock classes.
+
+* [Mockito](http://site.mockito.org)
+* [JMockit](http://jmockit.github.io/tutorial.html) (Deprecated)
+
+While some tests use [JMockit](http://jmockit.github.io/tutorial.html), its 
usage has been deprecated since it doesn't work well with
+eclipse. Specifically any test that uses JMockit will fail when run with 
eclipse. So if you
+need to use a mocking library use [Mockito](http://site.mockito.org).
+
+## Mocking Contexts
+
+There are several contexts used throughout Drill, for a complete description 
of each and how
+they are used please see 
[FragmentContextImpl](../../exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContextImpl.java).
+
+When doing tests you can use the following mock contexts:
+
+  * 
[MockFragmentContext](../../exec/java-exec/src/test/java/org/apache/drill/test/OperatorFixture.java)
 is a simple mock implementation of
+  the 
[FragmentContext](../../exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java).
 
 Review comment:
   The OperatorFixture uses the MockFragmentContext. I added this snippet to 
show how to retrieve the FragmentContext from the OperatorFixture.
   
   ```
   FragmentContext context = operatorFixture.getFragmentContext();
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469737#comment-16469737
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

ilooner commented on a change in pull request #1251: DRILL-6249: Adding more 
unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r187210052
 
 

 ##
 File path: docs/dev/MockingComponents.md
 ##
 @@ -0,0 +1,96 @@
+# Mocking Components
+
+There are a few techniques for mocking classes in unit tests:
+
+* Use a mocking library
+* Provide a simple impementation of an interface
+* Build a real instance of class using the class's builders / constructors
+* 
+
+## Mocking Libraries
+
+Drill uses two mocking libraries in order to mock classes.
+
+* [Mockito](http://site.mockito.org)
+* [JMockit](http://jmockit.github.io/tutorial.html) (Deprecated)
+
+While some tests use [JMockit](http://jmockit.github.io/tutorial.html), its 
usage has been deprecated since it doesn't work well with
+eclipse. Specifically any test that uses JMockit will fail when run with 
eclipse. So if you
+need to use a mocking library use [Mockito](http://site.mockito.org).
 
 Review comment:
   I just removed this detail since we are saying these libraries are 
deprecated and we no longer want to use them.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469735#comment-16469735
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

ilooner commented on a change in pull request #1251: DRILL-6249: Adding more 
unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r187209902
 
 

 ##
 File path: docs/dev/MockingComponents.md
 ##
 @@ -0,0 +1,96 @@
+# Mocking Components
+
+There are a few techniques for mocking classes in unit tests:
+
+* Use a mocking library
+* Provide a simple impementation of an interface
+* Build a real instance of class using the class's builders / constructors
+* 
 
 Review comment:
   Removed


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469734#comment-16469734
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

ilooner commented on a change in pull request #1251: DRILL-6249: Adding more 
unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r187209858
 
 

 ##
 File path: docs/dev/DevDocs.md
 ##
 @@ -6,4 +6,8 @@ For information about configuring your development enviornment 
see [Environment.
 
 ## Testing
 
-For information about how to do Integration and Unit Testing in Drill see 
[Testing.md](Testing.md).
\ No newline at end of file
+For information about how to do Integration and Unit Testing in Drill see 
[Testing.md](Testing.md).
 
 Review comment:
   Done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469719#comment-16469719
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

ilooner commented on a change in pull request #1251: DRILL-6249: Adding more 
unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r187209131
 
 

 ##
 File path: common/src/test/java/org/apache/drill/test/DirTestWatcher.java
 ##
 @@ -34,7 +34,7 @@
  * A {@link DirTestWatcher} is added to a test by declaring it as a JUnit 
{@link org.junit.Rule}. A {@link org.junit.Rule} is
  * a piece of code that is run before and after every JUnit test marked with 
the {@link org.junit.Test} annotation. When the
  * {@link DirTestWatcher} is added to a test class the {@link DirTestWatcher} 
will create a temp directory before each of your
- * {@link org.junit.Test}s and optionally delete the temp directory after each 
of your {@link org.junit.Test}s. The temp directory
+ * {@link org.junit.Test}s and optionally delete the temp directory after each 
of your {@link org.junit.Test}s. The base temp directory
 
 Review comment:
   It doesn't show the base name. You are right, you have to use `{@link 
org.junit.Test Test}`. Updated to do so.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469712#comment-16469712
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

ilooner commented on a change in pull request #1251: DRILL-6249: Adding more 
unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r187208621
 
 

 ##
 File path: README.md
 ##
 @@ -20,7 +20,7 @@ Please see the [Apache Drill 
Website](http://drill.apache.org/) or the [Apache D
 
 
 ## Join the community!
-Apache Drill is an Apache Foundation project and is seeking all types of 
contributions.  Please say hello on the Apache Drill mailing list or join our 
Google Hangouts for more information.  (More information can be found at the 
Apache Drill website).
+Apache Drill is an Apache Foundation project and is seeking all types of 
contributions. Please say hello on the Apache Drill mailing list or join our 
Google Hangouts for more information. (More information can be found at the 
Apache Drill website).
 
 Review comment:
   Done.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469707#comment-16469707
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

ilooner commented on a change in pull request #1251: DRILL-6249: Adding more 
unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r187207855
 
 

 ##
 File path: 
exec/java-exec/src/test/java/org/apache/drill/test/rowSet/file/JsonFileBuilder.java
 ##
 @@ -35,6 +36,16 @@
 import java.util.List;
 import java.util.Map;
 
+/**
+ * Overview
+ * 
+ *   This class is used to build a json file containing the data in a {@link 
RowSet}.
 
 Review comment:
   done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469704#comment-16469704
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

ilooner commented on a change in pull request #1251: DRILL-6249: Adding more 
unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r187207643
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/ops/RootFragmentContext.java
 ##
 @@ -17,6 +17,9 @@
  */
 package org.apache.drill.exec.ops;
 
+/**
+ * This interface is implemented by fragment roots.
 
 Review comment:
   Updated


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469706#comment-16469706
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

ilooner commented on a change in pull request #1251: DRILL-6249: Adding more 
unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r187207707
 
 

 ##
 File path: 
exec/java-exec/src/test/java/org/apache/drill/test/BaseDirTestWatcher.java
 ##
 @@ -27,17 +27,31 @@
 import java.nio.file.Paths;
 
 /**
+ * Overview
  * 
  * This is a {@link DirTestWatcher} which creates all the temporary 
directories required by a Drillbit and the various dfs.* storage 
workspaces. It also
  * provides convenience methods that do the following:
- * 
  *
  * 
  *   Copy project files to temp directories. This is useful for copying 
the sample data into a temp directory.
  *   Copy resource files to temp.
  *   Updating parquet metadata files.
  * 
+ * 
+ *
+ * 
+ *   When used the {@link BaseDirTestWatcher} creates the following 
directories in the base temp directory (for a description of where the 
base temp directory
 
 Review comment:
   done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469703#comment-16469703
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

ilooner commented on a change in pull request #1251: DRILL-6249: Adding more 
unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r187207501
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContextImpl.java
 ##
 @@ -70,9 +69,26 @@
 import io.netty.buffer.DrillBuf;
 
 /**
- * Contextual objects required for execution of a particular fragment.
- * This is the implementation; use FragmentContext
- * in code to allow tests to use test-time implementations.
+ * 
+ *   This is the core Context which implements all the Context interfaces:
+ *
+ *   
+ * {@link FragmentContext}: A context provided to non-exchange 
operators.
+ * {@link ExchangeFragmentContext}: A context provided to exchange 
operators.
+ * {@link RootFragmentContext}: A context provided to fragment 
roots.
+ * {@link ExecutorFragmentContext}: A context used by the 
Drillbit.
+ *   
+ *
+ *   The interfaces above expose resources to varying degrees. They are 
ordered from most restrictive ({@link FragmentContext})
+ *   to least restrictive ({@link ExecutorFragmentContext}).
+ * 
+ * 
+ *   Since {@link FragmentContextImpl} implements all of the interfaces listed 
above, the facade pattern is used in order
+ *   to cast a {@link FragmentContextImpl} object to the desired interface 
where-ever it is needed. The facade pattern
+ *   is powerful since it allows us to easily create minimal mock context 
objects to be used in unit tests. Without
+ *   the use of interfaces and the facade pattern we would have to create a 
complete {@link FragmentContextImpl} object
+ *   to unit test any part of the code that depends on a context.
+ * 
 
 Review comment:
   Added this to javadoc.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469700#comment-16469700
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

ilooner commented on a change in pull request #1251: DRILL-6249: Adding more 
unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r187207066
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java
 ##
 @@ -38,16 +38,8 @@
 import io.netty.buffer.DrillBuf;
 
 /**
- * Fragment context interface: separates implementation from definition.
- * Allows unit testing by mocking or reimplementing services with
- * test-time versions. The name is awkward, chosen to avoid renaming
- * the implementation class which is used in many places in legacy code.
- * New code should use this interface, and the names should eventually
- * be swapped with {@link FragmentContextImpl} becoming
- * FragmentContextImpl and this interface becoming
- * {@link FragmentContextImpl}.
+ * This interface provides the resources required by a non-exchange operator 
to execute.
 
 Review comment:
   fixed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469699#comment-16469699
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

ilooner commented on a change in pull request #1251: DRILL-6249: Adding more 
unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r187206328
 
 

 ##
 File path: docs/dev/Testing.md
 ##
 @@ -2,27 +2,108 @@
 
 Drill makes extensive use of [JUnit](http://junit.org/junit4/) and other 
libraries for testing. This page provides pointers to the information you need 
to work with Drill tests. We don't repeat that information; you will want to 
follow the links and read the original material to get a complete understanding 
of the libraries that Drill uses.
 
-Caveat: information here about Drill is "reverse engineered" from the code; 
this page has not yet had the benefit of insight from the developers who 
created Drill's test structure.
-
-# Topics
-
-"Classic" Drill testing techniques
+# Writing Tests
 
+* [Test Data Sets](TestDataSets.md)
+* [Temp Directory Utilities](TempDirectories.md)
 * [Testing with JUnit](JUnit.md)
 * [Test Logging](TestLogging.md)
+
+## Deprecated Drill Testing Techniques
+
+This is a list of old Drill testing machinery that we have cleaner machinery 
for now.
+
+* [BaseTestQuery](BaseTestQuery.md): Deprecated, use 
[ClusterTest](ClusterTest.md) instead.
+
+## Legacy Drill Testing Techniques
+
+This is a list of old Drill testing machinery for which there is no other 
alternative at the moment.
+
 * [Testing with Physical Plans and Mock Data](LegacyTestingFrameworks.md)
 
-"Updated" Drill testing techniques
+## Latest Drill Testing Techniques
 
+* [RowSet Framework](RowSetFramework.md)
 * [Cluster Fixture Framework](ClusterFixture.md)
 * [Operator Fixture Framework](OperatorFixture.md)
-* [The Mock Record Reader](MockRecordReader.md)
+* [ClusterTest](ClusterTest.md)
+* [Single Operator Unit Test](PhysicalOpUnitTestBase.md)
+* [Mocking Components](MockingComponents.md)
+* [Generated Code](GeneratedCode.md)
+
+## Categories
+
+Currently Drill uses Travis to run smoke tests for every PR and commit. All of 
Drill's unit tests cannot be run on Travis because Drill's tests take longer to 
run than the
+maximum allowed container time for the free tier of Travis. In order to decide 
which tests are run on Travis and which tests are not, tests are categorized 
using JUnit's
+`@Category` annotation. Currently the following categories are excluded from 
Travis:
+
+  - **SlowTest:** Tests that are slow.
+  - **UnlikelyTest:** Tests that cover parts of the code that are rarely or 
never touched.
+  - **SecurityTest:** Corner case tests for security features.
+  
+To mark a test with a category you can do the following:
+
+```
+@Category(SlowTest.class)
+public class MyTest {
+  // Testing code
+}
+```
+
+To mark a test with multiple categories you can do the following:
+
+```
+@Category({SlowTest.class, SecurityTest.class})
+public class MyTest {
+  // Testing code
+}
+```
+
+# Running Tests
+
+Drill tests run in parallel. The Model for parallel execution is to divide 
test classes between multiple
+forked test processes. Each test process then runs the test classes assigned 
to it sequentially.
+
+## Speeding Up Test Runs
+
+There are a couple knobs you can turn to make tests run faster on your machine.
+
+ * **Maven Build Threads**: `-T `
+ * **Sure Fire Fork Count**: `-DforkCount=`
+ * **Test Categories**
+ 
+### -T
+
+Maven allows you to use multiple threads to compile sub modules. Also when 
running tests each build
+thread forks its own surefire process, so the tests for different submodules 
are run in parallel. In order
+to leverage this use the `-T` flag. By default this option is effectively.
+
+Ex. In order to run the build using two maven threads use the following 
command.
+
+```
+mvn -T 2 clean install
+```
+
+### -DforkCount
+
+To run tests within a submodule in parallel you can use the `-DforkCount` 
option. By default this 
+
+Ex. Run 4 test processes in parallel
+
+```
+mvn clean install -DforkCount=4
+```
+
+**Note:** The `-DforkCount` option interacts with `-T`. When use together each 
build thread (`-T`) gets 
 
 Review comment:
   Fixed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>  

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469695#comment-16469695
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

ilooner commented on a change in pull request #1251: DRILL-6249: Adding more 
unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r187206176
 
 

 ##
 File path: docs/dev/Testing.md
 ##
 @@ -2,27 +2,108 @@
 
 Drill makes extensive use of [JUnit](http://junit.org/junit4/) and other 
libraries for testing. This page provides pointers to the information you need 
to work with Drill tests. We don't repeat that information; you will want to 
follow the links and read the original material to get a complete understanding 
of the libraries that Drill uses.
 
-Caveat: information here about Drill is "reverse engineered" from the code; 
this page has not yet had the benefit of insight from the developers who 
created Drill's test structure.
-
-# Topics
-
-"Classic" Drill testing techniques
+# Writing Tests
 
+* [Test Data Sets](TestDataSets.md)
+* [Temp Directory Utilities](TempDirectories.md)
 * [Testing with JUnit](JUnit.md)
 * [Test Logging](TestLogging.md)
+
+## Deprecated Drill Testing Techniques
+
+This is a list of old Drill testing machinery that we have cleaner machinery 
for now.
+
+* [BaseTestQuery](BaseTestQuery.md): Deprecated, use 
[ClusterTest](ClusterTest.md) instead.
+
+## Legacy Drill Testing Techniques
+
+This is a list of old Drill testing machinery for which there is no other 
alternative at the moment.
+
 * [Testing with Physical Plans and Mock Data](LegacyTestingFrameworks.md)
 
-"Updated" Drill testing techniques
+## Latest Drill Testing Techniques
 
+* [RowSet Framework](RowSetFramework.md)
 * [Cluster Fixture Framework](ClusterFixture.md)
 * [Operator Fixture Framework](OperatorFixture.md)
-* [The Mock Record Reader](MockRecordReader.md)
+* [ClusterTest](ClusterTest.md)
+* [Single Operator Unit Test](PhysicalOpUnitTestBase.md)
+* [Mocking Components](MockingComponents.md)
+* [Generated Code](GeneratedCode.md)
+
+## Categories
+
+Currently Drill uses Travis to run smoke tests for every PR and commit. All of 
Drill's unit tests cannot be run on Travis because Drill's tests take longer to 
run than the
+maximum allowed container time for the free tier of Travis. In order to decide 
which tests are run on Travis and which tests are not, tests are categorized 
using JUnit's
+`@Category` annotation. Currently the following categories are excluded from 
Travis:
+
+  - **SlowTest:** Tests that are slow.
+  - **UnlikelyTest:** Tests that cover parts of the code that are rarely or 
never touched.
+  - **SecurityTest:** Corner case tests for security features.
+  
+To mark a test with a category you can do the following:
+
+```
+@Category(SlowTest.class)
+public class MyTest {
+  // Testing code
+}
+```
+
+To mark a test with multiple categories you can do the following:
+
+```
+@Category({SlowTest.class, SecurityTest.class})
+public class MyTest {
+  // Testing code
+}
+```
+
+# Running Tests
+
+Drill tests run in parallel. The Model for parallel execution is to divide 
test classes between multiple
+forked test processes. Each test process then runs the test classes assigned 
to it sequentially.
+
+## Speeding Up Test Runs
+
+There are a couple knobs you can turn to make tests run faster on your machine.
+
+ * **Maven Build Threads**: `-T `
+ * **Sure Fire Fork Count**: `-DforkCount=`
+ * **Test Categories**
+ 
+### -T
+
+Maven allows you to use multiple threads to compile sub modules. Also when 
running tests each build
+thread forks its own surefire process, so the tests for different submodules 
are run in parallel. In order
+to leverage this use the `-T` flag. By default this option is effectively.
 
 Review comment:
   Fixed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will 

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469697#comment-16469697
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

ilooner commented on a change in pull request #1251: DRILL-6249: Adding more 
unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r187206270
 
 

 ##
 File path: docs/dev/Testing.md
 ##
 @@ -2,27 +2,108 @@
 
 Drill makes extensive use of [JUnit](http://junit.org/junit4/) and other 
libraries for testing. This page provides pointers to the information you need 
to work with Drill tests. We don't repeat that information; you will want to 
follow the links and read the original material to get a complete understanding 
of the libraries that Drill uses.
 
-Caveat: information here about Drill is "reverse engineered" from the code; 
this page has not yet had the benefit of insight from the developers who 
created Drill's test structure.
-
-# Topics
-
-"Classic" Drill testing techniques
+# Writing Tests
 
+* [Test Data Sets](TestDataSets.md)
+* [Temp Directory Utilities](TempDirectories.md)
 * [Testing with JUnit](JUnit.md)
 * [Test Logging](TestLogging.md)
+
+## Deprecated Drill Testing Techniques
+
+This is a list of old Drill testing machinery that we have cleaner machinery 
for now.
+
+* [BaseTestQuery](BaseTestQuery.md): Deprecated, use 
[ClusterTest](ClusterTest.md) instead.
+
+## Legacy Drill Testing Techniques
+
+This is a list of old Drill testing machinery for which there is no other 
alternative at the moment.
+
 * [Testing with Physical Plans and Mock Data](LegacyTestingFrameworks.md)
 
-"Updated" Drill testing techniques
+## Latest Drill Testing Techniques
 
+* [RowSet Framework](RowSetFramework.md)
 * [Cluster Fixture Framework](ClusterFixture.md)
 * [Operator Fixture Framework](OperatorFixture.md)
-* [The Mock Record Reader](MockRecordReader.md)
+* [ClusterTest](ClusterTest.md)
+* [Single Operator Unit Test](PhysicalOpUnitTestBase.md)
+* [Mocking Components](MockingComponents.md)
+* [Generated Code](GeneratedCode.md)
+
+## Categories
+
+Currently Drill uses Travis to run smoke tests for every PR and commit. All of 
Drill's unit tests cannot be run on Travis because Drill's tests take longer to 
run than the
+maximum allowed container time for the free tier of Travis. In order to decide 
which tests are run on Travis and which tests are not, tests are categorized 
using JUnit's
+`@Category` annotation. Currently the following categories are excluded from 
Travis:
+
+  - **SlowTest:** Tests that are slow.
+  - **UnlikelyTest:** Tests that cover parts of the code that are rarely or 
never touched.
+  - **SecurityTest:** Corner case tests for security features.
+  
+To mark a test with a category you can do the following:
+
+```
+@Category(SlowTest.class)
+public class MyTest {
+  // Testing code
+}
+```
+
+To mark a test with multiple categories you can do the following:
+
+```
+@Category({SlowTest.class, SecurityTest.class})
+public class MyTest {
+  // Testing code
+}
+```
+
+# Running Tests
+
+Drill tests run in parallel. The Model for parallel execution is to divide 
test classes between multiple
+forked test processes. Each test process then runs the test classes assigned 
to it sequentially.
+
+## Speeding Up Test Runs
+
+There are a couple knobs you can turn to make tests run faster on your machine.
+
+ * **Maven Build Threads**: `-T `
+ * **Sure Fire Fork Count**: `-DforkCount=`
+ * **Test Categories**
+ 
+### -T
+
+Maven allows you to use multiple threads to compile sub modules. Also when 
running tests each build
+thread forks its own surefire process, so the tests for different submodules 
are run in parallel. In order
+to leverage this use the `-T` flag. By default this option is effectively.
+
+Ex. In order to run the build using two maven threads use the following 
command.
+
+```
+mvn -T 2 clean install
+```
+
+### -DforkCount
+
+To run tests within a submodule in parallel you can use the `-DforkCount` 
option. By default this 
 
 Review comment:
   Fixed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the 

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469694#comment-16469694
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

ilooner commented on a change in pull request #1251: DRILL-6249: Adding more 
unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r187206049
 
 

 ##
 File path: docs/dev/Testing.md
 ##
 @@ -2,27 +2,108 @@
 
 Drill makes extensive use of [JUnit](http://junit.org/junit4/) and other 
libraries for testing. This page provides pointers to the information you need 
to work with Drill tests. We don't repeat that information; you will want to 
follow the links and read the original material to get a complete understanding 
of the libraries that Drill uses.
 
-Caveat: information here about Drill is "reverse engineered" from the code; 
this page has not yet had the benefit of insight from the developers who 
created Drill's test structure.
-
-# Topics
-
-"Classic" Drill testing techniques
+# Writing Tests
 
+* [Test Data Sets](TestDataSets.md)
+* [Temp Directory Utilities](TempDirectories.md)
 * [Testing with JUnit](JUnit.md)
 * [Test Logging](TestLogging.md)
+
+## Deprecated Drill Testing Techniques
+
+This is a list of old Drill testing machinery that we have cleaner machinery 
for now.
+
+* [BaseTestQuery](BaseTestQuery.md): Deprecated, use 
[ClusterTest](ClusterTest.md) instead.
+
+## Legacy Drill Testing Techniques
+
+This is a list of old Drill testing machinery for which there is no other 
alternative at the moment.
+
 * [Testing with Physical Plans and Mock Data](LegacyTestingFrameworks.md)
 
-"Updated" Drill testing techniques
+## Latest Drill Testing Techniques
 
+* [RowSet Framework](RowSetFramework.md)
 * [Cluster Fixture Framework](ClusterFixture.md)
 * [Operator Fixture Framework](OperatorFixture.md)
-* [The Mock Record Reader](MockRecordReader.md)
+* [ClusterTest](ClusterTest.md)
+* [Single Operator Unit Test](PhysicalOpUnitTestBase.md)
+* [Mocking Components](MockingComponents.md)
+* [Generated Code](GeneratedCode.md)
+
+## Categories
+
+Currently Drill uses Travis to run smoke tests for every PR and commit. All of 
Drill's unit tests cannot be run on Travis because Drill's tests take longer to 
run than the
+maximum allowed container time for the free tier of Travis. In order to decide 
which tests are run on Travis and which tests are not, tests are categorized 
using JUnit's
+`@Category` annotation. Currently the following categories are excluded from 
Travis:
+
+  - **SlowTest:** Tests that are slow.
+  - **UnlikelyTest:** Tests that cover parts of the code that are rarely or 
never touched.
+  - **SecurityTest:** Corner case tests for security features.
+  
+To mark a test with a category you can do the following:
+
+```
+@Category(SlowTest.class)
+public class MyTest {
+  // Testing code
+}
+```
+
+To mark a test with multiple categories you can do the following:
+
+```
+@Category({SlowTest.class, SecurityTest.class})
+public class MyTest {
+  // Testing code
+}
+```
+
+# Running Tests
+
+Drill tests run in parallel. The Model for parallel execution is to divide 
test classes between multiple
 
 Review comment:
   Fixed


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469693#comment-16469693
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

ilooner commented on a change in pull request #1251: DRILL-6249: Adding more 
unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r187205951
 
 

 ##
 File path: docs/dev/TestDataSets.md
 ##
 @@ -0,0 +1,146 @@
+# Data Sets
+
+Drill includes several data sets for testing, and also provides some tools for 
generating test data sets.
+
+## Bundled Data Sets
+
+There are three primary data sets bundled with drill for testing:
+
+  - **Sample Data:** These are parquet files in the 
[sample-data](../sample-data) folder.
+  - **Resource Data:** These are data files in the 
[exec/java-exec/src/test/resources](../exec/java-exec/src/test/resources) fold.
+  - **TPCH Data:** These are trimmed down versions of the tpch data sets. They 
are retrieved and bundled
+  in the [contrib/data](../contrib/data) maven submodule. They are also 
accessible on [Apache Drill's S3 
bucket](http://apache-drill.s3.amazonaws.com/files/sf-0.01_tpc-h_parquet.tgz).
+  When unit tests are running all of the files in these data set are available 
from the classpath storage plugin. The tpch
+  files include:
+- **customer.parquet**
+- **lineitem.parquet**
+- **nation.parquet**
+- **orders.parquet**
+- **part.parquet**
+- **partsup.parquet**
+- **region.parquet**
+- **supplier.parquet**
+  
+### Using Sample Data in Unit Tests
+
+When using the 
[BaseDirTestWatcher](../exec/java-exec/src/test/java/org/apache/drill/test/BaseDirTestWatcher.java)
 you
+can make [sample-data](../sample-data) accessible from the ```dfs``` storage 
plugin by doing the following:
+
+```
+public class TestMyClass {
+  @ClassRule
+  public static final BaseDirTestWatcher dirTestWatcher = new 
BaseDirTestWatcher();
+  
+  @BeforeClass
+  public static void setupFiles() {
+dirTestWatcher.copyFileToRoot(Paths.get("sample-data", "region.parquet"));
+  }
+  
+  @Test
+  public void simpleTest() {
+ // dfs.root.`sample-data/region.parquet` will be accessible from my test
+  }
+}
+```
+
+Or if you are extending 
[BaseTestQuery](../exec/java-exec/src/test/java/org/apache/drill/test/BaseDirTestWatcher.java)
+
+```
+public class TestMyClass extends BaseTestQuery {
+  @BeforeClass
+  public static void setupFiles() {
+dirTestWatcher.copyFileToRoot(Paths.get("sample-data", "region.parquet"));
+  }
+  
+  @Test
+  public void simpleTest() {
+ // dfs.root.`sample-data/region.parquet` will be accessible from my test
+  }
+}
+```
+
+### Using Resource Data in Unit Tests
+
+When using the 
[BaseDirTestWatcher](../exec/java-exec/src/test/java/org/apache/drill/test/BaseDirTestWatcher.java)
 you
+can make data from 
[exec/java-exec/src/test/resources](../exec/java-exec/src/test/resources) 
accessible from the ```dfs``` storage plugin by doing the following:
+
+```
+public class TestMyClass {
+  @ClassRule
+  public static final BaseDirTestWatcher dirTestWatcher = new 
BaseDirTestWatcher();
+  
+  @BeforeClass
+  public static void setupFiles() {
+dirTestWatcher.copyResourceToRoot(Paths.get("join", "empty_part"));
+  }
+  
+  @Test
+  public void simpleTest() {
+ // src/test/resources/join/empty_part is acessible at 
dfs.root.`join/empty_part` from my test
+  }
+}
+```
+
+Or if you are extending 
[BaseTestQuery](../exec/java-exec/src/test/java/org/apache/drill/test/BaseDirTestWatcher.java)
+
+```
+public class TestMyClass extends BaseTestQuery {
+  @BeforeClass
+  public static void setupFiles() {
+dirTestWatcher.copyResourceToRoot(Paths.get("join", "empty_part"));
+  }
+  
+  @Test
+  public void simpleTest() {
+ // src/test/resources/join/empty_part is acessible at 
dfs.root.`join/empty_part` from my test
+  }
+}
+```
+
+### Using TPCH Data in Unit Tests
+
+TPCH data is accessible via the classpath storage plugin
+
+```
+cp.`tpch/customer.parquet`
+```
+
+## Generating Data Sets
 
 Review comment:
   I created a separate Jira to do this 
https://issues.apache.org/jira/browse/DRILL-6399


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: 

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469689#comment-16469689
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

ilooner commented on a change in pull request #1251: DRILL-6249: Adding more 
unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r187205150
 
 

 ##
 File path: docs/dev/TestDataSets.md
 ##
 @@ -0,0 +1,146 @@
+# Data Sets
+
+Drill includes several data sets for testing, and also provides some tools for 
generating test data sets.
+
+## Bundled Data Sets
+
+There are three primary data sets bundled with drill for testing:
+
+  - **Sample Data:** These are parquet files in the 
[sample-data](../sample-data) folder.
+  - **Resource Data:** These are data files in the 
[exec/java-exec/src/test/resources](../exec/java-exec/src/test/resources) fold.
 
 Review comment:
   Fixed


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469688#comment-16469688
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

ilooner commented on a change in pull request #1251: DRILL-6249: Adding more 
unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r187205087
 
 

 ##
 File path: docs/dev/TestDataSets.md
 ##
 @@ -0,0 +1,146 @@
+# Data Sets
+
+Drill includes several data sets for testing, and also provides some tools for 
generating test data sets.
+
+## Bundled Data Sets
+
+There are three primary data sets bundled with drill for testing:
+
+  - **Sample Data:** These are parquet files in the 
[sample-data](../sample-data) folder.
+  - **Resource Data:** These are data files in the 
[exec/java-exec/src/test/resources](../exec/java-exec/src/test/resources) fold.
+  - **TPCH Data:** These are trimmed down versions of the tpch data sets. They 
are retrieved and bundled
+  in the [contrib/data](../contrib/data) maven submodule. They are also 
accessible on [Apache Drill's S3 
bucket](http://apache-drill.s3.amazonaws.com/files/sf-0.01_tpc-h_parquet.tgz).
+  When unit tests are running all of the files in these data set are available 
from the classpath storage plugin. The tpch
+  files include:
+- **customer.parquet**
+- **lineitem.parquet**
+- **nation.parquet**
+- **orders.parquet**
+- **part.parquet**
+- **partsup.parquet**
+- **region.parquet**
+- **supplier.parquet**
+  
+### Using Sample Data in Unit Tests
+
+When using the 
[BaseDirTestWatcher](../exec/java-exec/src/test/java/org/apache/drill/test/BaseDirTestWatcher.java)
 you
+can make [sample-data](../sample-data) accessible from the ```dfs``` storage 
plugin by doing the following:
+
+```
+public class TestMyClass {
+  @ClassRule
+  public static final BaseDirTestWatcher dirTestWatcher = new 
BaseDirTestWatcher();
+  
+  @BeforeClass
+  public static void setupFiles() {
+dirTestWatcher.copyFileToRoot(Paths.get("sample-data", "region.parquet"));
+  }
+  
+  @Test
+  public void simpleTest() {
+ // dfs.root.`sample-data/region.parquet` will be accessible from my test
+  }
+}
+```
+
+Or if you are extending 
[BaseTestQuery](../exec/java-exec/src/test/java/org/apache/drill/test/BaseDirTestWatcher.java)
+
+```
+public class TestMyClass extends BaseTestQuery {
+  @BeforeClass
+  public static void setupFiles() {
+dirTestWatcher.copyFileToRoot(Paths.get("sample-data", "region.parquet"));
+  }
+  
+  @Test
+  public void simpleTest() {
+ // dfs.root.`sample-data/region.parquet` will be accessible from my test
+  }
+}
 
 Review comment:
   Added examples for this in ExampleTest and referenced them in this doc.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469643#comment-16469643
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

ilooner commented on a change in pull request #1251: DRILL-6249: Adding more 
unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r187199672
 
 

 ##
 File path: docs/dev/TempDirectories.md
 ##
 @@ -0,0 +1,17 @@
+# Temp Directory Utilities
+
+The two basic temp directory classes are:
+
+ - 
[DirTestWatcher](../exec/java-exec/src/test/java/org/apache/drill/test/DirTestWatcher.java)
+ - 
[BaseDirTestWatcher](../exec/java-exec/src/test/java/org/apache/drill/test/BaseDirTestWatcher.java)
+  
+These classes are used to create temp directories for each of your unit tests. 
The advantage to using
+these temp directory classes are:
+
+ - All files are deleted after a unit test completes. This prevents a build 
machine being polluted with a
+ bunch of unit test files.
+ - Each unit test outputs its files to a unique well defined location. This 
makes it easy to find files
+ for debugging. Also since each temp directory is unique, it prevents multiple 
unit test runs from interferring
+ with one another on a build machine.
+ 
+For examples on how to use these classes, please read the javadoc for each 
class.
 
 Review comment:
   Thanks for the info. I will add some examples to ExampleTest and will 
document this in TestDataSets.md .


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469595#comment-16469595
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

ilooner commented on a change in pull request #1251: DRILL-6249: Adding more 
unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r187192496
 
 

 ##
 File path: docs/dev/RowSetFramework.md
 ##
 @@ -0,0 +1,62 @@
+# RowSet Framework
+
+The RowSet Framework allows you to create custom instances of:
+
+ * 
[BatchSchema](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/BatchSchema.java)
+ * 
[VectorContainer](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java)
+ * 
[RecordBatch](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/RecordBatch.java)
+ 
+It also allows the comparison of data container in 
[VectorContainer](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java)s
 through the use
+of the 
[RowSetComparison](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSetComparison.java)
 and 
+[RowSetUtilities](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSetUtilities.java).
+
+## Creating A 
[BatchSchema](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/BatchSchema.java)
+
+The 
[SchemaBuilder](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/schema/SchemaBuilder.java)
 class can be used
+to create an instance 
[BatchSchema](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/BatchSchema.java).
 An example 
+of how to to this can be found the **secondTest()** method of 
[ExampleTest](../../exec/java-exec/src/test/java/org/apache/drill/test/ExampleTest.java).
+
+## Creating Test 
[VectorContainer](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java)
+
+[VectorContainer](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java)s
 populated with data can be created with the 
+[RowSetBuilder](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSetBuilder.java).
 In order to use it do the following:
+
+ 1. Create an allocator
+```
+RootAllocator allocator = new RootAllocator(Long.MAX_VALUE);
+```
+ 1. Create the desired BatchSchema using the 
[SchemaBuilder](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/schema/SchemaBuilder.java).
+```
+BatchSchema schema = new SchemaBuilder()
+.add(...)
+.add(...)
+.build();
+```
+ 1. Create a 
[RowSetBuilder](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSetBuilder.java)
 and add
+records to it. Then build a 
[RowSet](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSet.java).
+```
+RowSet rowSet = new RowSetBuilder(allocator, schema)
+  .addRow(110, "green", new float[]{5.5f, 2.3f}, new String[]{"1a", "1b"})
 
 Review comment:
   Fixed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469547#comment-16469547
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

ilooner commented on a change in pull request #1251: DRILL-6249: Adding more 
unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r187183549
 
 

 ##
 File path: docs/dev/RowSetFramework.md
 ##
 @@ -0,0 +1,62 @@
+# RowSet Framework
+
+The RowSet Framework allows you to create custom instances of:
+
+ * 
[BatchSchema](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/BatchSchema.java)
+ * 
[VectorContainer](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java)
+ * 
[RecordBatch](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/RecordBatch.java)
 
 Review comment:
   Added documentation for TupleMetadata. Highlighted the two other points in 
the doc.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469526#comment-16469526
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

ilooner commented on a change in pull request #1251: DRILL-6249: Adding more 
unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r187179092
 
 

 ##
 File path: docs/dev/PhysicalOpUnitTestBase.md
 ##
 @@ -0,0 +1,31 @@
+# Single Operator Unit Test
+
+It is possible to run an end to end test of an operator in isolation by 
extending 
+[PhysicalOpUnitTestBase](../../exec/java-exec/src/test/java/org/apache/drill/exec/physical/unit/PhysicalOpUnitTestBase.java).
+
+A simple example of an operator level unit test is the following:
+
+```
+public class BasicPhysicalOpUnitTest extends PhysicalOpUnitTestBase {
+
+ @Test
+ public void testSimpleProject() {
+   Project projectConf = new Project(parseExprs("x+5", "x"), null);
+   List jsonBatches = Lists.newArrayList(
+   "[{\"x\": 5 },{\"x\": 10 }]",
+   "[{\"x\": 20 },{\"x\": 30 },{\"x\": 40 }]");
+   opTestBuilder()
+   .physicalOperator(projectConf)
+   .inputDataStreamJson(jsonBatches)
+   .baselineColumns("x")
+   .baselineValues(10l)
+   .baselineValues(15l)
+   .baselineValues(25l)
+   .baselineValues(35l)
+   .baselineValues(45l)
+   .go();
 
 Review comment:
   I created a separate jira https://issues.apache.org/jira/browse/DRILL-6397. 
PhysicalOpUnitTestBase is already using OperatorFixture underneath the hood, so 
we would just have to update OperatorTestBuilder to use RowSets for baselines.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469517#comment-16469517
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

ilooner commented on a change in pull request #1251: DRILL-6249: Adding more 
unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r187178022
 
 

 ##
 File path: docs/dev/MockingComponents.md
 ##
 @@ -0,0 +1,96 @@
+# Mocking Components
+
+There are a few techniques for mocking classes in unit tests:
+
+* Use a mocking library
+* Provide a simple impementation of an interface
+* Build a real instance of class using the class's builders / constructors
+* 
+
+## Mocking Libraries
+
+Drill uses two mocking libraries in order to mock classes.
+
+* [Mockito](http://site.mockito.org)
+* [JMockit](http://jmockit.github.io/tutorial.html) (Deprecated)
+
+While some tests use [JMockit](http://jmockit.github.io/tutorial.html), its 
usage has been deprecated since it doesn't work well with
+eclipse. Specifically any test that uses JMockit will fail when run with 
eclipse. So if you
+need to use a mocking library use [Mockito](http://site.mockito.org).
+
+## Mocking Contexts
+
+There are several contexts used throughout Drill, for a complete description 
of each and how
+they are used please see 
[FragmentContextImpl](../../exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContextImpl.java).
+
+When doing tests you can use the following mock contexts:
+
+  * 
[MockFragmentContext](../../exec/java-exec/src/test/java/org/apache/drill/test/OperatorFixture.java)
 is a simple mock implementation of
+  the 
[FragmentContext](../../exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java).
+
+## Creating An Instance of 
[QueryId](../../protocol/src/main/java/org/apache/drill/exec/proto/beans/QueryId.java)
+
+```
+UserBitShared.QueryId queryId = UserBitShared.QueryId.newBuilder()
+  .setPart1(1L)
+  .setPart2(2L)
+  .build();
+```
+
+## Creating 
[FragmentHandle](../../protocol/src/main/java/org/apache/drill/exec/proto/beans/FragmentHandle.java)
+
+```
+ExecProtos.FragmentHandle fragmentHandle = 
ExecProtos.FragmentHandle.newBuilder()
+  .setQueryId(queryId)
+  .setMinorFragmentId(1)
+  .setMajorFragmentId(2)
+  .build();
+```
+
+## Creating A 
[DrillConfig](../../common/src/main/java/org/apache/drill/common/config/DrillConfig.java)
+
+There are a few ways to create a 
[DrillConfig](../../common/src/main/java/org/apache/drill/common/config/DrillConfig.java).
 See the class for all of them. The simplest way is to
+ create a default config.
+
+```
+DrillConfig config = DrillConfig.create();
+```
+
+## Creating A 
[SpillSet](../../exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/spill/SpillSet.java)
+
+ 1. Create a 
[PhysicalOperator](../../exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/PhysicalOperator.java).
+```
+HashJoinPOP pop = new HashJoinPOP(null, null, null, JoinRelType.FULL);
+```
+ 1. Create a 
[DrillConfig](../../common/src/main/java/org/apache/drill/common/config/DrillConfig.java).
+ 1. Create a 
[FragmentHandle](../../protocol/src/main/java/org/apache/drill/exec/proto/beans/FragmentHandle.java)
 as described above.
+ 1. Create a 
[SpillSet](../../exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/spill/SpillSet.java).
+```
+SpillSet spillSet = new SpillSet(config, fragmentHandle, pop);
+```
+ 
+## Creating A 
[PersistentStoreProvider](../../exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/PersistentStoreProvider.java)
+
+```
+LocalPersistentStoreProvider provider = new 
LocalPersistentStoreProvider(drillConfig);
+provider.start();
+```
+ 
+## Creating A 
[LogicalPlanPersistence](../../logical/src/main/java/org/apache/drill/common/config/LogicalPlanPersistence.java)
+
+```
+LogicalPlanPersistence logicalPlanPersistence = 
PhysicalPlanReaderTestFactory.defaultLogicalPlanPersistence(drillConfig);
+```
+
+## Creating An Instance Of An Option Manager
 
 Review comment:
   Agreed. Updated doc.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how 

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469515#comment-16469515
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

ilooner commented on a change in pull request #1251: DRILL-6249: Adding more 
unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r187178022
 
 

 ##
 File path: docs/dev/MockingComponents.md
 ##
 @@ -0,0 +1,96 @@
+# Mocking Components
+
+There are a few techniques for mocking classes in unit tests:
+
+* Use a mocking library
+* Provide a simple impementation of an interface
+* Build a real instance of class using the class's builders / constructors
+* 
+
+## Mocking Libraries
+
+Drill uses two mocking libraries in order to mock classes.
+
+* [Mockito](http://site.mockito.org)
+* [JMockit](http://jmockit.github.io/tutorial.html) (Deprecated)
+
+While some tests use [JMockit](http://jmockit.github.io/tutorial.html), its 
usage has been deprecated since it doesn't work well with
+eclipse. Specifically any test that uses JMockit will fail when run with 
eclipse. So if you
+need to use a mocking library use [Mockito](http://site.mockito.org).
+
+## Mocking Contexts
+
+There are several contexts used throughout Drill, for a complete description 
of each and how
+they are used please see 
[FragmentContextImpl](../../exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContextImpl.java).
+
+When doing tests you can use the following mock contexts:
+
+  * 
[MockFragmentContext](../../exec/java-exec/src/test/java/org/apache/drill/test/OperatorFixture.java)
 is a simple mock implementation of
+  the 
[FragmentContext](../../exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java).
+
+## Creating An Instance of 
[QueryId](../../protocol/src/main/java/org/apache/drill/exec/proto/beans/QueryId.java)
+
+```
+UserBitShared.QueryId queryId = UserBitShared.QueryId.newBuilder()
+  .setPart1(1L)
+  .setPart2(2L)
+  .build();
+```
+
+## Creating 
[FragmentHandle](../../protocol/src/main/java/org/apache/drill/exec/proto/beans/FragmentHandle.java)
+
+```
+ExecProtos.FragmentHandle fragmentHandle = 
ExecProtos.FragmentHandle.newBuilder()
+  .setQueryId(queryId)
+  .setMinorFragmentId(1)
+  .setMajorFragmentId(2)
+  .build();
+```
+
+## Creating A 
[DrillConfig](../../common/src/main/java/org/apache/drill/common/config/DrillConfig.java)
+
+There are a few ways to create a 
[DrillConfig](../../common/src/main/java/org/apache/drill/common/config/DrillConfig.java).
 See the class for all of them. The simplest way is to
+ create a default config.
+
+```
+DrillConfig config = DrillConfig.create();
+```
+
+## Creating A 
[SpillSet](../../exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/spill/SpillSet.java)
+
+ 1. Create a 
[PhysicalOperator](../../exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/PhysicalOperator.java).
+```
+HashJoinPOP pop = new HashJoinPOP(null, null, null, JoinRelType.FULL);
+```
+ 1. Create a 
[DrillConfig](../../common/src/main/java/org/apache/drill/common/config/DrillConfig.java).
+ 1. Create a 
[FragmentHandle](../../protocol/src/main/java/org/apache/drill/exec/proto/beans/FragmentHandle.java)
 as described above.
+ 1. Create a 
[SpillSet](../../exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/spill/SpillSet.java).
+```
+SpillSet spillSet = new SpillSet(config, fragmentHandle, pop);
+```
+ 
+## Creating A 
[PersistentStoreProvider](../../exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/PersistentStoreProvider.java)
+
+```
+LocalPersistentStoreProvider provider = new 
LocalPersistentStoreProvider(drillConfig);
+provider.start();
+```
+ 
+## Creating A 
[LogicalPlanPersistence](../../logical/src/main/java/org/apache/drill/common/config/LogicalPlanPersistence.java)
+
+```
+LogicalPlanPersistence logicalPlanPersistence = 
PhysicalPlanReaderTestFactory.defaultLogicalPlanPersistence(drillConfig);
+```
+
+## Creating An Instance Of An Option Manager
 
 Review comment:
   Agreed. Updated dock.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about 

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469431#comment-16469431
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

ilooner commented on a change in pull request #1251: DRILL-6249: Adding more 
unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r187165138
 
 

 ##
 File path: docs/dev/MockingComponents.md
 ##
 @@ -0,0 +1,96 @@
+# Mocking Components
+
+There are a few techniques for mocking classes in unit tests:
+
+* Use a mocking library
+* Provide a simple impementation of an interface
+* Build a real instance of class using the class's builders / constructors
 
 Review comment:
   Agreed. Modified the description so this is clearly marked as a deprecated 
method, and explained why it is deprecated and should not be used.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469429#comment-16469429
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

ilooner commented on a change in pull request #1251: DRILL-6249: Adding more 
unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r187164790
 
 

 ##
 File path: docs/dev/LicenseHeaders.md
 ##
 @@ -0,0 +1,34 @@
+# License Headers
+
+Drill uses two license header checkers:
+
+* [Apache RAT Plugin](http://creadur.apache.org/rat/apache-rat-plugin/)
+* [License Maven Plugin](http://code.mycila.com/license-maven-plugin/)
+
+## Why Two?
+
+[Apache RAT Plugin](http://creadur.apache.org/rat/apache-rat-plugin/) is used 
because it is the standard license header
+checker for Apache projects. 
+
+[License Maven Plugin](http://code.mycila.com/license-maven-plugin/) performs 
stricter license checks and disallows
+putting license headers in java docs.
+
+## Doing License Checks
+
+The license checks are disabled locally by default and are enabled on Travis. 
If you'd like to perform
+license checks locally you can do the following:
+
+```
+ mvn license:check -Dlicense.skip=false
+```
+
+## Auto Formatting Headers
+
+If the license checks fail and you can't figure out what's wrong with your 
headers, you can auto-format
+your license headers with the following command:
+
+```
+mvn license:format -Dlicense.skip=false
+```
+
+This command will also add license headers to files without them.
 
 Review comment:
   I've already tested this. Running license:format is safe and will only 
impact new files that you are adding in your commit, all pre-existing files in 
the project are correctly formatted and will not be impacted. I've also already 
added all the appropriate exclusions.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469427#comment-16469427
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

ilooner commented on a change in pull request #1251: DRILL-6249: Adding more 
unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r187164186
 
 

 ##
 File path: docs/dev/LicenseHeaders.md
 ##
 @@ -0,0 +1,34 @@
+# License Headers
+
+Drill uses two license header checkers:
+
+* [Apache RAT Plugin](http://creadur.apache.org/rat/apache-rat-plugin/)
+* [License Maven Plugin](http://code.mycila.com/license-maven-plugin/)
+
+## Why Two?
+
+[Apache RAT Plugin](http://creadur.apache.org/rat/apache-rat-plugin/) is used 
because it is the standard license header
+checker for Apache projects. 
+
+[License Maven Plugin](http://code.mycila.com/license-maven-plugin/) performs 
stricter license checks and disallows
+putting license headers in java docs.
 
 Review comment:
   Added description.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469418#comment-16469418
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

ilooner commented on a change in pull request #1251: DRILL-6249: Adding more 
unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r187163429
 
 

 ##
 File path: docs/dev/GeneratedCode.md
 ##
 @@ -0,0 +1,42 @@
+# Testing Generated Code
+
+## Writing Unit Tests For Generated Code
+
+An example of unit testing generated code without running all of Drill is the 
**priorityQueueOrderingTest()** test in 
+[TopNBatchTest](../../exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TopN/TopNBatchTest.java).
 That test tests the 
+[PriorityQueueTemplate](../../exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/TopN/PriorityQueueTemplate.java)
 class separately from the rest of Drill.
+
+The main accomplished by creating instances of the following classes:
+
+ * 
[FunctionLookupContext](../../exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionLookupContext.java)
+ * 
[CodeCompiler](../../exec/java-exec/src/main/java/org/apache/drill/exec/compile/CodeCompiler.java)
+ 
+## Creating A 
[FunctionLookupContext](../../exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionLookupContext.java)
+
+```
+new FunctionImplementationRegistry(drillConfig)
+```
+
+## Creating A 
[CodeCompiler](../../exec/java-exec/src/main/java/org/apache/drill/exec/compile/CodeCompiler.java)
+
+ 1. Create a 
[DrillConfig](../../common/src/main/java/org/apache/drill/common/config/DrillConfig.java)
 as shown in [MockingComponents.md](MockingComponents.md).
+ 1. Create a 
[SystemOptionManager](../../exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SystemOptionManager.java)
 as shown in 
+ [MockingComponents.md](MockingComponents.md).
+ 1. Create an instance of 
[CodeCompiler](../../exec/java-exec/src/main/java/org/apache/drill/exec/compile/CodeCompiler.java).
+```
+new CodeCompiler(drillConfig, optionManager)
+```
+
+## Debugging Generated Code
+
+It is possible to set break points in generated code.
+
+### Instructions For IntelliJ
+
+ 1. File→Project structure…→Modules→distribution→Sources → Add content root 
+ 1. Chose /tmp/drill/codegen 
+ 1. Mark it as Sources directory.
+ 1. Set saveCodeForDebugging(true) for the code generator of interest
+ 1. Run the unit test of interest
+ 1. Now some generated classes should appear in Intellij under the 
distribution module
+ 1. Set a break point in a generated class and run the unit test in debug mode
 
 Review comment:
   Added your instructions for eclipse.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469404#comment-16469404
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

ilooner commented on a change in pull request #1251: DRILL-6249: Adding more 
unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r187162074
 
 

 ##
 File path: docs/dev/GeneratedCode.md
 ##
 @@ -0,0 +1,42 @@
+# Testing Generated Code
+
+## Writing Unit Tests For Generated Code
+
+An example of unit testing generated code without running all of Drill is the 
**priorityQueueOrderingTest()** test in 
+[TopNBatchTest](../../exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TopN/TopNBatchTest.java).
 That test tests the 
+[PriorityQueueTemplate](../../exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/TopN/PriorityQueueTemplate.java)
 class separately from the rest of Drill.
+
+The main accomplished by creating instances of the following classes:
+
+ * 
[FunctionLookupContext](../../exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionLookupContext.java)
+ * 
[CodeCompiler](../../exec/java-exec/src/main/java/org/apache/drill/exec/compile/CodeCompiler.java)
+ 
+## Creating A 
[FunctionLookupContext](../../exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionLookupContext.java)
+
+```
+new FunctionImplementationRegistry(drillConfig)
+```
+
+## Creating A 
[CodeCompiler](../../exec/java-exec/src/main/java/org/apache/drill/exec/compile/CodeCompiler.java)
+
+ 1. Create a 
[DrillConfig](../../common/src/main/java/org/apache/drill/common/config/DrillConfig.java)
 as shown in [MockingComponents.md](MockingComponents.md).
+ 1. Create a 
[SystemOptionManager](../../exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SystemOptionManager.java)
 as shown in 
+ [MockingComponents.md](MockingComponents.md).
 
 Review comment:
   I agree that mocking libraries are not a good way to test. I meant this to 
be a cheat sheet about how to instantiate classes, not a description of how to 
use a mocking library. I have renamed it to InstantiatingComponents.md to avoid 
this confusion. I am also updating the document with references to 
OperatorFixture and ClusterFixture where appropriate.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469402#comment-16469402
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

ilooner commented on a change in pull request #1251: DRILL-6249: Adding more 
unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r187161268
 
 

 ##
 File path: docs/dev/GeneratedCode.md
 ##
 @@ -0,0 +1,42 @@
+# Testing Generated Code
+
+## Writing Unit Tests For Generated Code
+
+An example of unit testing generated code without running all of Drill is the 
**priorityQueueOrderingTest()** test in 
+[TopNBatchTest](../../exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TopN/TopNBatchTest.java).
 That test tests the 
+[PriorityQueueTemplate](../../exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/TopN/PriorityQueueTemplate.java)
 class separately from the rest of Drill.
+
+The main accomplished by creating instances of the following classes:
+
+ * 
[FunctionLookupContext](../../exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionLookupContext.java)
+ * 
[CodeCompiler](../../exec/java-exec/src/main/java/org/apache/drill/exec/compile/CodeCompiler.java)
+ 
+## Creating A 
[FunctionLookupContext](../../exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionLookupContext.java)
+
+```
+new FunctionImplementationRegistry(drillConfig)
+```
+
+## Creating A 
[CodeCompiler](../../exec/java-exec/src/main/java/org/apache/drill/exec/compile/CodeCompiler.java)
+
+ 1. Create a 
[DrillConfig](../../common/src/main/java/org/apache/drill/common/config/DrillConfig.java)
 as shown in [MockingComponents.md](MockingComponents.md).
+ 1. Create a 
[SystemOptionManager](../../exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SystemOptionManager.java)
 as shown in 
 
 Review comment:
   Updated to reference OperatorFixture.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469397#comment-16469397
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

ilooner commented on a change in pull request #1251: DRILL-6249: Adding more 
unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r187160309
 
 

 ##
 File path: docs/dev/GeneratedCode.md
 ##
 @@ -0,0 +1,42 @@
+# Testing Generated Code
+
+## Writing Unit Tests For Generated Code
+
+An example of unit testing generated code without running all of Drill is the 
**priorityQueueOrderingTest()** test in 
+[TopNBatchTest](../../exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TopN/TopNBatchTest.java).
 That test tests the 
+[PriorityQueueTemplate](../../exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/TopN/PriorityQueueTemplate.java)
 class separately from the rest of Drill.
+
+The main accomplished by creating instances of the following classes:
+
+ * 
[FunctionLookupContext](../../exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionLookupContext.java)
+ * 
[CodeCompiler](../../exec/java-exec/src/main/java/org/apache/drill/exec/compile/CodeCompiler.java)
+ 
+## Creating A 
[FunctionLookupContext](../../exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionLookupContext.java)
+
+```
+new FunctionImplementationRegistry(drillConfig)
+```
+
+## Creating A 
[CodeCompiler](../../exec/java-exec/src/main/java/org/apache/drill/exec/compile/CodeCompiler.java)
+
+ 1. Create a 
[DrillConfig](../../common/src/main/java/org/apache/drill/common/config/DrillConfig.java)
 as shown in [MockingComponents.md](MockingComponents.md).
 
 Review comment:
   Updated to reference ClusterFixture and Operator Fixture.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469373#comment-16469373
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

ilooner commented on a change in pull request #1251: DRILL-6249: Adding more 
unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r187153599
 
 

 ##
 File path: docs/dev/GeneratedCode.md
 ##
 @@ -0,0 +1,42 @@
+# Testing Generated Code
+
+## Writing Unit Tests For Generated Code
+
+An example of unit testing generated code without running all of Drill is the 
**priorityQueueOrderingTest()** test in 
+[TopNBatchTest](../../exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TopN/TopNBatchTest.java).
 That test tests the 
+[PriorityQueueTemplate](../../exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/TopN/PriorityQueueTemplate.java)
 class separately from the rest of Drill.
+
+The main accomplished by creating instances of the following classes:
 
 Review comment:
   Fixed. Thanks for catching.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468428#comment-16468428
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding 
more unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r186932798
 
 

 ##
 File path: docs/dev/TempDirectories.md
 ##
 @@ -0,0 +1,17 @@
+# Temp Directory Utilities
+
+The two basic temp directory classes are:
+
+ - 
[DirTestWatcher](../exec/java-exec/src/test/java/org/apache/drill/test/DirTestWatcher.java)
+ - 
[BaseDirTestWatcher](../exec/java-exec/src/test/java/org/apache/drill/test/BaseDirTestWatcher.java)
+  
+These classes are used to create temp directories for each of your unit tests. 
The advantage to using
+these temp directory classes are:
+
+ - All files are deleted after a unit test completes. This prevents a build 
machine being polluted with a
+ bunch of unit test files.
+ - Each unit test outputs its files to a unique well defined location. This 
makes it easy to find files
+ for debugging. Also since each temp directory is unique, it prevents multiple 
unit test runs from interferring
+ with one another on a build machine.
+ 
+For examples on how to use these classes, please read the javadoc for each 
class.
 
 Review comment:
   Using the `ClusterFixture`, there is an easy way to define a schema that 
points to the test directory so you can use those directories to hold query 
input files.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468416#comment-16468416
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding 
more unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r186884344
 
 

 ##
 File path: docs/dev/GeneratedCode.md
 ##
 @@ -0,0 +1,42 @@
+# Testing Generated Code
+
+## Writing Unit Tests For Generated Code
+
+An example of unit testing generated code without running all of Drill is the 
**priorityQueueOrderingTest()** test in 
+[TopNBatchTest](../../exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TopN/TopNBatchTest.java).
 That test tests the 
+[PriorityQueueTemplate](../../exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/TopN/PriorityQueueTemplate.java)
 class separately from the rest of Drill.
+
+The main accomplished by creating instances of the following classes:
+
+ * 
[FunctionLookupContext](../../exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionLookupContext.java)
+ * 
[CodeCompiler](../../exec/java-exec/src/main/java/org/apache/drill/exec/compile/CodeCompiler.java)
+ 
+## Creating A 
[FunctionLookupContext](../../exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionLookupContext.java)
+
+```
+new FunctionImplementationRegistry(drillConfig)
+```
+
+## Creating A 
[CodeCompiler](../../exec/java-exec/src/main/java/org/apache/drill/exec/compile/CodeCompiler.java)
+
+ 1. Create a 
[DrillConfig](../../common/src/main/java/org/apache/drill/common/config/DrillConfig.java)
 as shown in [MockingComponents.md](MockingComponents.md).
 
 Review comment:
   There is a far easier way using the `ClusterFixture` (server-level tests) or 
`OperatorFixture` (lower-level tests).


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468439#comment-16468439
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding 
more unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r186932667
 
 

 ##
 File path: docs/dev/RowSetFramework.md
 ##
 @@ -0,0 +1,62 @@
+# RowSet Framework
+
+The RowSet Framework allows you to create custom instances of:
+
+ * 
[BatchSchema](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/BatchSchema.java)
+ * 
[VectorContainer](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java)
+ * 
[RecordBatch](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/RecordBatch.java)
+ 
+It also allows the comparison of data container in 
[VectorContainer](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java)s
 through the use
+of the 
[RowSetComparison](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSetComparison.java)
 and 
+[RowSetUtilities](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSetUtilities.java).
+
+## Creating A 
[BatchSchema](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/BatchSchema.java)
+
+The 
[SchemaBuilder](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/schema/SchemaBuilder.java)
 class can be used
+to create an instance 
[BatchSchema](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/BatchSchema.java).
 An example 
+of how to to this can be found the **secondTest()** method of 
[ExampleTest](../../exec/java-exec/src/test/java/org/apache/drill/test/ExampleTest.java).
+
+## Creating Test 
[VectorContainer](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java)
+
+[VectorContainer](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java)s
 populated with data can be created with the 
+[RowSetBuilder](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSetBuilder.java).
 In order to use it do the following:
+
+ 1. Create an allocator
+```
+RootAllocator allocator = new RootAllocator(Long.MAX_VALUE);
+```
+ 1. Create the desired BatchSchema using the 
[SchemaBuilder](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/schema/SchemaBuilder.java).
+```
+BatchSchema schema = new SchemaBuilder()
+.add(...)
+.add(...)
+.build();
+```
+ 1. Create a 
[RowSetBuilder](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSetBuilder.java)
 and add
+records to it. Then build a 
[RowSet](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSet.java).
+```
+RowSet rowSet = new RowSetBuilder(allocator, schema)
+  .addRow(110, "green", new float[]{5.5f, 2.3f}, new String[]{"1a", "1b"})
 
 Review comment:
   Remember that improvement we discussed many months back? Those methods are 
now available.
   ```
   static import ...RowSetUtilities.strArray;
   static import ...RowSetUtilities.floatArray; // Needed, only doubleArray at 
present
   
 .addRow(110, "green", floatArray(5.5f, 2.3f), strArray(1a", "1b"))
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468437#comment-16468437
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding 
more unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r186933533
 
 

 ##
 File path: docs/dev/TestDataSets.md
 ##
 @@ -0,0 +1,146 @@
+# Data Sets
+
+Drill includes several data sets for testing, and also provides some tools for 
generating test data sets.
+
+## Bundled Data Sets
+
+There are three primary data sets bundled with drill for testing:
+
+  - **Sample Data:** These are parquet files in the 
[sample-data](../sample-data) folder.
+  - **Resource Data:** These are data files in the 
[exec/java-exec/src/test/resources](../exec/java-exec/src/test/resources) fold.
+  - **TPCH Data:** These are trimmed down versions of the tpch data sets. They 
are retrieved and bundled
+  in the [contrib/data](../contrib/data) maven submodule. They are also 
accessible on [Apache Drill's S3 
bucket](http://apache-drill.s3.amazonaws.com/files/sf-0.01_tpc-h_parquet.tgz).
+  When unit tests are running all of the files in these data set are available 
from the classpath storage plugin. The tpch
+  files include:
+- **customer.parquet**
+- **lineitem.parquet**
+- **nation.parquet**
+- **orders.parquet**
+- **part.parquet**
+- **partsup.parquet**
+- **region.parquet**
+- **supplier.parquet**
+  
+### Using Sample Data in Unit Tests
+
+When using the 
[BaseDirTestWatcher](../exec/java-exec/src/test/java/org/apache/drill/test/BaseDirTestWatcher.java)
 you
+can make [sample-data](../sample-data) accessible from the ```dfs``` storage 
plugin by doing the following:
+
+```
+public class TestMyClass {
+  @ClassRule
+  public static final BaseDirTestWatcher dirTestWatcher = new 
BaseDirTestWatcher();
+  
+  @BeforeClass
+  public static void setupFiles() {
+dirTestWatcher.copyFileToRoot(Paths.get("sample-data", "region.parquet"));
+  }
+  
+  @Test
+  public void simpleTest() {
+ // dfs.root.`sample-data/region.parquet` will be accessible from my test
+  }
+}
+```
+
+Or if you are extending 
[BaseTestQuery](../exec/java-exec/src/test/java/org/apache/drill/test/BaseDirTestWatcher.java)
+
+```
+public class TestMyClass extends BaseTestQuery {
+  @BeforeClass
+  public static void setupFiles() {
+dirTestWatcher.copyFileToRoot(Paths.get("sample-data", "region.parquet"));
+  }
+  
+  @Test
+  public void simpleTest() {
+ // dfs.root.`sample-data/region.parquet` will be accessible from my test
+  }
+}
+```
+
+### Using Resource Data in Unit Tests
+
+When using the 
[BaseDirTestWatcher](../exec/java-exec/src/test/java/org/apache/drill/test/BaseDirTestWatcher.java)
 you
+can make data from 
[exec/java-exec/src/test/resources](../exec/java-exec/src/test/resources) 
accessible from the ```dfs``` storage plugin by doing the following:
+
+```
+public class TestMyClass {
+  @ClassRule
+  public static final BaseDirTestWatcher dirTestWatcher = new 
BaseDirTestWatcher();
+  
+  @BeforeClass
+  public static void setupFiles() {
+dirTestWatcher.copyResourceToRoot(Paths.get("join", "empty_part"));
+  }
+  
+  @Test
+  public void simpleTest() {
+ // src/test/resources/join/empty_part is acessible at 
dfs.root.`join/empty_part` from my test
+  }
+}
+```
+
+Or if you are extending 
[BaseTestQuery](../exec/java-exec/src/test/java/org/apache/drill/test/BaseDirTestWatcher.java)
+
+```
+public class TestMyClass extends BaseTestQuery {
+  @BeforeClass
+  public static void setupFiles() {
+dirTestWatcher.copyResourceToRoot(Paths.get("join", "empty_part"));
+  }
+  
+  @Test
+  public void simpleTest() {
+ // src/test/resources/join/empty_part is acessible at 
dfs.root.`join/empty_part` from my test
+  }
+}
+```
+
+### Using TPCH Data in Unit Tests
+
+TPCH data is accessible via the classpath storage plugin
+
+```
+cp.`tpch/customer.parquet`
+```
+
+## Generating Data Sets
 
 Review comment:
   Two other possibilities.
   
   To test a specific operator, there is the `MiniPlanUnitTestBase` and 
associated cases that Jinfeng created to test operators one by one. (The code 
sets up a plan, then uses that, along with a reader, to generate data that 
exercises the operator.) There is an opportunity here to generate data from a 
RowSet or consume the data as a RowSet for validation.
   
   The other form, for "operator" and "sub-operator" tests is the RowSet 
mechanism that can build up the exact set of vectors required. This is often 
the only way to get the more obscure types that no readers will produce.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For 

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468415#comment-16468415
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding 
more unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r186884623
 
 

 ##
 File path: docs/dev/GeneratedCode.md
 ##
 @@ -0,0 +1,42 @@
+# Testing Generated Code
+
+## Writing Unit Tests For Generated Code
+
+An example of unit testing generated code without running all of Drill is the 
**priorityQueueOrderingTest()** test in 
+[TopNBatchTest](../../exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TopN/TopNBatchTest.java).
 That test tests the 
+[PriorityQueueTemplate](../../exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/TopN/PriorityQueueTemplate.java)
 class separately from the rest of Drill.
+
+The main accomplished by creating instances of the following classes:
+
+ * 
[FunctionLookupContext](../../exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionLookupContext.java)
+ * 
[CodeCompiler](../../exec/java-exec/src/main/java/org/apache/drill/exec/compile/CodeCompiler.java)
+ 
+## Creating A 
[FunctionLookupContext](../../exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionLookupContext.java)
+
+```
+new FunctionImplementationRegistry(drillConfig)
+```
+
+## Creating A 
[CodeCompiler](../../exec/java-exec/src/main/java/org/apache/drill/exec/compile/CodeCompiler.java)
+
+ 1. Create a 
[DrillConfig](../../common/src/main/java/org/apache/drill/common/config/DrillConfig.java)
 as shown in [MockingComponents.md](MockingComponents.md).
+ 1. Create a 
[SystemOptionManager](../../exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SystemOptionManager.java)
 as shown in 
 
 Review comment:
   This is done automagically using the `OperatorFixture`. That class sets up 
everything needed to run a test without starting a server: configs, option 
manager, memory allocator, etc.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468421#comment-16468421
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding 
more unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r186886431
 
 

 ##
 File path: docs/dev/RowSetFramework.md
 ##
 @@ -0,0 +1,62 @@
+# RowSet Framework
 
 Review comment:
   Thanks much for writing this up!
   
   Suggestion: build on this to discuss the related ClusterFixture and 
OperatorFixture that let you run queries or operator-level code using the row 
sets. See the docs in my Wiki and in ExampleTest.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468456#comment-16468456
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding 
more unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r186933908
 
 

 ##
 File path: docs/dev/Testing.md
 ##
 @@ -2,27 +2,108 @@
 
 Drill makes extensive use of [JUnit](http://junit.org/junit4/) and other 
libraries for testing. This page provides pointers to the information you need 
to work with Drill tests. We don't repeat that information; you will want to 
follow the links and read the original material to get a complete understanding 
of the libraries that Drill uses.
 
-Caveat: information here about Drill is "reverse engineered" from the code; 
this page has not yet had the benefit of insight from the developers who 
created Drill's test structure.
-
-# Topics
-
-"Classic" Drill testing techniques
+# Writing Tests
 
+* [Test Data Sets](TestDataSets.md)
+* [Temp Directory Utilities](TempDirectories.md)
 * [Testing with JUnit](JUnit.md)
 * [Test Logging](TestLogging.md)
+
+## Deprecated Drill Testing Techniques
+
+This is a list of old Drill testing machinery that we have cleaner machinery 
for now.
+
+* [BaseTestQuery](BaseTestQuery.md): Deprecated, use 
[ClusterTest](ClusterTest.md) instead.
+
+## Legacy Drill Testing Techniques
+
+This is a list of old Drill testing machinery for which there is no other 
alternative at the moment.
+
 * [Testing with Physical Plans and Mock Data](LegacyTestingFrameworks.md)
 
-"Updated" Drill testing techniques
+## Latest Drill Testing Techniques
 
+* [RowSet Framework](RowSetFramework.md)
 * [Cluster Fixture Framework](ClusterFixture.md)
 * [Operator Fixture Framework](OperatorFixture.md)
-* [The Mock Record Reader](MockRecordReader.md)
+* [ClusterTest](ClusterTest.md)
+* [Single Operator Unit Test](PhysicalOpUnitTestBase.md)
+* [Mocking Components](MockingComponents.md)
+* [Generated Code](GeneratedCode.md)
+
+## Categories
+
+Currently Drill uses Travis to run smoke tests for every PR and commit. All of 
Drill's unit tests cannot be run on Travis because Drill's tests take longer to 
run than the
+maximum allowed container time for the free tier of Travis. In order to decide 
which tests are run on Travis and which tests are not, tests are categorized 
using JUnit's
+`@Category` annotation. Currently the following categories are excluded from 
Travis:
+
+  - **SlowTest:** Tests that are slow.
+  - **UnlikelyTest:** Tests that cover parts of the code that are rarely or 
never touched.
+  - **SecurityTest:** Corner case tests for security features.
+  
+To mark a test with a category you can do the following:
+
+```
+@Category(SlowTest.class)
+public class MyTest {
+  // Testing code
+}
+```
+
+To mark a test with multiple categories you can do the following:
+
+```
+@Category({SlowTest.class, SecurityTest.class})
+public class MyTest {
+  // Testing code
+}
+```
+
+# Running Tests
+
+Drill tests run in parallel. The Model for parallel execution is to divide 
test classes between multiple
+forked test processes. Each test process then runs the test classes assigned 
to it sequentially.
+
+## Speeding Up Test Runs
+
+There are a couple knobs you can turn to make tests run faster on your machine.
+
+ * **Maven Build Threads**: `-T `
+ * **Sure Fire Fork Count**: `-DforkCount=`
+ * **Test Categories**
+ 
+### -T
+
+Maven allows you to use multiple threads to compile sub modules. Also when 
running tests each build
+thread forks its own surefire process, so the tests for different submodules 
are run in parallel. In order
+to leverage this use the `-T` flag. By default this option is effectively.
 
 Review comment:
   Sorry, can't parse that last sentence. Is effectively... what?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in 

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468432#comment-16468432
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding 
more unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r186933819
 
 

 ##
 File path: docs/dev/Testing.md
 ##
 @@ -2,27 +2,108 @@
 
 Drill makes extensive use of [JUnit](http://junit.org/junit4/) and other 
libraries for testing. This page provides pointers to the information you need 
to work with Drill tests. We don't repeat that information; you will want to 
follow the links and read the original material to get a complete understanding 
of the libraries that Drill uses.
 
-Caveat: information here about Drill is "reverse engineered" from the code; 
this page has not yet had the benefit of insight from the developers who 
created Drill's test structure.
-
-# Topics
-
-"Classic" Drill testing techniques
+# Writing Tests
 
+* [Test Data Sets](TestDataSets.md)
+* [Temp Directory Utilities](TempDirectories.md)
 * [Testing with JUnit](JUnit.md)
 * [Test Logging](TestLogging.md)
+
+## Deprecated Drill Testing Techniques
+
+This is a list of old Drill testing machinery that we have cleaner machinery 
for now.
+
+* [BaseTestQuery](BaseTestQuery.md): Deprecated, use 
[ClusterTest](ClusterTest.md) instead.
+
+## Legacy Drill Testing Techniques
+
+This is a list of old Drill testing machinery for which there is no other 
alternative at the moment.
+
 * [Testing with Physical Plans and Mock Data](LegacyTestingFrameworks.md)
 
-"Updated" Drill testing techniques
+## Latest Drill Testing Techniques
 
+* [RowSet Framework](RowSetFramework.md)
 * [Cluster Fixture Framework](ClusterFixture.md)
 * [Operator Fixture Framework](OperatorFixture.md)
-* [The Mock Record Reader](MockRecordReader.md)
+* [ClusterTest](ClusterTest.md)
+* [Single Operator Unit Test](PhysicalOpUnitTestBase.md)
+* [Mocking Components](MockingComponents.md)
+* [Generated Code](GeneratedCode.md)
+
+## Categories
+
+Currently Drill uses Travis to run smoke tests for every PR and commit. All of 
Drill's unit tests cannot be run on Travis because Drill's tests take longer to 
run than the
+maximum allowed container time for the free tier of Travis. In order to decide 
which tests are run on Travis and which tests are not, tests are categorized 
using JUnit's
+`@Category` annotation. Currently the following categories are excluded from 
Travis:
+
+  - **SlowTest:** Tests that are slow.
+  - **UnlikelyTest:** Tests that cover parts of the code that are rarely or 
never touched.
+  - **SecurityTest:** Corner case tests for security features.
+  
+To mark a test with a category you can do the following:
+
+```
+@Category(SlowTest.class)
+public class MyTest {
+  // Testing code
+}
+```
+
+To mark a test with multiple categories you can do the following:
+
+```
+@Category({SlowTest.class, SecurityTest.class})
+public class MyTest {
+  // Testing code
+}
+```
+
+# Running Tests
+
+Drill tests run in parallel. The Model for parallel execution is to divide 
test classes between multiple
 
 Review comment:
   Model --> model


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468448#comment-16468448
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding 
more unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r186935208
 
 

 ##
 File path: docs/dev/DevDocs.md
 ##
 @@ -6,4 +6,8 @@ For information about configuring your development enviornment 
see [Environment.
 
 ## Testing
 
-For information about how to do Integration and Unit Testing in Drill see 
[Testing.md](Testing.md).
\ No newline at end of file
+For information about how to do Integration and Unit Testing in Drill see 
[Testing.md](Testing.md).
 
 Review comment:
   Integration, Unit Testing --> no capitalization


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468452#comment-16468452
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding 
more unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r186937205
 
 

 ##
 File path: docs/dev/RowSetFramework.md
 ##
 @@ -0,0 +1,62 @@
+# RowSet Framework
+
+The RowSet Framework allows you to create custom instances of:
+
+ * 
[BatchSchema](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/BatchSchema.java)
+ * 
[VectorContainer](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java)
+ * 
[RecordBatch](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/RecordBatch.java)
+ 
+It also allows the comparison of data container in 
[VectorContainer](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java)s
 through the use
+of the 
[RowSetComparison](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSetComparison.java)
 and 
+[RowSetUtilities](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSetUtilities.java).
+
+## Creating A 
[BatchSchema](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/BatchSchema.java)
+
+The 
[SchemaBuilder](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/schema/SchemaBuilder.java)
 class can be used
+to create an instance 
[BatchSchema](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/BatchSchema.java).
 An example 
+of how to to this can be found the **secondTest()** method of 
[ExampleTest](../../exec/java-exec/src/test/java/org/apache/drill/test/ExampleTest.java).
+
+## Creating Test 
[VectorContainer](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java)
+
+[VectorContainer](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java)s
 populated with data can be created with the 
+[RowSetBuilder](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSetBuilder.java).
 In order to use it do the following:
+
+ 1. Create an allocator
+```
+RootAllocator allocator = new RootAllocator(Long.MAX_VALUE);
 
 Review comment:
   Better: use the `OperatorFixture`. See `RowSetTest`. (That test exercises 
many ways to do things; it might be confusing as a result. Other tests show 
doing things the easiest possible way.)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468429#comment-16468429
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding 
more unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r186885946
 
 

 ##
 File path: docs/dev/MockingComponents.md
 ##
 @@ -0,0 +1,96 @@
+# Mocking Components
+
+There are a few techniques for mocking classes in unit tests:
+
+* Use a mocking library
+* Provide a simple impementation of an interface
+* Build a real instance of class using the class's builders / constructors
+* 
+
+## Mocking Libraries
+
+Drill uses two mocking libraries in order to mock classes.
+
+* [Mockito](http://site.mockito.org)
+* [JMockit](http://jmockit.github.io/tutorial.html) (Deprecated)
+
+While some tests use [JMockit](http://jmockit.github.io/tutorial.html), its 
usage has been deprecated since it doesn't work well with
+eclipse. Specifically any test that uses JMockit will fail when run with 
eclipse. So if you
+need to use a mocking library use [Mockito](http://site.mockito.org).
+
+## Mocking Contexts
+
+There are several contexts used throughout Drill, for a complete description 
of each and how
+they are used please see 
[FragmentContextImpl](../../exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContextImpl.java).
+
+When doing tests you can use the following mock contexts:
+
+  * 
[MockFragmentContext](../../exec/java-exec/src/test/java/org/apache/drill/test/OperatorFixture.java)
 is a simple mock implementation of
+  the 
[FragmentContext](../../exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java).
+
+## Creating An Instance of 
[QueryId](../../protocol/src/main/java/org/apache/drill/exec/proto/beans/QueryId.java)
+
+```
+UserBitShared.QueryId queryId = UserBitShared.QueryId.newBuilder()
+  .setPart1(1L)
+  .setPart2(2L)
+  .build();
+```
+
+## Creating 
[FragmentHandle](../../protocol/src/main/java/org/apache/drill/exec/proto/beans/FragmentHandle.java)
+
+```
+ExecProtos.FragmentHandle fragmentHandle = 
ExecProtos.FragmentHandle.newBuilder()
+  .setQueryId(queryId)
+  .setMinorFragmentId(1)
+  .setMajorFragmentId(2)
+  .build();
+```
+
+## Creating A 
[DrillConfig](../../common/src/main/java/org/apache/drill/common/config/DrillConfig.java)
+
+There are a few ways to create a 
[DrillConfig](../../common/src/main/java/org/apache/drill/common/config/DrillConfig.java).
 See the class for all of them. The simplest way is to
+ create a default config.
+
+```
+DrillConfig config = DrillConfig.create();
+```
+
+## Creating A 
[SpillSet](../../exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/spill/SpillSet.java)
+
+ 1. Create a 
[PhysicalOperator](../../exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/PhysicalOperator.java).
+```
+HashJoinPOP pop = new HashJoinPOP(null, null, null, JoinRelType.FULL);
+```
+ 1. Create a 
[DrillConfig](../../common/src/main/java/org/apache/drill/common/config/DrillConfig.java).
+ 1. Create a 
[FragmentHandle](../../protocol/src/main/java/org/apache/drill/exec/proto/beans/FragmentHandle.java)
 as described above.
+ 1. Create a 
[SpillSet](../../exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/spill/SpillSet.java).
+```
+SpillSet spillSet = new SpillSet(config, fragmentHandle, pop);
+```
+ 
+## Creating A 
[PersistentStoreProvider](../../exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/PersistentStoreProvider.java)
+
+```
+LocalPersistentStoreProvider provider = new 
LocalPersistentStoreProvider(drillConfig);
+provider.start();
+```
+ 
+## Creating A 
[LogicalPlanPersistence](../../logical/src/main/java/org/apache/drill/common/config/LogicalPlanPersistence.java)
+
+```
+LogicalPlanPersistence logicalPlanPersistence = 
PhysicalPlanReaderTestFactory.defaultLogicalPlanPersistence(drillConfig);
+```
+
+## Creating An Instance Of An Option Manager
 
 Review comment:
   Use the `OperatorFixture` or `ClusterFixture` which sets up all of this in a 
standard, repeatable way. Using those fixtures centralizes this stuff so we 
don't have to change it all if we again make changes to the option manager (as 
we did over the last year.)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
> 

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468454#comment-16468454
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding 
more unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r186937319
 
 

 ##
 File path: docs/dev/RowSetFramework.md
 ##
 @@ -0,0 +1,62 @@
+# RowSet Framework
+
+The RowSet Framework allows you to create custom instances of:
+
+ * 
[BatchSchema](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/BatchSchema.java)
+ * 
[VectorContainer](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java)
+ * 
[RecordBatch](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/RecordBatch.java)
+ 
+It also allows the comparison of data container in 
[VectorContainer](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java)s
 through the use
+of the 
[RowSetComparison](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSetComparison.java)
 and 
+[RowSetUtilities](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSetUtilities.java).
+
+## Creating A 
[BatchSchema](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/BatchSchema.java)
+
+The 
[SchemaBuilder](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/schema/SchemaBuilder.java)
 class can be used
+to create an instance 
[BatchSchema](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/BatchSchema.java).
 An example 
+of how to to this can be found the **secondTest()** method of 
[ExampleTest](../../exec/java-exec/src/test/java/org/apache/drill/test/ExampleTest.java).
+
+## Creating Test 
[VectorContainer](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java)
+
+[VectorContainer](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java)s
 populated with data can be created with the 
+[RowSetBuilder](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSetBuilder.java).
 In order to use it do the following:
+
+ 1. Create an allocator
+```
+RootAllocator allocator = new RootAllocator(Long.MAX_VALUE);
+```
+ 1. Create the desired BatchSchema using the 
[SchemaBuilder](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/schema/SchemaBuilder.java).
+```
+BatchSchema schema = new SchemaBuilder()
+.add(...)
+.add(...)
+.build();
+```
+ 1. Create a 
[RowSetBuilder](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSetBuilder.java)
 and add
 
 Review comment:
   As noted elsewhere, use the `TupleSchema` form to get a schema that can 
build complex types: `BatchSchema` is pretty limited beyond flat rows with 
scalar types.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468444#comment-16468444
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding 
more unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r186934976
 
 

 ##
 File path: common/src/test/java/org/apache/drill/test/DirTestWatcher.java
 ##
 @@ -34,7 +34,7 @@
  * A {@link DirTestWatcher} is added to a test by declaring it as a JUnit 
{@link org.junit.Rule}. A {@link org.junit.Rule} is
  * a piece of code that is run before and after every JUnit test marked with 
the {@link org.junit.Test} annotation. When the
  * {@link DirTestWatcher} is added to a test class the {@link DirTestWatcher} 
will create a temp directory before each of your
- * {@link org.junit.Test}s and optionally delete the temp directory after each 
of your {@link org.junit.Test}s. The temp directory
+ * {@link org.junit.Test}s and optionally delete the temp directory after each 
of your {@link org.junit.Test}s. The base temp directory
 
 Review comment:
   I forget, does the above form show just the base name or do you need to add 
that: `{@link org.junit.Test Test}`?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468443#comment-16468443
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding 
more unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r186936042
 
 

 ##
 File path: docs/dev/MockingComponents.md
 ##
 @@ -0,0 +1,96 @@
+# Mocking Components
+
+There are a few techniques for mocking classes in unit tests:
+
+* Use a mocking library
+* Provide a simple impementation of an interface
+* Build a real instance of class using the class's builders / constructors
+* 
+
+## Mocking Libraries
+
+Drill uses two mocking libraries in order to mock classes.
+
+* [Mockito](http://site.mockito.org)
+* [JMockit](http://jmockit.github.io/tutorial.html) (Deprecated)
+
+While some tests use [JMockit](http://jmockit.github.io/tutorial.html), its 
usage has been deprecated since it doesn't work well with
+eclipse. Specifically any test that uses JMockit will fail when run with 
eclipse. So if you
+need to use a mocking library use [Mockito](http://site.mockito.org).
+
+## Mocking Contexts
+
+There are several contexts used throughout Drill, for a complete description 
of each and how
+they are used please see 
[FragmentContextImpl](../../exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContextImpl.java).
+
+When doing tests you can use the following mock contexts:
+
+  * 
[MockFragmentContext](../../exec/java-exec/src/test/java/org/apache/drill/test/OperatorFixture.java)
 is a simple mock implementation of
+  the 
[FragmentContext](../../exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java).
 
 Review comment:
   Generally, mocking these is needed when doing lower-level tests. But, that 
is exactly the problem that `OperatorFixture` attempts to solve.
   
   The one possible exception is when testing the root operators such as 
network exchanges: those will need more functionality then the 
`OperatorFixture` provides. Maybe we need to invent a `NetworkFixture` class to 
add in however we want to setup networking, etc. for tests. Sounds like a great 
project for when someone starts mucking about with the root operators.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468427#comment-16468427
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding 
more unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r186934424
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/ops/RootFragmentContext.java
 ##
 @@ -17,6 +17,9 @@
  */
 package org.apache.drill.exec.ops;
 
+/**
+ * This interface is implemented by fragment roots.
 
 Review comment:
   Provides services needed by the fragment executor.
   
   (If that is accurate; I'm guessing that's what a "fragment root" is...)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468433#comment-16468433
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding 
more unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r186933958
 
 

 ##
 File path: docs/dev/Testing.md
 ##
 @@ -2,27 +2,108 @@
 
 Drill makes extensive use of [JUnit](http://junit.org/junit4/) and other 
libraries for testing. This page provides pointers to the information you need 
to work with Drill tests. We don't repeat that information; you will want to 
follow the links and read the original material to get a complete understanding 
of the libraries that Drill uses.
 
-Caveat: information here about Drill is "reverse engineered" from the code; 
this page has not yet had the benefit of insight from the developers who 
created Drill's test structure.
-
-# Topics
-
-"Classic" Drill testing techniques
+# Writing Tests
 
+* [Test Data Sets](TestDataSets.md)
+* [Temp Directory Utilities](TempDirectories.md)
 * [Testing with JUnit](JUnit.md)
 * [Test Logging](TestLogging.md)
+
+## Deprecated Drill Testing Techniques
+
+This is a list of old Drill testing machinery that we have cleaner machinery 
for now.
+
+* [BaseTestQuery](BaseTestQuery.md): Deprecated, use 
[ClusterTest](ClusterTest.md) instead.
+
+## Legacy Drill Testing Techniques
+
+This is a list of old Drill testing machinery for which there is no other 
alternative at the moment.
+
 * [Testing with Physical Plans and Mock Data](LegacyTestingFrameworks.md)
 
-"Updated" Drill testing techniques
+## Latest Drill Testing Techniques
 
+* [RowSet Framework](RowSetFramework.md)
 * [Cluster Fixture Framework](ClusterFixture.md)
 * [Operator Fixture Framework](OperatorFixture.md)
-* [The Mock Record Reader](MockRecordReader.md)
+* [ClusterTest](ClusterTest.md)
+* [Single Operator Unit Test](PhysicalOpUnitTestBase.md)
+* [Mocking Components](MockingComponents.md)
+* [Generated Code](GeneratedCode.md)
+
+## Categories
+
+Currently Drill uses Travis to run smoke tests for every PR and commit. All of 
Drill's unit tests cannot be run on Travis because Drill's tests take longer to 
run than the
+maximum allowed container time for the free tier of Travis. In order to decide 
which tests are run on Travis and which tests are not, tests are categorized 
using JUnit's
+`@Category` annotation. Currently the following categories are excluded from 
Travis:
+
+  - **SlowTest:** Tests that are slow.
+  - **UnlikelyTest:** Tests that cover parts of the code that are rarely or 
never touched.
+  - **SecurityTest:** Corner case tests for security features.
+  
+To mark a test with a category you can do the following:
+
+```
+@Category(SlowTest.class)
+public class MyTest {
+  // Testing code
+}
+```
+
+To mark a test with multiple categories you can do the following:
+
+```
+@Category({SlowTest.class, SecurityTest.class})
+public class MyTest {
+  // Testing code
+}
+```
+
+# Running Tests
+
+Drill tests run in parallel. The Model for parallel execution is to divide 
test classes between multiple
+forked test processes. Each test process then runs the test classes assigned 
to it sequentially.
+
+## Speeding Up Test Runs
+
+There are a couple knobs you can turn to make tests run faster on your machine.
+
+ * **Maven Build Threads**: `-T `
+ * **Sure Fire Fork Count**: `-DforkCount=`
+ * **Test Categories**
+ 
+### -T
+
+Maven allows you to use multiple threads to compile sub modules. Also when 
running tests each build
+thread forks its own surefire process, so the tests for different submodules 
are run in parallel. In order
+to leverage this use the `-T` flag. By default this option is effectively.
+
+Ex. In order to run the build using two maven threads use the following 
command.
+
+```
+mvn -T 2 clean install
+```
+
+### -DforkCount
+
+To run tests within a submodule in parallel you can use the `-DforkCount` 
option. By default this 
 
 Review comment:
   By default this.. what?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation 

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468425#comment-16468425
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding 
more unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r186932197
 
 

 ##
 File path: docs/dev/RowSetFramework.md
 ##
 @@ -0,0 +1,62 @@
+# RowSet Framework
+
+The RowSet Framework allows you to create custom instances of:
+
+ * 
[BatchSchema](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/BatchSchema.java)
+ * 
[VectorContainer](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java)
+ * 
[RecordBatch](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/RecordBatch.java)
 
 Review comment:
   True, but these are low-level constructs. BatchSchema is quite limited in 
its ability to model complex types. Instead, to use these "advanced" types, use 
the TupleMetadata class, which the schema builder also builds.
   
   RowSet wraps a VectorContainer.
   
   RecordBatch is badly misnamed, it is actually an operator.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468423#comment-16468423
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding 
more unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r186886168
 
 

 ##
 File path: docs/dev/PhysicalOpUnitTestBase.md
 ##
 @@ -0,0 +1,31 @@
+# Single Operator Unit Test
+
+It is possible to run an end to end test of an operator in isolation by 
extending 
+[PhysicalOpUnitTestBase](../../exec/java-exec/src/test/java/org/apache/drill/exec/physical/unit/PhysicalOpUnitTestBase.java).
+
+A simple example of an operator level unit test is the following:
+
+```
+public class BasicPhysicalOpUnitTest extends PhysicalOpUnitTestBase {
+
+ @Test
+ public void testSimpleProject() {
+   Project projectConf = new Project(parseExprs("x+5", "x"), null);
+   List jsonBatches = Lists.newArrayList(
+   "[{\"x\": 5 },{\"x\": 10 }]",
+   "[{\"x\": 20 },{\"x\": 30 },{\"x\": 40 }]");
+   opTestBuilder()
+   .physicalOperator(projectConf)
+   .inputDataStreamJson(jsonBatches)
+   .baselineColumns("x")
+   .baselineValues(10l)
+   .baselineValues(15l)
+   .baselineValues(25l)
+   .baselineValues(35l)
+   .baselineValues(45l)
+   .go();
 
 Review comment:
   Would be great to extend/modify/replace this with something based on, or 
similar to the `OperatorFixture`, using the `RowSet` stuff to allow much better 
comparisons than the above. The above, for example, is not very strict about 
values and (as I recall) cannot differentiate between nullable and non-nullable 
types.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468436#comment-16468436
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding 
more unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r186934484
 
 

 ##
 File path: 
exec/java-exec/src/test/java/org/apache/drill/test/BaseDirTestWatcher.java
 ##
 @@ -27,17 +27,31 @@
 import java.nio.file.Paths;
 
 /**
+ * Overview
  * 
  * This is a {@link DirTestWatcher} which creates all the temporary 
directories required by a Drillbit and the various dfs.* storage 
workspaces. It also
  * provides convenience methods that do the following:
- * 
  *
  * 
  *   Copy project files to temp directories. This is useful for copying 
the sample data into a temp directory.
  *   Copy resource files to temp.
  *   Updating parquet metadata files.
  * 
+ * 
+ *
+ * 
+ *   When used the {@link BaseDirTestWatcher} creates the following 
directories in the base temp directory (for a description of where the 
base temp directory
 
 Review comment:
   When used the --> The


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468441#comment-16468441
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding 
more unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r186934529
 
 

 ##
 File path: 
exec/java-exec/src/test/java/org/apache/drill/test/rowSet/file/JsonFileBuilder.java
 ##
 @@ -35,6 +36,16 @@
 import java.util.List;
 import java.util.Map;
 
+/**
+ * Overview
+ * 
+ *   This class is used to build a json file containing the data in a {@link 
RowSet}.
 
 Review comment:
   This class is used to build --> Builds


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468422#comment-16468422
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding 
more unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r186885712
 
 

 ##
 File path: docs/dev/MockingComponents.md
 ##
 @@ -0,0 +1,96 @@
+# Mocking Components
+
+There are a few techniques for mocking classes in unit tests:
+
+* Use a mocking library
+* Provide a simple impementation of an interface
+* Build a real instance of class using the class's builders / constructors
 
 Review comment:
   General advice: don't mock! Use the operator fixture instead. Design code to 
avoid mocking. Mocking is often a strong signal that the code has unwanted 
dependencies that should be refactored.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468455#comment-16468455
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding 
more unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r186934852
 
 

 ##
 File path: README.md
 ##
 @@ -20,7 +20,7 @@ Please see the [Apache Drill 
Website](http://drill.apache.org/) or the [Apache D
 
 
 ## Join the community!
-Apache Drill is an Apache Foundation project and is seeking all types of 
contributions.  Please say hello on the Apache Drill mailing list or join our 
Google Hangouts for more information.  (More information can be found at the 
Apache Drill website).
+Apache Drill is an Apache Foundation project and is seeking all types of 
contributions. Please say hello on the Apache Drill mailing list or join our 
Google Hangouts for more information. (More information can be found at the 
Apache Drill website).
 
 Review comment:
   Very nice. Maybe split into shorter lines for easier editing commenting.
   
   On the end "mailing list, join our Google Hangouts, or see the Apache Drill 
website for more information." Then, enclose "Apache Drill website" in a link 
to that website. Maybe have a link for the mail list and hangouts to the 
corresponding pages in the Community part of the site.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468451#comment-16468451
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding 
more unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r186937028
 
 

 ##
 File path: docs/dev/RowSetFramework.md
 ##
 @@ -0,0 +1,62 @@
+# RowSet Framework
+
+The RowSet Framework allows you to create custom instances of:
+
+ * 
[BatchSchema](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/BatchSchema.java)
+ * 
[VectorContainer](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java)
+ * 
[RecordBatch](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/RecordBatch.java)
+ 
+It also allows the comparison of data container in 
[VectorContainer](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java)s
 through the use
+of the 
[RowSetComparison](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSetComparison.java)
 and 
+[RowSetUtilities](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSetUtilities.java).
+
+## Creating A 
[BatchSchema](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/BatchSchema.java)
+
+The 
[SchemaBuilder](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/schema/SchemaBuilder.java)
 class can be used
+to create an instance 
[BatchSchema](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/BatchSchema.java).
 An example 
+of how to to this can be found the **secondTest()** method of 
[ExampleTest](../../exec/java-exec/src/test/java/org/apache/drill/test/ExampleTest.java).
+
+## Creating Test 
[VectorContainer](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java)
 
 Review comment:
   In general, as noted elsewhere, create a `TupleSchema`, then use that to 
generate a `RowSet` that can be filled with data. From that you can get the 
`VectorContainer` if you need it. You can also wrap an existing 
`VectorContainer` so you can compare it with a `RowSet` with expected results.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468450#comment-16468450
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding 
more unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r186934045
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java
 ##
 @@ -38,16 +38,8 @@
 import io.netty.buffer.DrillBuf;
 
 /**
- * Fragment context interface: separates implementation from definition.
- * Allows unit testing by mocking or reimplementing services with
- * test-time versions. The name is awkward, chosen to avoid renaming
- * the implementation class which is used in many places in legacy code.
- * New code should use this interface, and the names should eventually
- * be swapped with {@link FragmentContextImpl} becoming
- * FragmentContextImpl and this interface becoming
- * {@link FragmentContextImpl}.
+ * This interface provides the resources required by a non-exchange operator 
to execute.
 
 Review comment:
   This interface provides --> Provides


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468435#comment-16468435
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding 
more unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r186933018
 
 

 ##
 File path: docs/dev/TestDataSets.md
 ##
 @@ -0,0 +1,146 @@
+# Data Sets
+
+Drill includes several data sets for testing, and also provides some tools for 
generating test data sets.
+
+## Bundled Data Sets
+
+There are three primary data sets bundled with drill for testing:
+
+  - **Sample Data:** These are parquet files in the 
[sample-data](../sample-data) folder.
+  - **Resource Data:** These are data files in the 
[exec/java-exec/src/test/resources](../exec/java-exec/src/test/resources) fold.
+  - **TPCH Data:** These are trimmed down versions of the tpch data sets. They 
are retrieved and bundled
+  in the [contrib/data](../contrib/data) maven submodule. They are also 
accessible on [Apache Drill's S3 
bucket](http://apache-drill.s3.amazonaws.com/files/sf-0.01_tpc-h_parquet.tgz).
+  When unit tests are running all of the files in these data set are available 
from the classpath storage plugin. The tpch
+  files include:
+- **customer.parquet**
+- **lineitem.parquet**
+- **nation.parquet**
+- **orders.parquet**
+- **part.parquet**
+- **partsup.parquet**
+- **region.parquet**
+- **supplier.parquet**
+  
+### Using Sample Data in Unit Tests
+
+When using the 
[BaseDirTestWatcher](../exec/java-exec/src/test/java/org/apache/drill/test/BaseDirTestWatcher.java)
 you
+can make [sample-data](../sample-data) accessible from the ```dfs``` storage 
plugin by doing the following:
+
+```
+public class TestMyClass {
+  @ClassRule
+  public static final BaseDirTestWatcher dirTestWatcher = new 
BaseDirTestWatcher();
+  
+  @BeforeClass
+  public static void setupFiles() {
+dirTestWatcher.copyFileToRoot(Paths.get("sample-data", "region.parquet"));
+  }
+  
+  @Test
+  public void simpleTest() {
+ // dfs.root.`sample-data/region.parquet` will be accessible from my test
+  }
+}
+```
+
+Or if you are extending 
[BaseTestQuery](../exec/java-exec/src/test/java/org/apache/drill/test/BaseDirTestWatcher.java)
+
+```
+public class TestMyClass extends BaseTestQuery {
+  @BeforeClass
+  public static void setupFiles() {
+dirTestWatcher.copyFileToRoot(Paths.get("sample-data", "region.parquet"));
+  }
+  
+  @Test
+  public void simpleTest() {
+ // dfs.root.`sample-data/region.parquet` will be accessible from my test
+  }
+}
 
 Review comment:
   This can be done very simply with `ClusterFixture`. See 
`ClusterFixture.defineWorkspace(...)`


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468453#comment-16468453
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding 
more unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r186935750
 
 

 ##
 File path: docs/dev/MockingComponents.md
 ##
 @@ -0,0 +1,96 @@
+# Mocking Components
+
+There are a few techniques for mocking classes in unit tests:
+
+* Use a mocking library
+* Provide a simple impementation of an interface
+* Build a real instance of class using the class's builders / constructors
+* 
+
+## Mocking Libraries
+
+Drill uses two mocking libraries in order to mock classes.
+
+* [Mockito](http://site.mockito.org)
+* [JMockit](http://jmockit.github.io/tutorial.html) (Deprecated)
+
+While some tests use [JMockit](http://jmockit.github.io/tutorial.html), its 
usage has been deprecated since it doesn't work well with
+eclipse. Specifically any test that uses JMockit will fail when run with 
eclipse. So if you
+need to use a mocking library use [Mockito](http://site.mockito.org).
 
 Review comment:
   eclipse --> Eclipse
   
   Note that this may no longer be true if we have 1) Moved to JDK 8, and 2) 
updated JMockit to the latest version. Upgrading, if not done, will be a pain 
because the author decided to change many things and many tests need 
refactoring as a result. Probably another reason to avoid JMockit...


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468447#comment-16468447
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding 
more unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r186938309
 
 

 ##
 File path: docs/dev/Testing.md
 ##
 @@ -2,27 +2,108 @@
 
 Drill makes extensive use of [JUnit](http://junit.org/junit4/) and other 
libraries for testing. This page provides pointers to the information you need 
to work with Drill tests. We don't repeat that information; you will want to 
follow the links and read the original material to get a complete understanding 
of the libraries that Drill uses.
 
-Caveat: information here about Drill is "reverse engineered" from the code; 
this page has not yet had the benefit of insight from the developers who 
created Drill's test structure.
-
-# Topics
-
-"Classic" Drill testing techniques
+# Writing Tests
 
 Review comment:
   I've not read the full, original material. This stuff is really quite good.
   
   The one suggestion I'd make is to provide some overall design hints. For 
example, when should someone use `ClusterFixture` to create an integration test?
   
   When should they use `OperatorFixture` to do a lower-level test?
   
   The answer I followed was that any detailed testing of an operator or other 
internal mechanism can only be done at the unit level. (All the `RowSet`, 
`ExternalSort` and `ResultSetLoader` stuff are examples.) This may mean 
designing the code so it can be tested by controlling dependencies.
   
   Use the `RowSet` mechanism to set up a complete set of input data sets: 
types, cardinalities, nesting levels: whatever is relevant. As shown in the 
sort tests, clever coding can automate much of this stuff. There is even a 
function to generate data for every data type given an int so that can populate 
types with data that is easy to compare.
   
   Then, once the detailed internal tests are done, use integration tests to 
ensure the whole system works.
   
   At present, for most readers, integration tests are really the only approach 
because of the dependencies that Scan has. The in-flight work in the "batch 
sizing" stuff solves this problem, but only for newer readers designed to use 
that framework.
   
   The general rule is: test as close to your code as possible, refactoring and 
breaking dependencies where needed to accomplish this.
   
   On the other hand, if the test is more of a SQL or planner-level concept, 
then testing at the query level might be fine.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468424#comment-16468424
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding 
more unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r186885108
 
 

 ##
 File path: docs/dev/GeneratedCode.md
 ##
 @@ -0,0 +1,42 @@
+# Testing Generated Code
+
+## Writing Unit Tests For Generated Code
+
+An example of unit testing generated code without running all of Drill is the 
**priorityQueueOrderingTest()** test in 
+[TopNBatchTest](../../exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TopN/TopNBatchTest.java).
 That test tests the 
+[PriorityQueueTemplate](../../exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/TopN/PriorityQueueTemplate.java)
 class separately from the rest of Drill.
+
+The main accomplished by creating instances of the following classes:
+
+ * 
[FunctionLookupContext](../../exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionLookupContext.java)
+ * 
[CodeCompiler](../../exec/java-exec/src/main/java/org/apache/drill/exec/compile/CodeCompiler.java)
+ 
+## Creating A 
[FunctionLookupContext](../../exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionLookupContext.java)
+
+```
+new FunctionImplementationRegistry(drillConfig)
+```
+
+## Creating A 
[CodeCompiler](../../exec/java-exec/src/main/java/org/apache/drill/exec/compile/CodeCompiler.java)
+
+ 1. Create a 
[DrillConfig](../../common/src/main/java/org/apache/drill/common/config/DrillConfig.java)
 as shown in [MockingComponents.md](MockingComponents.md).
+ 1. Create a 
[SystemOptionManager](../../exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SystemOptionManager.java)
 as shown in 
+ [MockingComponents.md](MockingComponents.md).
+ 1. Create an instance of 
[CodeCompiler](../../exec/java-exec/src/main/java/org/apache/drill/exec/compile/CodeCompiler.java).
+```
+new CodeCompiler(drillConfig, optionManager)
+```
+
+## Debugging Generated Code
+
+It is possible to set break points in generated code.
+
+### Instructions For IntelliJ
+
+ 1. File→Project structure…→Modules→distribution→Sources → Add content root 
+ 1. Chose /tmp/drill/codegen 
+ 1. Mark it as Sources directory.
+ 1. Set saveCodeForDebugging(true) for the code generator of interest
+ 1. Run the unit test of interest
+ 1. Now some generated classes should appear in Intellij under the 
distribution module
+ 1. Set a break point in a generated class and run the unit test in debug mode
 
 Review comment:
   To step into the generated code, set a breakpoint just before we call into 
the setup method. Step into that method which will step into doSetup. This 
opens the generated code file (at least in Eclipse.)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468431#comment-16468431
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding 
more unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r186933636
 
 

 ##
 File path: docs/dev/Testing.md
 ##
 @@ -2,27 +2,108 @@
 
 Drill makes extensive use of [JUnit](http://junit.org/junit4/) and other 
libraries for testing. This page provides pointers to the information you need 
to work with Drill tests. We don't repeat that information; you will want to 
follow the links and read the original material to get a complete understanding 
of the libraries that Drill uses.
 
-Caveat: information here about Drill is "reverse engineered" from the code; 
this page has not yet had the benefit of insight from the developers who 
created Drill's test structure.
-
-# Topics
-
-"Classic" Drill testing techniques
+# Writing Tests
 
+* [Test Data Sets](TestDataSets.md)
+* [Temp Directory Utilities](TempDirectories.md)
 * [Testing with JUnit](JUnit.md)
 * [Test Logging](TestLogging.md)
+
+## Deprecated Drill Testing Techniques
+
+This is a list of old Drill testing machinery that we have cleaner machinery 
for now.
+
+* [BaseTestQuery](BaseTestQuery.md): Deprecated, use 
[ClusterTest](ClusterTest.md) instead.
 
 Review comment:
   Thanks! Should go without saying, but if there are any features in the old 
class that are missing from the new one, we should go ahead and move them 
across.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468442#comment-16468442
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding 
more unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r186933813
 
 

 ##
 File path: docs/dev/Testing.md
 ##
 @@ -2,27 +2,108 @@
 
 Drill makes extensive use of [JUnit](http://junit.org/junit4/) and other 
libraries for testing. This page provides pointers to the information you need 
to work with Drill tests. We don't repeat that information; you will want to 
follow the links and read the original material to get a complete understanding 
of the libraries that Drill uses.
 
-Caveat: information here about Drill is "reverse engineered" from the code; 
this page has not yet had the benefit of insight from the developers who 
created Drill's test structure.
-
-# Topics
-
-"Classic" Drill testing techniques
+# Writing Tests
 
+* [Test Data Sets](TestDataSets.md)
+* [Temp Directory Utilities](TempDirectories.md)
 * [Testing with JUnit](JUnit.md)
 * [Test Logging](TestLogging.md)
+
+## Deprecated Drill Testing Techniques
+
+This is a list of old Drill testing machinery that we have cleaner machinery 
for now.
+
+* [BaseTestQuery](BaseTestQuery.md): Deprecated, use 
[ClusterTest](ClusterTest.md) instead.
+
+## Legacy Drill Testing Techniques
+
+This is a list of old Drill testing machinery for which there is no other 
alternative at the moment.
+
 * [Testing with Physical Plans and Mock Data](LegacyTestingFrameworks.md)
 
-"Updated" Drill testing techniques
+## Latest Drill Testing Techniques
 
+* [RowSet Framework](RowSetFramework.md)
 * [Cluster Fixture Framework](ClusterFixture.md)
 * [Operator Fixture Framework](OperatorFixture.md)
-* [The Mock Record Reader](MockRecordReader.md)
+* [ClusterTest](ClusterTest.md)
+* [Single Operator Unit Test](PhysicalOpUnitTestBase.md)
+* [Mocking Components](MockingComponents.md)
+* [Generated Code](GeneratedCode.md)
 
 Review comment:
   Maybe add Jinfeng's mini plan framework. I don't think that has gotten the 
attention it deserves. In the figure, would be great to combine that with the 
OperatorFixture framework to avoid many of the ad-hoc bits.
   
   Also, `OperatorFixture` to test below the server level. Many of the row set 
and result set sets used this to great effect.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468446#comment-16468446
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding 
more unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r186933973
 
 

 ##
 File path: docs/dev/Testing.md
 ##
 @@ -2,27 +2,108 @@
 
 Drill makes extensive use of [JUnit](http://junit.org/junit4/) and other 
libraries for testing. This page provides pointers to the information you need 
to work with Drill tests. We don't repeat that information; you will want to 
follow the links and read the original material to get a complete understanding 
of the libraries that Drill uses.
 
-Caveat: information here about Drill is "reverse engineered" from the code; 
this page has not yet had the benefit of insight from the developers who 
created Drill's test structure.
-
-# Topics
-
-"Classic" Drill testing techniques
+# Writing Tests
 
+* [Test Data Sets](TestDataSets.md)
+* [Temp Directory Utilities](TempDirectories.md)
 * [Testing with JUnit](JUnit.md)
 * [Test Logging](TestLogging.md)
+
+## Deprecated Drill Testing Techniques
+
+This is a list of old Drill testing machinery that we have cleaner machinery 
for now.
+
+* [BaseTestQuery](BaseTestQuery.md): Deprecated, use 
[ClusterTest](ClusterTest.md) instead.
+
+## Legacy Drill Testing Techniques
+
+This is a list of old Drill testing machinery for which there is no other 
alternative at the moment.
+
 * [Testing with Physical Plans and Mock Data](LegacyTestingFrameworks.md)
 
-"Updated" Drill testing techniques
+## Latest Drill Testing Techniques
 
+* [RowSet Framework](RowSetFramework.md)
 * [Cluster Fixture Framework](ClusterFixture.md)
 * [Operator Fixture Framework](OperatorFixture.md)
-* [The Mock Record Reader](MockRecordReader.md)
+* [ClusterTest](ClusterTest.md)
+* [Single Operator Unit Test](PhysicalOpUnitTestBase.md)
+* [Mocking Components](MockingComponents.md)
+* [Generated Code](GeneratedCode.md)
+
+## Categories
+
+Currently Drill uses Travis to run smoke tests for every PR and commit. All of 
Drill's unit tests cannot be run on Travis because Drill's tests take longer to 
run than the
+maximum allowed container time for the free tier of Travis. In order to decide 
which tests are run on Travis and which tests are not, tests are categorized 
using JUnit's
+`@Category` annotation. Currently the following categories are excluded from 
Travis:
+
+  - **SlowTest:** Tests that are slow.
+  - **UnlikelyTest:** Tests that cover parts of the code that are rarely or 
never touched.
+  - **SecurityTest:** Corner case tests for security features.
+  
+To mark a test with a category you can do the following:
+
+```
+@Category(SlowTest.class)
+public class MyTest {
+  // Testing code
+}
+```
+
+To mark a test with multiple categories you can do the following:
+
+```
+@Category({SlowTest.class, SecurityTest.class})
+public class MyTest {
+  // Testing code
+}
+```
+
+# Running Tests
+
+Drill tests run in parallel. The Model for parallel execution is to divide 
test classes between multiple
+forked test processes. Each test process then runs the test classes assigned 
to it sequentially.
+
+## Speeding Up Test Runs
+
+There are a couple knobs you can turn to make tests run faster on your machine.
+
+ * **Maven Build Threads**: `-T `
+ * **Sure Fire Fork Count**: `-DforkCount=`
+ * **Test Categories**
+ 
+### -T
+
+Maven allows you to use multiple threads to compile sub modules. Also when 
running tests each build
+thread forks its own surefire process, so the tests for different submodules 
are run in parallel. In order
+to leverage this use the `-T` flag. By default this option is effectively.
+
+Ex. In order to run the build using two maven threads use the following 
command.
+
+```
+mvn -T 2 clean install
+```
+
+### -DforkCount
+
+To run tests within a submodule in parallel you can use the `-DforkCount` 
option. By default this 
+
+Ex. Run 4 test processes in parallel
+
+```
+mvn clean install -DforkCount=4
+```
+
+**Note:** The `-DforkCount` option interacts with `-T`. When use together each 
build thread (`-T`) gets 
 
 Review comment:
   use --> used


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy 

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468426#comment-16468426
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding 
more unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r186933202
 
 

 ##
 File path: docs/dev/TestDataSets.md
 ##
 @@ -0,0 +1,146 @@
+# Data Sets
+
+Drill includes several data sets for testing, and also provides some tools for 
generating test data sets.
+
+## Bundled Data Sets
+
+There are three primary data sets bundled with drill for testing:
+
+  - **Sample Data:** These are parquet files in the 
[sample-data](../sample-data) folder.
+  - **Resource Data:** These are data files in the 
[exec/java-exec/src/test/resources](../exec/java-exec/src/test/resources) fold.
 
 Review comment:
   fold --> folder
   
   These files are often crafted to show specific edge cases needed for some 
specific test.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468445#comment-16468445
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding 
more unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r186935543
 
 

 ##
 File path: docs/dev/MockingComponents.md
 ##
 @@ -0,0 +1,96 @@
+# Mocking Components
+
+There are a few techniques for mocking classes in unit tests:
+
+* Use a mocking library
+* Provide a simple impementation of an interface
+* Build a real instance of class using the class's builders / constructors
+* 
 
 Review comment:
   Extra asterisk


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468419#comment-16468419
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding 
more unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r186885551
 
 

 ##
 File path: docs/dev/LicenseHeaders.md
 ##
 @@ -0,0 +1,34 @@
+# License Headers
+
+Drill uses two license header checkers:
+
+* [Apache RAT Plugin](http://creadur.apache.org/rat/apache-rat-plugin/)
+* [License Maven Plugin](http://code.mycila.com/license-maven-plugin/)
+
+## Why Two?
+
+[Apache RAT Plugin](http://creadur.apache.org/rat/apache-rat-plugin/) is used 
because it is the standard license header
+checker for Apache projects. 
+
+[License Maven Plugin](http://code.mycila.com/license-maven-plugin/) performs 
stricter license checks and disallows
+putting license headers in java docs.
+
+## Doing License Checks
+
+The license checks are disabled locally by default and are enabled on Travis. 
If you'd like to perform
+license checks locally you can do the following:
+
+```
+ mvn license:check -Dlicense.skip=false
+```
+
+## Auto Formatting Headers
+
+If the license checks fail and you can't figure out what's wrong with your 
headers, you can auto-format
+your license headers with the following command:
+
+```
+mvn license:format -Dlicense.skip=false
+```
+
+This command will also add license headers to files without them.
 
 Review comment:
   This seems a bit of a risk... Will it only do this for Java files? RAT will 
check all files, even data files. If the license plugin adds headers to these 
files, the files can end up being corrupted. Maybe just add a note to check all 
changed files prior to commit to identify any issues and customize (what) to 
exclude unwanted files.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468420#comment-16468420
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding 
more unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r186885267
 
 

 ##
 File path: docs/dev/LicenseHeaders.md
 ##
 @@ -0,0 +1,34 @@
+# License Headers
+
+Drill uses two license header checkers:
+
+* [Apache RAT Plugin](http://creadur.apache.org/rat/apache-rat-plugin/)
+* [License Maven Plugin](http://code.mycila.com/license-maven-plugin/)
+
+## Why Two?
+
+[Apache RAT Plugin](http://creadur.apache.org/rat/apache-rat-plugin/) is used 
because it is the standard license header
+checker for Apache projects. 
+
+[License Maven Plugin](http://code.mycila.com/license-maven-plugin/) performs 
stricter license checks and disallows
+putting license headers in java docs.
 
 Review comment:
   Explain what this means: it simply means to start a license header with /* 
and not /**


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468438#comment-16468438
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding 
more unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r186934308
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContextImpl.java
 ##
 @@ -70,9 +69,26 @@
 import io.netty.buffer.DrillBuf;
 
 /**
- * Contextual objects required for execution of a particular fragment.
- * This is the implementation; use FragmentContext
- * in code to allow tests to use test-time implementations.
+ * 
+ *   This is the core Context which implements all the Context interfaces:
+ *
+ *   
+ * {@link FragmentContext}: A context provided to non-exchange 
operators.
+ * {@link ExchangeFragmentContext}: A context provided to exchange 
operators.
+ * {@link RootFragmentContext}: A context provided to fragment 
roots.
+ * {@link ExecutorFragmentContext}: A context used by the 
Drillbit.
+ *   
+ *
+ *   The interfaces above expose resources to varying degrees. They are 
ordered from most restrictive ({@link FragmentContext})
+ *   to least restrictive ({@link ExecutorFragmentContext}).
+ * 
+ * 
+ *   Since {@link FragmentContextImpl} implements all of the interfaces listed 
above, the facade pattern is used in order
+ *   to cast a {@link FragmentContextImpl} object to the desired interface 
where-ever it is needed. The facade pattern
+ *   is powerful since it allows us to easily create minimal mock context 
objects to be used in unit tests. Without
+ *   the use of interfaces and the facade pattern we would have to create a 
complete {@link FragmentContextImpl} object
+ *   to unit test any part of the code that depends on a context.
+ * 
 
 Review comment:
   General guideline: use the most narrow interface for the task. For example, 
"internal" operators don't need visibility to the networking functionality. 
Using the narrow interface allows unit testing without mocking.
   
   Often, the surrounding structure already has exposed the most narrow 
interface. If there are opportunities clean up older code, we can do so as 
needed to make testing easier.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468440#comment-16468440
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding 
more unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r186935126
 
 

 ##
 File path: docs/dev/ClusterTest.md
 ##
 @@ -0,0 +1,4 @@
+# ClusterTest
 
 Review comment:
   And `OperatorTest` for non-server (operator-level) tests. Sets up all the 
bits and pieces needed to work with Drill internals without an enclosing server.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468414#comment-16468414
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding 
more unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r186884221
 
 

 ##
 File path: docs/dev/GeneratedCode.md
 ##
 @@ -0,0 +1,42 @@
+# Testing Generated Code
+
+## Writing Unit Tests For Generated Code
+
+An example of unit testing generated code without running all of Drill is the 
**priorityQueueOrderingTest()** test in 
+[TopNBatchTest](../../exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TopN/TopNBatchTest.java).
 That test tests the 
+[PriorityQueueTemplate](../../exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/TopN/PriorityQueueTemplate.java)
 class separately from the rest of Drill.
+
+The main accomplished by creating instances of the following classes:
 
 Review comment:
   Awkward... The main (what?) accomplished...


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468449#comment-16468449
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding 
more unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r186936197
 
 

 ##
 File path: docs/dev/MockingComponents.md
 ##
 @@ -0,0 +1,96 @@
+# Mocking Components
+
+There are a few techniques for mocking classes in unit tests:
+
+* Use a mocking library
+* Provide a simple impementation of an interface
+* Build a real instance of class using the class's builders / constructors
+* 
+
+## Mocking Libraries
+
+Drill uses two mocking libraries in order to mock classes.
+
+* [Mockito](http://site.mockito.org)
+* [JMockit](http://jmockit.github.io/tutorial.html) (Deprecated)
+
+While some tests use [JMockit](http://jmockit.github.io/tutorial.html), its 
usage has been deprecated since it doesn't work well with
+eclipse. Specifically any test that uses JMockit will fail when run with 
eclipse. So if you
+need to use a mocking library use [Mockito](http://site.mockito.org).
+
+## Mocking Contexts
+
+There are several contexts used throughout Drill, for a complete description 
of each and how
+they are used please see 
[FragmentContextImpl](../../exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContextImpl.java).
+
+When doing tests you can use the following mock contexts:
+
+  * 
[MockFragmentContext](../../exec/java-exec/src/test/java/org/apache/drill/test/OperatorFixture.java)
 is a simple mock implementation of
+  the 
[FragmentContext](../../exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java).
+
+## Creating An Instance of 
[QueryId](../../protocol/src/main/java/org/apache/drill/exec/proto/beans/QueryId.java)
+
+```
+UserBitShared.QueryId queryId = UserBitShared.QueryId.newBuilder()
+  .setPart1(1L)
+  .setPart2(2L)
+  .build();
+```
+
+## Creating 
[FragmentHandle](../../protocol/src/main/java/org/apache/drill/exec/proto/beans/FragmentHandle.java)
+
+```
+ExecProtos.FragmentHandle fragmentHandle = 
ExecProtos.FragmentHandle.newBuilder()
+  .setQueryId(queryId)
+  .setMinorFragmentId(1)
+  .setMajorFragmentId(2)
+  .build();
+```
+
+## Creating A 
[DrillConfig](../../common/src/main/java/org/apache/drill/common/config/DrillConfig.java)
+
+There are a few ways to create a 
[DrillConfig](../../common/src/main/java/org/apache/drill/common/config/DrillConfig.java).
 See the class for all of them. The simplest way is to
+ create a default config.
+
+```
+DrillConfig config = DrillConfig.create();
 
 Review comment:
   See the `ConfigBuilder`. It is the guts behind the `ClusterFixture` and 
`OperatorFixture` for configs. It allows the fluent-style construction used in 
these two contexts.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468434#comment-16468434
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding 
more unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r186932374
 
 

 ##
 File path: docs/dev/RowSetFramework.md
 ##
 @@ -0,0 +1,62 @@
+# RowSet Framework
+
+The RowSet Framework allows you to create custom instances of:
+
+ * 
[BatchSchema](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/BatchSchema.java)
+ * 
[VectorContainer](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java)
+ * 
[RecordBatch](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/RecordBatch.java)
+ 
+It also allows the comparison of data container in 
[VectorContainer](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java)s
 through the use
+of the 
[RowSetComparison](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSetComparison.java)
 and 
+[RowSetUtilities](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSetUtilities.java).
+
+## Creating A 
[BatchSchema](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/BatchSchema.java)
+
+The 
[SchemaBuilder](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/schema/SchemaBuilder.java)
 class can be used
+to create an instance 
[BatchSchema](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/BatchSchema.java).
 An example 
+of how to to this can be found the **secondTest()** method of 
[ExampleTest](../../exec/java-exec/src/test/java/org/apache/drill/test/ExampleTest.java).
+
+## Creating Test 
[VectorContainer](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java)
+
+[VectorContainer](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java)s
 populated with data can be created with the 
+[RowSetBuilder](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSetBuilder.java).
 In order to use it do the following:
+
+ 1. Create an allocator
+```
+RootAllocator allocator = new RootAllocator(Long.MAX_VALUE);
+```
+ 1. Create the desired BatchSchema using the 
[SchemaBuilder](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/schema/SchemaBuilder.java).
+```
+BatchSchema schema = new SchemaBuilder()
+.add(...)
+.add(...)
+.build();
+```
+ 1. Create a 
[RowSetBuilder](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSetBuilder.java)
 and add
+records to it. Then build a 
[RowSet](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSet.java).
+```
+RowSet rowSet = new RowSetBuilder(allocator, schema)
+  .addRow(110, "green", new float[]{5.5f, 2.3f}, new String[]{"1a", "1b"})
+  .addRow(109, "blue", new float[]{1.5f}, new String[]{"2a"})
+  .addRow(108, "red", new float[]{-11.1f, 0.0f, .5f}, new String[]{"3a", 
"3b", "3c"})
+  .build();
+```
+ 1. Create a 
[VectorContainer](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java)
 from the 
+
[RowSet](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSet.java).
+```
+VectorContainer container = rowSet.container();
+```
+
+## Creating A Mock Record Batch With Data
+
+Create a 
[RowSet](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSet.java)
 and then create a
+[RowSetBatch](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSetBatch)
+
+```
+RecordBatch batch = new RowSetBatch(rowSet);
+```
 
 Review comment:
   All are good examples. Thanks much for this material!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is 

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468430#comment-16468430
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding 
more unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r186932267
 
 

 ##
 File path: docs/dev/RowSetFramework.md
 ##
 @@ -0,0 +1,62 @@
+# RowSet Framework
+
+The RowSet Framework allows you to create custom instances of:
+
+ * 
[BatchSchema](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/BatchSchema.java)
+ * 
[VectorContainer](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java)
+ * 
[RecordBatch](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/RecordBatch.java)
+ 
+It also allows the comparison of data container in 
[VectorContainer](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java)s
 through the use
+of the 
[RowSetComparison](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSetComparison.java)
 and 
 
 Review comment:
   The easiest way to use the above is with the RowSetUtilities.verify() method.
   
   In general, the best way to compare a `VectorContainer` is to wrap it in a 
`RowSet` using `OperatorFixture.wrap()`. This becomes very important for 
"advanced" types such as maps, unions, lists, etc.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468418#comment-16468418
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding 
more unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r186884833
 
 

 ##
 File path: docs/dev/GeneratedCode.md
 ##
 @@ -0,0 +1,42 @@
+# Testing Generated Code
+
+## Writing Unit Tests For Generated Code
+
+An example of unit testing generated code without running all of Drill is the 
**priorityQueueOrderingTest()** test in 
+[TopNBatchTest](../../exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TopN/TopNBatchTest.java).
 That test tests the 
+[PriorityQueueTemplate](../../exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/TopN/PriorityQueueTemplate.java)
 class separately from the rest of Drill.
+
+The main accomplished by creating instances of the following classes:
+
+ * 
[FunctionLookupContext](../../exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionLookupContext.java)
+ * 
[CodeCompiler](../../exec/java-exec/src/main/java/org/apache/drill/exec/compile/CodeCompiler.java)
+ 
+## Creating A 
[FunctionLookupContext](../../exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionLookupContext.java)
+
+```
+new FunctionImplementationRegistry(drillConfig)
+```
+
+## Creating A 
[CodeCompiler](../../exec/java-exec/src/main/java/org/apache/drill/exec/compile/CodeCompiler.java)
+
+ 1. Create a 
[DrillConfig](../../common/src/main/java/org/apache/drill/common/config/DrillConfig.java)
 as shown in [MockingComponents.md](MockingComponents.md).
+ 1. Create a 
[SystemOptionManager](../../exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SystemOptionManager.java)
 as shown in 
+ [MockingComponents.md](MockingComponents.md).
 
 Review comment:
   Mocking is a messy way to test. We have used it, but it has always caused 
issues. Please see the operator-fixture based tests for a clean, non-mocking 
way to test operators and units below the operator level.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467190#comment-16467190
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

vvysotskyi commented on issue #1251: DRILL-6249: Adding more unit testing 
documentation.
URL: https://github.com/apache/drill/pull/1251#issuecomment-387354275
 
 
   @ilooner, thanks for adding testing documentation! Changes in 
`GeneratedCode.md` looks good to me.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467187#comment-16467187
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

arina-ielchiieva commented on issue #1251: DRILL-6249: Adding more unit testing 
documentation.
URL: https://github.com/apache/drill/pull/1251#issuecomment-387353189
 
 
   LGTM.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466421#comment-16466421
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

ilooner commented on issue #1251: DRILL-6249: Adding more unit testing 
documentation.
URL: https://github.com/apache/drill/pull/1251#issuecomment-387190754
 
 
   @vvysotskyi Please review GeneratedCode.md
   @paul-rogers @arina-ielchiieva please review


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466414#comment-16466414
 ] 

ASF GitHub Bot commented on DRILL-6249:
---

ilooner opened a new pull request #1251: DRILL-6249: Adding more unit testing 
documentation.
URL: https://github.com/apache/drill/pull/1251
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in 
> Drill. Instead of writing the doc and having it be lost in Google Drive 
> somewhere I am going to add a Markdown doc to the drill repo and link to it 
> in the README.md. This is appropriate since these docs will only be used by 
> developers, and the way we unit test will change as the code changes. So the 
> unit testing docs should be kept in the same repo as the code so it can be 
> updated and kept in sync with the rest of Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)