[GitHub] drill pull request: DRILL-4531: Add a Drill customized rule for pu...

2016-03-24 Thread jinfengni
Github user jinfengni commented on the pull request: https://github.com/apache/drill/pull/444#issuecomment-201151771 Adding assertion to DrillAggregateRel did not show new failure when using the default FilterAggregateTransposeRule. In the rule, copy() method is used to create new ins

[GitHub] drill pull request: DRILL-4531: Add a Drill customized rule for pu...

2016-03-24 Thread jacques-n
Github user jacques-n commented on the pull request: https://github.com/apache/drill/pull/444#issuecomment-201135594 In general, this pattern should fail earlier. There should be an AbstractConverter between these rels. We should probably make sure that a rels assert their convention

[GitHub] drill pull request: DRILL-4531: Add a Drill customized rule for pu...

2016-03-24 Thread jacques-n
Github user jacques-n commented on the pull request: https://github.com/apache/drill/pull/444#issuecomment-201124173 I think there is another bug here. DrillAggregateRel should assert when a none convention is passed in. If you add that assertion where do we fail? We should fix that

[GitHub] drill pull request: DRILL-4531: Add a Drill customized rule for pu...

2016-03-24 Thread jinfengni
Github user jinfengni commented on the pull request: https://github.com/apache/drill/pull/444#issuecomment-201105381 We did have other rules getting applied mixed convention. For DRILL-3855, I'll fix push filter/project past setop rules. I'll open another JIRA to go through all the ot

[GitHub] drill pull request: DRILL-4531: Add a Drill customized rule for pu...

2016-03-24 Thread amansinha100
Github user amansinha100 commented on the pull request: https://github.com/apache/drill/pull/444#issuecomment-201086825 +1. Are there other rules that also may be getting applied with mixed convention (Calcite logical and Drill logical) ? I am surprised that we don't encountered mor

[GitHub] drill pull request: DRILL-4531: Add a Drill customized rule for pu...

2016-03-24 Thread jinfengni
Github user jinfengni commented on a diff in the pull request: https://github.com/apache/drill/pull/444#discussion_r57406061 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillFilterAggregateTransposeRule.java --- @@ -0,0 +1,28 @@ +/** + * Li

[jira] [Created] (DRILL-4540) Queries with UNION fails with data type mis-match for limit 0

2016-03-24 Thread Krystal (JIRA)
Krystal created DRILL-4540: -- Summary: Queries with UNION fails with data type mis-match for limit 0 Key: DRILL-4540 URL: https://issues.apache.org/jira/browse/DRILL-4540 Project: Apache Drill Issue

Re: Proposal: Create v2 branch to work on breaking changes

2016-03-24 Thread Parth Chandra
What's our proposal for backward compatibility between 1.x and 2.x? My thoughts: Optional - Allow a mixture of 1.x and 2.x drillbits in a cluster. Required - 1.x clients should be able to talk to 2.x drillbits. On Thu, Mar 24, 2016 at 8:55 AM, Jacques Nadeau wrote: > There are some changes

Re: Proposal: Create v2 branch to work on breaking changes

2016-03-24 Thread Jason Altekruse
I think that there are a lot of good discussions and initial efforts for larger system changes that would be easier to move forward with on a "dev" branch. On top of the items listed so far, I think that we need to pick back up your proposal on workload management and put that work here as well. A

[jira] [Created] (DRILL-4539) Add support for Null Equality Joins

2016-03-24 Thread Jacques Nadeau (JIRA)
Jacques Nadeau created DRILL-4539: - Summary: Add support for Null Equality Joins Key: DRILL-4539 URL: https://issues.apache.org/jira/browse/DRILL-4539 Project: Apache Drill Issue Type: Improv

Re: Proposal: Create v2 branch to work on breaking changes

2016-03-24 Thread Jacques Nadeau
I also propose that we should turn on the union type as part of this as well. I've opened DRILL-4538 to track that. -- Jacques Nadeau CTO and Co-Founder, Dremio On Thu, Mar 24, 2016 at 8:55 AM, Jacques Nadeau wrote: > There are some changes that either have reviews pending or are in progress >

[jira] [Created] (DRILL-4538) Turn on Union type by default

2016-03-24 Thread Jacques Nadeau (JIRA)
Jacques Nadeau created DRILL-4538: - Summary: Turn on Union type by default Key: DRILL-4538 URL: https://issues.apache.org/jira/browse/DRILL-4538 Project: Apache Drill Issue Type: Improvement

Re: Drill on YARN

2016-03-24 Thread Jacques Nadeau
Your proposed allocation approach makes a lot of sense. I think it will solve a large number of use cases. Thanks for giving an overview of the different frameworks. I wonder if they got too focused on the simple use case Have you looked at LLama to see whether we could extend it for our needs

[GitHub] drill pull request: DRILL-4531: Add a Drill customized rule for pu...

2016-03-24 Thread jinfengni
Github user jinfengni commented on a diff in the pull request: https://github.com/apache/drill/pull/444#discussion_r57388288 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillFilterAggregateTransposeRule.java --- @@ -0,0 +1,28 @@ +/** + * Li

[GitHub] drill pull request: DRILL-4531: Add a Drill customized rule for pu...

2016-03-24 Thread jacques-n
Github user jacques-n commented on a diff in the pull request: https://github.com/apache/drill/pull/444#discussion_r57388058 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillFilterAggregateTransposeRule.java --- @@ -0,0 +1,28 @@ +/** + * Li

[GitHub] drill pull request: DRILL-4531: Add a Drill customized rule for pu...

2016-03-24 Thread jinfengni
Github user jinfengni commented on a diff in the pull request: https://github.com/apache/drill/pull/444#discussion_r57387788 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillFilterAggregateTransposeRule.java --- @@ -0,0 +1,28 @@ +/** + * Li

[GitHub] drill pull request: DRILL-4531: Add a Drill customized rule for pu...

2016-03-24 Thread jinfengni
Github user jinfengni commented on a diff in the pull request: https://github.com/apache/drill/pull/444#discussion_r57387288 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillFilterAggregateTransposeRule.java --- @@ -0,0 +1,28 @@ +/** + * Li

[jira] [Created] (DRILL-4537) FileSystemStoragePlugin optimizer rules leaking outside StoragePlugin

2016-03-24 Thread Jacques Nadeau (JIRA)
Jacques Nadeau created DRILL-4537: - Summary: FileSystemStoragePlugin optimizer rules leaking outside StoragePlugin Key: DRILL-4537 URL: https://issues.apache.org/jira/browse/DRILL-4537 Project: Apache

[GitHub] drill pull request: DRILL-4199: Add Support for HBase 1.X

2016-03-24 Thread jacques-n
Github user jacques-n commented on a diff in the pull request: https://github.com/apache/drill/pull/443#discussion_r57384658 --- Diff: contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/config/HBasePersistentStore.java --- @@ -25,36 +25,40 @@ import java.ut

[GitHub] drill pull request: DRILL-4531: Add a Drill customized rule for pu...

2016-03-24 Thread jacques-n
Github user jacques-n commented on a diff in the pull request: https://github.com/apache/drill/pull/444#discussion_r57384554 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillFilterAggregateTransposeRule.java --- @@ -0,0 +1,28 @@ +/** + * Li

[GitHub] drill pull request: DRILL-4199: Add Support for HBase 1.X

2016-03-24 Thread adityakishore
Github user adityakishore commented on a diff in the pull request: https://github.com/apache/drill/pull/443#discussion_r57384412 --- Diff: contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/config/HBasePersistentStore.java --- @@ -25,36 +25,40 @@ import jav

[GitHub] drill pull request: DRILL-4531: Add a Drill customized rule for pu...

2016-03-24 Thread amansinha100
Github user amansinha100 commented on a diff in the pull request: https://github.com/apache/drill/pull/444#discussion_r57380087 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillFilterAggregateTransposeRule.java --- @@ -0,0 +1,28 @@ +/** + *

[GitHub] drill pull request: DRILL-4199: Add Support for HBase 1.X

2016-03-24 Thread jacques-n
Github user jacques-n commented on a diff in the pull request: https://github.com/apache/drill/pull/443#discussion_r57379100 --- Diff: contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/config/HBasePersistentStore.java --- @@ -25,36 +25,40 @@ import java.ut

[GitHub] drill pull request: DRILL-4199: Add Support for HBase 1.X

2016-03-24 Thread adityakishore
Github user adityakishore commented on the pull request: https://github.com/apache/drill/pull/443#issuecomment-200991092 I'll take a look at the plugin life-cycle model and see if we can tie HBase connection life-cycle to it. As far as I know HBase follows a per-user connectio

[GitHub] drill pull request: DRILL-4199: Add Support for HBase 1.X

2016-03-24 Thread adityakishore
Github user adityakishore commented on a diff in the pull request: https://github.com/apache/drill/pull/443#discussion_r57376985 --- Diff: contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/config/HBasePersistentStore.java --- @@ -25,36 +25,40 @@ import jav

Re: [DISCUSS] Remove required type

2016-03-24 Thread Jacques Nadeau
Sorry if that is what you thought I was referring to. My main question at the top of this thread was about the customer impact. Since I'm now proposing a coupling so there is no regression I think your customer concern should be addressed. My statement about theoretical regressions was specificall

[GitHub] drill pull request: DRILL-4531: Add a Drill customized rule for pu...

2016-03-24 Thread jinfengni
GitHub user jinfengni opened a pull request: https://github.com/apache/drill/pull/444 DRILL-4531: Add a Drill customized rule for pushing filter past aggre… …gate You can merge this pull request into a Git repository by running: $ git pull https://github.com/jinfengni/incub

[GitHub] drill pull request: DRILL-4525: Allow SqlBetweenOperator to accept...

2016-03-24 Thread julianhyde
Github user julianhyde commented on the pull request: https://github.com/apache/drill/pull/439#issuecomment-200946428 Calcite allows strings to be implicitly converted to DATE and TIMESTAMP values in comparisons, as does the SQL standard. Even if, as @jinfengni says, the standard does

Re: [DISCUSS] Remove required type

2016-03-24 Thread Parth Chandra
For the record I disagree with the assertion that the fear is around 'theoretical' performance regressions. I mentioned in a previous post that I have customer data with required fields. As real as it gets. On Thu, Mar 24, 2016 at 9:01 AM, Jacques Nadeau wrote: > I've created DRILL-4534 to tr

[GitHub] drill pull request: DRILL-4514 : Add describe schema ...

2016-03-24 Thread arina-ielchiieva
Github user arina-ielchiieva commented on the pull request: https://github.com/apache/drill/pull/436#issuecomment-200934256 Created Calcite Jira - https://issues.apache.org/jira/browse/CALCITE-1168 --- If your project is set up for it, you can reply to this email and have your reply a

[GitHub] drill pull request: DRILL-4525: Allow SqlBetweenOperator to accept...

2016-03-24 Thread hsuanyi
Github user hsuanyi commented on the pull request: https://github.com/apache/drill/pull/439#issuecomment-200919168 In addition to @jinfengni's point: @sudheeshkatkam did a proposal to add a new SqlAvgAggFunction (the inference mechanism of the original is not ideal for Drill). It w

[GitHub] drill pull request: DRILL-4525: Allow SqlBetweenOperator to accept...

2016-03-24 Thread jacques-n
Github user jacques-n commented on the pull request: https://github.com/apache/drill/pull/439#issuecomment-200918110 Got it. We should open a discussion on this on the Calcite list. We shouldn't have to jump through hoops to change the implementation. Maybe the overridden operator tab

[GitHub] drill pull request: DRILL-4525: Allow SqlBetweenOperator to accept...

2016-03-24 Thread jinfengni
Github user jinfengni commented on the pull request: https://github.com/apache/drill/pull/439#issuecomment-200907542 My understanding is adding another operator will not work, since Calcite parser has put the built-in between operator in the sql expression tree[1]. The validator logic

[jira] [Created] (DRILL-4534) Replace declarative null type with observed null type in execution layer

2016-03-24 Thread Jacques Nadeau (JIRA)
Jacques Nadeau created DRILL-4534: - Summary: Replace declarative null type with observed null type in execution layer Key: DRILL-4534 URL: https://issues.apache.org/jira/browse/DRILL-4534 Project: Apa

[GitHub] drill pull request: DRILL-4525: Allow SqlBetweenOperator to accept...

2016-03-24 Thread jacques-n
Github user jacques-n commented on the pull request: https://github.com/apache/drill/pull/439#issuecomment-200902815 I'm confused by a couple things. One, it sounds there is an issue with the convertlet that Sean is trying to manage. Can't we just make are own convertlets? Second, if

Re: [DISCUSS] Remove required type

2016-03-24 Thread Jacques Nadeau
I've created DRILL-4534 to track this issue. The reality is that the fear around theoretical performance regressions is holding this back. By making this both the removal of the required type and the switch to columnar null evaluation, those fears should be allayed. I've created both as subtasks u

Re: [DISCUSS] Remove required type

2016-03-24 Thread Jacques Nadeau
My numbers show a declarative approach is unnecessary in execution. >> Having the right tools would help... Declarative is great in planning and should continue to exist. The right tools will continue to exist. It seems like a number of people here are worried performance of future features. I'm

[jira] [Created] (DRILL-4536) Modify Project such that NULL_IF_NULL handling operates columnar

2016-03-24 Thread Jacques Nadeau (JIRA)
Jacques Nadeau created DRILL-4536: - Summary: Modify Project such that NULL_IF_NULL handling operates columnar Key: DRILL-4536 URL: https://issues.apache.org/jira/browse/DRILL-4536 Project: Apache Dril

[jira] [Created] (DRILL-4535) Remove declarative null type

2016-03-24 Thread Jacques Nadeau (JIRA)
Jacques Nadeau created DRILL-4535: - Summary: Remove declarative null type Key: DRILL-4535 URL: https://issues.apache.org/jira/browse/DRILL-4535 Project: Apache Drill Issue Type: Sub-task

Proposal: Create v2 branch to work on breaking changes

2016-03-24 Thread Jacques Nadeau
There are some changes that either have reviews pending or are in progress that would require breaking changes to Drill core. Examples Include: DRILL-4455 (arrow integration) DRILL-4417 (jdbc/odbc/rpc changes) DRILL-4534 (improve null performance) I've created a new 2.0.0 release version in JIRA

Re: [DISCUSS] Remove required type

2016-03-24 Thread Aman Sinha
With regard to the following: *>> The only time we use the "required" path is if the underlying data >> guarantees that all the data will be non-null. I believe that path is >> rarely used, poorly tested and provides only a small gain in performance >> when used.* The main reason this code path

[jira] [Created] (DRILL-4533) Error should be more informative when creating/updating storage plugin definition

2016-03-24 Thread Chris Matta (JIRA)
Chris Matta created DRILL-4533: -- Summary: Error should be more informative when creating/updating storage plugin definition Key: DRILL-4533 URL: https://issues.apache.org/jira/browse/DRILL-4533 Project:

[GitHub] drill pull request: DRILL-4525: Allow SqlBetweenOperator to accept...

2016-03-24 Thread jinfengni
Github user jinfengni commented on the pull request: https://github.com/apache/drill/pull/439#issuecomment-200883747 Looks like the root cause of the problem is Calcite does not allow the comparison of Date and Timestamp. {code} select CAST('1990-01-01' AS DATE) < CAST('20

[GitHub] drill pull request: DRILL-4364: Image Metadata Format Plugin

2016-03-24 Thread zfong
Github user zfong commented on the pull request: https://github.com/apache/drill/pull/367#issuecomment-200871277 @vkorukanti, @jaltekruse, @adityakishore - since each of you have written format plugins in the past, can one of you help code review this change? --- If your project is s

Re: REST API with Basic Auth

2016-03-24 Thread John Omernik
Glad I could help. Thanks for the feedback. On Thu, Mar 24, 2016 at 9:06 AM, Jean-Claude Cote wrote: > Works great. Thanks John. > > On Wed, 23 Mar 2016 at 20:53 Jean-Claude Cote wrote: > > > Hey John, > > > > I looked at the Drill code and it does use the Jetty FormAuthenticator > and > > no

Re: REST API with Basic Auth

2016-03-24 Thread Jean-Claude Cote
Works great. Thanks John. On Wed, 23 Mar 2016 at 20:53 Jean-Claude Cote wrote: > Hey John, > > I looked at the Drill code and it does use the Jetty FormAuthenticator and > not the BasicAuthenticator. So what I was trying will not work. > > I'll do as you suggested and post the login form to get