Re: Announcing new committer: Ellen Friedman

Congrats Ellen! 2015-11-23 8:56 GMT-08:00 Hsuan Yi Chu : > Congrats! > > On Mon, Nov 23, 2015 at 8:50 AM, Sudheesh Katkam > wrote: > > > Congratulations and welcome, Ellen! > > > > > On Nov 23, 2015, at 12:32 AM, Julian Hyde wrote: >

[jira] [Created] (DRILL-4122) Create unit test suite for checking quality of hashing for hash based operators

Aman Sinha created DRILL-4122: - Summary: Create unit test suite for checking quality of hashing for hash based operators Key: DRILL-4122 URL: https://issues.apache.org/jira/browse/DRILL-4122 Project:

Moving directory based pruning to fire earlier

As part of DRILL-3996 Jinfeng mentioned that he plans to move the directory based pruning rule earlier than column based pruning. I want to expand on that a little, provide the motivation and gather thoughts/ feedback. Currently both the

Re: Announcing new committer: Ellen Friedman

Congratulations, Ellen! Thanks for all you’ve done for Drill so far. Julian > On Nov 22, 2015, at 5:50 PM, Worthy LaFollette wrote: > > Congrats, Welcome! > > On Sun, Nov 22, 2015 at 6:38 PM, Jacques Nadeau wrote: > >> The Apache Drill PMC is very

Re: Announcing new committer: Ellen Friedman

Congrats Ellen. To the point, we wouldn’t have had Drill as a topic at @DfwDataScience and @DallasMongoDB if she had not evangelized with us. :) SCott > On Nov 22, 2015, at 6:55 PM, Andrew Brust > wrote: > > Excellent news! > > -Original Message-

[ANNOUNCE] Release of Apache Drill 1.3.0

It is our pleasure to announce the 1.3.0 release of Apache Drill. This release includes a number new features as well as the typical plethora of improvements and bug fixes. Highlights include: - Enhanced Amazon S3 Support - Support for Heterogeneous Types when querying JSON or MongoDB

[GitHub] drill pull request: DRILL-3543: Add stats for external sort to a q...

Github user sudheeshkatkam commented on the pull request: https://github.com/apache/drill/pull/280#issuecomment-159083286 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

Re: Moving directory based pruning to fire earlier

Currently all rules based on Calcite logical rels and Drill logical rels are put together and are fired together. As part of DRILL-3996, Jinfeng will break it down into different phases. I should be able to take advantage of this and move the directory based partition pruning to fire based on

[GitHub] drill pull request: DRILL-3543: Add stats for external sort to a q...

Github user asfgit closed the pull request at: https://github.com/apache/drill/pull/280 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

Re: Drill Test Framework configuration file update

I think it's a good change to parameterize the whole connection string. This also allows us to easily switch to other data sources. +1 On Mon, Nov 23, 2015 at 3:08 PM, rahul challapalli < challapallira...@gmail.com> wrote: > Drillers, > > In the extended set of tests [1], we are directly

[GitHub] drill pull request: DRILL-3543: Add stats for external sort to a q...

Github user adeneche commented on the pull request: https://github.com/apache/drill/pull/280#issuecomment-159081152 @sudheeshkatkam can you please review ? thx --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

Re: Moving directory based pruning to fire earlier

Does that mean we would use hep planner to do directory pruning as the first stage of logical planning? I think it does make sense to allow the rules, which can definitely reduce the cost be fired before volcano. How about expression reduction? I believe sometimes pruning need the simplified

Re: Moving directory based pruning to fire earlier

Anybody think we should just get rid of Drels (Rel > Drel > Prel) and use Calcite's logical representation directly (Rel > Prel)? -- Jacques Nadeau CTO and Co-Founder, Dremio On Mon, Nov 23, 2015 at 1:57 PM, Mehant Baid wrote: > Currently all rules based on Calcite

Drill Test Framework configuration file update

Drillers, In the extended set of tests [1], we are directly connecting to a single drillbit through jdbc instead of talking to zookeeper. This was done as a workaround for DRILL-3171. Since the fix for DRILL-3171 has been committed now, I am updating the framework to use zookeeper. As part of

[GitHub] drill pull request: DRILL-4056: Avro corruption bug with UTF-8 str...

Github user jaltekruse commented on the pull request: https://github.com/apache/drill/pull/266#issuecomment-159106823 This was merged into the 1.3 release branch, but I just realized it was not merged into master. Will get it in shortly. --- If your project is set up for it, you can

[GitHub] drill pull request: DRILL-4048: Fix reading required dictionary en...

Github user jaltekruse commented on the pull request: https://github.com/apache/drill/pull/247#issuecomment-159107022 Fixed in a5a1aa6b487d0642641ce70f2c2500465956f9ec --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] drill pull request: DRILL-4056: Avro corruption bug with UTF-8 str...

GitHub user jaltekruse reopened a pull request: https://github.com/apache/drill/pull/266 DRILL-4056: Avro corruption bug with UTF-8 strings You can merge this pull request into a Git repository by running: $ git pull https://github.com/jaltekruse/incubator-drill

Re: Moving directory based pruning to fire earlier

@Jacques, are you talking about removing the convention DRILL_LOGICAL? DrillRel extends Calcite's LogialRel. It overrides some LogicalRel's methods, and adds new methods. Therefore, even we remove DRILL_LOGICAL convention, we still have to maintain a set of extended class from Calcite Logical.

Re: Moving directory based pruning to fire earlier

Yes, my suggestion is removal of DRILL_LOGICAL. @Hsuan, this is independent from the number of phases and I'm not suggesting changing that. My main thought was: if we only need to override one or two rels, do only that rather than having a wholesale copy of every operator with a bunch of basic

Re: Moving directory based pruning to fire earlier

Yes. You don’t need an “implement” method (or yours can just throw). You could use your own serialization to/from JSON or you could use RelJsonWriter/RelJsonReader. Julian > On Nov 23, 2015, at 5:31 PM, Jacques Nadeau wrote: > > We could create serializers and

[GitHub] drill pull request: DRILL-4056: Avro corruption bug with UTF-8 str...

Github user jaltekruse closed the pull request at: https://github.com/apache/drill/pull/266 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[jira] [Created] (DRILL-4124) Make all uses of AutoCloseables use addSuppressed exceptions to avoid noise in logs

Julien Le Dem created DRILL-4124: Summary: Make all uses of AutoCloseables use addSuppressed exceptions to avoid noise in logs Key: DRILL-4124 URL: https://issues.apache.org/jira/browse/DRILL-4124

[GitHub] drill pull request: DRILL-4124: Make all uses of AutoCloseables us...

GitHub user julienledem opened a pull request: https://github.com/apache/drill/pull/281 DRILL-4124: Make all uses of AutoCloseables use addSuppressed exceptions to avoid noise in logs You can merge this pull request into a Git repository by running: $ git pull

Re: Moving directory based pruning to fire earlier

I’m not sure what properties / behavior you want to override but remember that Calcite specifies a lot of brings as traits or metadata. For example, “double RelNode.getRows()" is deprecated and you would these days use RelMetadataQuery.getRowCount(). You would not need to sub-class a RelNode

Re: Drill Test Framework configuration file update

+1 Note that if this breaking change goes in, all current users must update the config file to have this new property. > On Nov 23, 2015, at 3:21 PM, Chun Chang wrote: > > I think it's a good change to parameterize the whole connection string. > This also allows us to

Re: Moving directory based pruning to fire earlier

We could create serializers and deserializers for the logical plan stuff. It looks like we can resolve the costing through metadata providers unless I misunderstood what Julian was suggesting. -- Jacques Nadeau CTO and Co-Founder, Dremio On Mon, Nov 23, 2015 at 5:12 PM, Jinfeng Ni

Build failed in Jenkins: drill-scm #597

See Changes: [adeneche] DRILL-3543: Add stats for external sort to a query profile -- [...truncated 4098 lines...] [INFO] [INFO] --- maven-enforcer-plugin:1.3.1:enforce (avoid_bad_dependencies) @

Re: Drill Test Framework configuration file update

We definitely could make it backward compatible. But I feel forcing the change may not be a bad thing. On Mon, Nov 23, 2015 at 3:32 PM, Sudheesh Katkam wrote: > +1 > > Note that if this breaking change goes in, all current users must update > the config file to have this

[GitHub] drill pull request: DRILL-4048: Fix reading required dictionary en...

Github user jaltekruse closed the pull request at: https://github.com/apache/drill/pull/247 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[jira] [Created] (DRILL-4125) Illegal argument exception during merge join

Victoria Markman created DRILL-4125: --- Summary: Illegal argument exception during merge join Key: DRILL-4125 URL: https://issues.apache.org/jira/browse/DRILL-4125 Project: Apache Drill

Re: Moving directory based pruning to fire earlier

@Jacaues, Every DrillLogicalRel has to override computeSelfCost(), and implement implement() method. The latter is to get Logical Plan, which is one of three input types Drill should accept (SQL, Logical Plan, Physical Plan). So, for now, we do have to override/exend all DrillLogicalRel. On

[GitHub] drill pull request: DRILL-4094: Respect skipTests in JDBC plugin t...

Github user asfgit closed the pull request at: https://github.com/apache/drill/pull/261 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

Jenkins build is back to normal : drill-scm #598

See

Re: Moving directory based pruning to fire earlier

You can carry on using your own formula, but move the formula into a metadata provider. You just don’t need to create a subclass in order for it to get called. For example, if you’ve written public class DrillLogicalFilter extends LogicalFilter { public double getRows() { return

[GitHub] drill pull request: DRILL-4119: Modify hash32 functions to combine...

GitHub user amansinha100 opened a pull request: https://github.com/apache/drill/pull/279 DRILL-4119: Modify hash32 functions to combine the msb and lsb bytes … …of a 64-bit hash value (previously, we were casting to integer). - Use this new set of functions (for all

Re: Announcing new committer: Ellen Friedman

Congratulations and welcome, Ellen! > On Nov 23, 2015, at 12:32 AM, Julian Hyde wrote: > > Congratulations, Ellen! Thanks for all you’ve done for Drill so far. > > Julian > > >> On Nov 22, 2015, at 5:50 PM, Worthy LaFollette wrote: >> >> Congrats,

[GitHub] drill pull request: DRILL-4119: Modify hash32 functions to combine...

Github user mehant commented on a diff in the pull request: https://github.com/apache/drill/pull/279#discussion_r45643009 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/Hash32AsDouble.java --- @@ -0,0 +1,340 @@ +/** + * Licensed to the Apache

Re: Moving directory based pruning to fire earlier

The general idea of multi-phase pruning makes sense to me. I am wondering, though, are we referring to introducing a new planning phase before the logical or separating out the logic so as to make directory pruning kick off ahead of column partitioning? 2015-11-23 10:33 GMT-08:00 Mehant Baid

[jira] [Created] (DRILL-4123) DirectoryExplorers should refer to fully qualified variable names

Hanifi Gunes created DRILL-4123: --- Summary: DirectoryExplorers should refer to fully qualified variable names Key: DRILL-4123 URL: https://issues.apache.org/jira/browse/DRILL-4123 Project: Apache Drill

Re: Announcing new committer: Ellen Friedman

Congrats Ellen :D On Mon, Nov 23, 2015 at 8:50 AM, Sudheesh Katkam wrote: > Congratulations and welcome, Ellen! > > > On Nov 23, 2015, at 12:32 AM, Julian Hyde wrote: > > > > Congratulations, Ellen! Thanks for all you’ve done for Drill so far. > > > >

Re: Announcing new committer: Ellen Friedman

Congrats! On Mon, Nov 23, 2015 at 8:50 AM, Sudheesh Katkam wrote: > Congratulations and welcome, Ellen! > > > On Nov 23, 2015, at 12:32 AM, Julian Hyde wrote: > > > > Congratulations, Ellen! Thanks for all you’ve done for Drill so far. > > > > Julian > >