GitHub user vkorukanti opened a pull request: https://github.com/apache/drill/pull/425
DRILL-1328: Support table statistics Patch attached to the JIRA is seems to be useful for generating table stats and using them for query planning. I rebased the patch to latest master, fixed few issues and added few tests. It still needs work to make it a full fledged feature, but I think the current state of the patch is good enough to commit and make improvements/fixes later. @jinfengni and @amansinha100 : Could you please review the patch? You can merge this pull request into a Git repository by running: $ git pull https://github.com/vkorukanti/drill DRILL-1328-r1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/425.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #425 ---- commit 079d109ee40d1be0ce6f5cfed5a091da41a477dc Author: Cliff Buchanan <cbucha...@maprtech.com> Date: 2014-08-21T21:59:53Z DRILL-1328: Support table statistics PRE: Add "append" concept to directory write. * This is so stats can be stored in [table].stats.drill and be appended to be writing a new file into the directory. FUNCS: Statistics functions as UDFs: Currently using FieldReader to ensure consistent output type so that Unpivot doesn't get confused. All stats columns should be Nullable, so that stats functions can return NULL when N/A. * custom versions of "count" that always return BigInt * HyperLogLog based NDV that returns BigInt that works only on VarChars * HyperLogLog with binary output that only works on VarChars OPS: Updated protobufs for new ops OPS: Implemented StatisticsAggregate OPS: Implemented StatisticsUnpivot ANALYZE: AnalyzeTable functionality * JavaCC syntax more-or-less copied from LucidDB. * (Basic) AnalyzePrule: DrillAnalyzeRel -> UnpivotPrel and StatsAggPrel ANALYZE: Add getMetadataTable() to AbstractSchema USAGE: Change field access in QueryWrapper USAGE: Add getDrillTable() to DrillScanRelBase and ScanPrel * since ScanPrel does not inherit from DrillScanRelBase, this requires adding a DrillTable to the constructor * This is done so that a custom ReflectiveRelMetadataProvider can access the DrillTable associated with Logical/Physical scans. USAGE: Attach DrillTableMetadata to DrillTable. * DrillTableMetadata represents the data scanned from a corresponding ".stats.drill" table * In order to avoid doing query execution right after the ".stats.drill" table is found, metadata is not actually collected until the MaterializationVisitor is used. ** Currently, the metadata source must be a string (so that a SQL query can be created). Doing this with a table is probably more complicated. ** Query is set up to extract only the most recent statistics results for each column. USAGE: Configure DrillJoinRelBase to use NDV metadata when available. USAGE: attach metadata to table USAGE: implement optiq provider ---- --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---