Hi,
I was missing the implementations of operators, and I added the built in
EnumerableRules until I create my own, in order to fix it. However, the
plan I get from Volcano Optimizer is different from the one I get from
HepPlanner, although I use the same rules. My problem is about Projection
push-down. The hepPlanner pushes Projections to the bottom of the RelNode
tree, and VolcanoPlanner keeps them always at the top (doesn't push them
through joins). I use these rules in both :
ProjectRemoveRule.INSTANCE,
ProjectJoinTransposeRule.INSTANCE,
ProjectFilterTransposeRule.INSTANCE, /*it is better to use filter
first and then project*/
ProjectTableScanRule.INSTANCE,
ProjectWindowTransposeRule.INSTANCE,
ProjectMergeRule.INSTANCE
and
EnumerableRules.ENUMERABLE_TABLE_SCAN_RULE,
EnumerableRules.ENUMERABLE_PROJECT_RULE,
...
Finally , when trying to use aggregate I get this error:
Exception in thread "main" java.lang.AssertionError: Internal error: Error
while applying rule EnumerableTableScanRule, args
[rel#4:LogicalTableScan.NONE.[](table=[s, orders])]
at org.apache.calcite.util.Util.newInternal(Util.java:792)
at
org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:236)
at
org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:819)
at org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:334)
at org.apache.calcite.prepare.PlannerImpl.transform(PlannerImpl.java:308)
at calcite.VolcanoTester.main(VolcanoTester.java:106)
Caused by: java.lang.AssertionError: Internal error: Error occurred while
applying rule EnumerableTableScanRule
at org.apache.calcite.util.Util.newInternal(Util.java:792)
at
org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo(VolcanoRuleCall.java:148)
at
org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:225)
at
org.apache.calcite.rel.convert.ConverterRule.onMatch(ConverterRule.java:117)
at
org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:213)
... 4 more
Caused by: java.lang.NullPointerException
at org.apache.calcite.schema.Statistics$2.isKey(Statistics.java:70)
at
org.apache.calcite.prepare.RelOptTableImpl.isKey(RelOptTableImpl.java:288)
at
org.apache.calcite.rel.metadata.RelMdColumnUniqueness.areColumnsUnique(RelMdColumnUniqueness.java:76)
at GeneratedMetadataHandler_ColumnUniqueness.areColumnsUnique_$(Unknown
Source)
at GeneratedMetadataHandler_ColumnUniqueness.areColumnsUnique(Unknown
Source)
at
org.apache.calcite.rel.metadata.RelMetadataQuery.areColumnsUnique(RelMetadataQuery.java:461)
at
org.apache.calcite.rel.metadata.RelMdUtil.areColumnsDefinitelyUnique(RelMdUtil.java:216)
at
org.apache.calcite.rel.metadata.RelMdDistinctRowCount.getDistinctRowCount(RelMdDistinctRowCount.java:75)
at GeneratedMetadataHandler_DistinctRowCount.getDistinctRowCount_$(Unknown
Source)
at GeneratedMetadataHandler_DistinctRowCount.getDistinctRowCount(Unknown
Source)
at
org.apache.calcite.rel.metadata.RelMetadataQuery.getDistinctRowCount(RelMetadataQuery.java:700)
at
org.apache.calcite.rel.metadata.RelMdDistinctRowCount.getDistinctRowCount(RelMdDistinctRowCount.java:292)
at GeneratedMetadataHandler_DistinctRowCount.getDistinctRowCount_$(Unknown
Source)
at GeneratedMetadataHandler_DistinctRowCount.getDistinctRowCount(Unknown
Source)
at
org.apache.calcite.rel.metadata.RelMetadataQuery.getDistinctRowCount(RelMetadataQuery.java:700)
at
org.apache.calcite.rel.metadata.RelMdDistinctRowCount.getDistinctRowCount(RelMdDistinctRowCount.java:138)
at GeneratedMetadataHandler_DistinctRowCount.getDistinctRowCount_$(Unknown
Source)
at GeneratedMetadataHandler_DistinctRowCount.getDistinctRowCount(Unknown
Source)
at
org.apache.calcite.rel.metadata.RelMetadataQuery.getDistinctRowCount(RelMetadataQuery.java:700)
at
org.apache.calcite.rel.metadata.RelMdDistinctRowCount.getDistinctRowCount(RelMdDistinctRowCount.java:292)
at GeneratedMetadataHandler_DistinctRowCount.getDistinctRowCount_$(Unknown
Source)
at GeneratedMetadataHandler_DistinctRowCount.getDistinctRowCount(Unknown
Source)
at
org.apache.calcite.rel.metadata.RelMetadataQuery.getDistinctRowCount(RelMetadataQuery.java:700)
at
org.apache.calcite.rel.metadata.RelMdRowCount.getRowCount(RelMdRowCount.java:194)
at GeneratedMetadataHandler_RowCount.getRowCount_$(Unknown Source)
at GeneratedMetadataHandler_RowCount.getRowCount(Unknown Source)
at
org.apache.calcite.rel.metadata.RelMetadataQuery.getRowCount(RelMetadataQuery.java:201)
at org.apache.calcite.rel.core.Aggregate.computeSelfCost(Aggregate.java:304)
at
org.apache.calcite.rel.metadata.RelMdPercentageOriginalRows.getNonCumulativeCost(RelMdPercentageOriginalRows.java:162)
at
GeneratedMetadataHandler_NonCumulativeCost.getNonCumulativeCost_$(Unknown
Source)
at GeneratedMetadataHandler_NonCumulativeCost.getNonCumulativeCost(Unknown
Source)
at
org.apache.calcite.rel.metadata.RelMetadataQuery.getNonCumulativeCost(RelMetadataQuery.java:258)
at
org.apache.calcite.plan.volcano.VolcanoPlanner.getCost(VolcanoPlanner.java:1128)
at
org.apache.calcite.plan.volcano.RelSubset.propagateCostImprovements0(RelSubset.java:336)
at
org.apache.calcite.plan.volcano.RelSubset.propagateCostImprovements(RelSubset.java:319)
at
org.apache.calcite.plan.volcano.RelSubset.propagateCostImprovements0(RelSubset.java:348)
at
org.apache.calcite.plan.volcano.RelSubset.propagateCostImprovements(RelSubset.java:319)
at
org.apache.calcite.plan.volcano.RelSubset.propagateCostImprovements0(RelSubset.java:348)
at
org.apache.calcite.plan.volcano.RelSubset.propagateCostImprovements(RelSubset.java:319)
at
org.apache.calcite.plan.volcano.VolcanoPlanner.addRelToSet(VolcanoPlanner.java:1830)
at
org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl(VolcanoPlanner.java:1766)
at
org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:1032)
at
org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1052)
at
org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1942)
at
org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo(VolcanoRuleCall.java:136)
... 7 more
I define the Statistics in the tables I use like this:
public Statistic getStatistic() {
int rowCount = rows.size();
return Statistics.of(rowCount, null); //add List<ImmutableBitSet>
}
Thanks,
George
2016-10-16 7:28 GMT+03:00 Jungtaek Lim <kabh...@gmail.com>:
> Hi George,
>
> This patch is ported version (with small fixes) of Milinda's samza-sql
> implementation for Storm SQL.
> https://github.com/apache/storm/pull/1736
>
> In this patch I removed adding HepPlanner and just rely on Volcano Planner
> (so the patch may be the closer thing what you want).
> For now I also remove code regarding metadata since I'm not clear on how it
> works and what it helps, but I'll re-address once I can find its usage and
> benefits.
>
> Hope this helps.
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
> 2016년 10월 4일 (화) 오후 7:08, Γιώργος Θεοδωράκης <giwrgosrth...@gmail.com>님이
> 작성:
>
> > I think I did as you said:
> >
> > https://github.com/giwrgostheod/Calcite-Saber/blob/master/src/main/java/
> calcite/VolcanoTester.java
> >
> > and I get for every query I use:
> > Exception in thread "main"
> > org.apache.calcite.plan.RelOptPlanner$CannotPlanException: Node
> > [rel#10:Subset#2.NONE.[]] could not be implemented; planner state:
> > Root: rel#10:Subset#2.NONE.[]
> > Original rel:
> > ....
> > at
> >
> > org.apache.calcite.plan.volcano.RelSubset$CheapestPlanReplacer.visit(
> RelSubset.java:443)
> > at
> >
> > org.apache.calcite.plan.volcano.RelSubset.buildCheapestPlan(RelSubset.
> java:293)
> > at
> >
> > org.apache.calcite.plan.volcano.VolcanoPlanner.
> findBestExp(VolcanoPlanner.java:835)
> > at org.apache.calcite.tools.Programs$RuleSetProgram.run(
> Programs.java:334)
> > at org.apache.calcite.prepare.PlannerImpl.transform(
> PlannerImpl.java:308)
> > at calcite.VolcanoTester.main(VolcanoTester.java:77)
> >
> > My table's is defined here :
> >
> > https://github.com/giwrgostheod/Calcite-Saber/blob/master/src/main/java/
> calcite/utils/OrdersTableFactory.java
> >
> >
> > Thank you for your time,
> > George
> >
> >
> > 2016-10-04 2:38 GMT+03:00 Jordan Halterman <jordan.halter...@gmail.com>:
> >
> > > The link you provided is a pretty good example. Build a FrameworkConfig
> > > with your schema, parser config, and other information, and use it to
> > > create a Planner. That Planner uses a VolcanoPlanner internally. What's
> > > missing from that particular example is just the addition of programs.
> > > Programs are effectively sets of rules you will use to optimize your
> > query.
> > > So, to add your FilterProjectTransposeRule to the planner, call this
> when
> > > building your FrameworkConfig:
> > >
> > > .programs(Programs.ofRules(FilterProjectTransposeRule.INSTANCE))
> > >
> > > That adds your program(s) to the set of programs in the planner, and
> > those
> > > programs can be accessed to optimize the query. Use the planner to
> > parse()
> > > your query, validate() your query, and then convert() your query into a
> > > logical plan. Then call...
> > >
> > > RelTraitSet traitSet = planner.emptyTraitSet().
> replace(Convention.NONE);
> > > planner.transform(0. traitSet, logicalPlan);
> > >
> > > to apply the rules you added to the configuration. That should use the
> > > VolcanoPlanner to apply the rules you added in your Program. The trait
> > set
> > > that's passed to that method is the required output trait set. So, if
> you
> > > wanted to convert the logical plan into some physical convention, you'd
> > > pass your physical convention instead of Convention.NONE. I can respond
> > > with a full example if you need it in a bit. I just don't have the
> > capacity
> > > to write it ATM.
> > >
> > > On Mon, Oct 3, 2016 at 8:51 AM, Γιώργος Θεοδωράκης <
> > > giwrgosrth...@gmail.com>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > I want to parse an Sql query and transform it to an optimized
> > relational
> > > > plan (not convert it to physical !!) using calcite rules based on my
> > > > database schema and metadata. Right now, the only helpful example I
> > have
> > > > found for my purpose is taken from
> > > > https://github.com/milinda/samza-sql/blob/master/samza-
> > > >
> > sql-planner/src/main/java/org/apache/samza/sql/planner/QueryPlanner.java
> > > > ,
> > > > in which a simple Planner is used for parsing and validating Sql and
> a
> > > > HepPlanner is used for searching for an optimized plan based on
> > imported
> > > > rules.
> > > >
> > > > Is there any way to use in my case the VolcanoPlanner? The only
> > examples
> > > I
> > > > have seen so far from the test classes suggest that it should be used
> > for
> > > > converting relational expressions to physical ones. How can I make
> the
> > > > Volcano Planner "see" my SchemaPlus schema ,when I can only define
> > > > RelOptSchema? Can someone provide me with a complete example of using
> > > > Volcano Planner and adding rules, such
> > > > as FilterProjectTransposeRule.INSTANCE?
> > > >
> > > > Thanks in advance,
> > > > George
> > > >
> > >
> >
>
