I fixed the second error by changing my Statistics to:
public Statistic getStatistic() {
int rowCount = rows.size();
return Statistics.of(rowCount, ImmutableList.<ImmutableBitSet>of());
}
Any suggestions of a better solution are welcome. The project push-down
problem still exists...
2016-10-27 15:42 GMT+03:00 Γιώργος Θεοδωράκης <giwrgosrth...@gmail.com>:
> Hi,
> I was missing the implementations of operators, and I added the built in
> EnumerableRules until I create my own, in order to fix it. However, the
> plan I get from Volcano Optimizer is different from the one I get from
> HepPlanner, although I use the same rules. My problem is about Projection
> push-down. The hepPlanner pushes Projections to the bottom of the RelNode
> tree, and VolcanoPlanner keeps them always at the top (doesn't push them
> through joins). I use these rules in both :
>
> ProjectRemoveRule.INSTANCE,
> ProjectJoinTransposeRule.INSTANCE,
> ProjectFilterTransposeRule.INSTANCE, /*it is better to use filter
> first and then project*/
> ProjectTableScanRule.INSTANCE,
> ProjectWindowTransposeRule.INSTANCE,
> ProjectMergeRule.INSTANCE
>
> and
> EnumerableRules.ENUMERABLE_TABLE_SCAN_RULE,
> EnumerableRules.ENUMERABLE_PROJECT_RULE,
> ...
>
> Finally , when trying to use aggregate I get this error:
>
> Exception in thread "main" java.lang.AssertionError: Internal error: Error
> while applying rule EnumerableTableScanRule, args
> [rel#4:LogicalTableScan.NONE.[](table=[s, orders])]
> at org.apache.calcite.util.Util.newInternal(Util.java:792)
> at org.apache.calcite.plan.volcano.VolcanoRuleCall.
> onMatch(VolcanoRuleCall.java:236)
> at org.apache.calcite.plan.volcano.VolcanoPlanner.
> findBestExp(VolcanoPlanner.java:819)
> at org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:334)
> at org.apache.calcite.prepare.PlannerImpl.transform(PlannerImpl.java:308)
> at calcite.VolcanoTester.main(VolcanoTester.java:106)
> Caused by: java.lang.AssertionError: Internal error: Error occurred while
> applying rule EnumerableTableScanRule
> at org.apache.calcite.util.Util.newInternal(Util.java:792)
> at org.apache.calcite.plan.volcano.VolcanoRuleCall.
> transformTo(VolcanoRuleCall.java:148)
> at org.apache.calcite.plan.RelOptRuleCall.transformTo(
> RelOptRuleCall.java:225)
> at org.apache.calcite.rel.convert.ConverterRule.onMatch(
> ConverterRule.java:117)
> at org.apache.calcite.plan.volcano.VolcanoRuleCall.
> onMatch(VolcanoRuleCall.java:213)
> ... 4 more
> Caused by: java.lang.NullPointerException
> at org.apache.calcite.schema.Statistics$2.isKey(Statistics.java:70)
> at org.apache.calcite.prepare.RelOptTableImpl.isKey(
> RelOptTableImpl.java:288)
> at org.apache.calcite.rel.metadata.RelMdColumnUniqueness.areColumnsUnique(
> RelMdColumnUniqueness.java:76)
> at GeneratedMetadataHandler_ColumnUniqueness.areColumnsUnique_$(Unknown
> Source)
> at GeneratedMetadataHandler_ColumnUniqueness.areColumnsUnique(Unknown
> Source)
> at org.apache.calcite.rel.metadata.RelMetadataQuery.areColumnsUnique(
> RelMetadataQuery.java:461)
> at org.apache.calcite.rel.metadata.RelMdUtil.areColumnsDefinitelyUnique(
> RelMdUtil.java:216)
> at org.apache.calcite.rel.metadata.RelMdDistinctRowCount.
> getDistinctRowCount(RelMdDistinctRowCount.java:75)
> at GeneratedMetadataHandler_DistinctRowCount.getDistinctRowCount_$(Unknown
> Source)
> at GeneratedMetadataHandler_DistinctRowCount.getDistinctRowCount(Unknown
> Source)
> at org.apache.calcite.rel.metadata.RelMetadataQuery.getDistinctRowCount(
> RelMetadataQuery.java:700)
> at org.apache.calcite.rel.metadata.RelMdDistinctRowCount.
> getDistinctRowCount(RelMdDistinctRowCount.java:292)
> at GeneratedMetadataHandler_DistinctRowCount.getDistinctRowCount_$(Unknown
> Source)
> at GeneratedMetadataHandler_DistinctRowCount.getDistinctRowCount(Unknown
> Source)
> at org.apache.calcite.rel.metadata.RelMetadataQuery.getDistinctRowCount(
> RelMetadataQuery.java:700)
> at org.apache.calcite.rel.metadata.RelMdDistinctRowCount.
> getDistinctRowCount(RelMdDistinctRowCount.java:138)
> at GeneratedMetadataHandler_DistinctRowCount.getDistinctRowCount_$(Unknown
> Source)
> at GeneratedMetadataHandler_DistinctRowCount.getDistinctRowCount(Unknown
> Source)
> at org.apache.calcite.rel.metadata.RelMetadataQuery.getDistinctRowCount(
> RelMetadataQuery.java:700)
> at org.apache.calcite.rel.metadata.RelMdDistinctRowCount.
> getDistinctRowCount(RelMdDistinctRowCount.java:292)
> at GeneratedMetadataHandler_DistinctRowCount.getDistinctRowCount_$(Unknown
> Source)
> at GeneratedMetadataHandler_DistinctRowCount.getDistinctRowCount(Unknown
> Source)
> at org.apache.calcite.rel.metadata.RelMetadataQuery.getDistinctRowCount(
> RelMetadataQuery.java:700)
> at org.apache.calcite.rel.metadata.RelMdRowCount.
> getRowCount(RelMdRowCount.java:194)
> at GeneratedMetadataHandler_RowCount.getRowCount_$(Unknown Source)
> at GeneratedMetadataHandler_RowCount.getRowCount(Unknown Source)
> at org.apache.calcite.rel.metadata.RelMetadataQuery.
> getRowCount(RelMetadataQuery.java:201)
> at org.apache.calcite.rel.core.Aggregate.computeSelfCost(
> Aggregate.java:304)
> at org.apache.calcite.rel.metadata.RelMdPercentageOriginalRows.
> getNonCumulativeCost(RelMdPercentageOriginalRows.java:162)
> at GeneratedMetadataHandler_NonCumulativeCost.getNonCumulativeCost_$(Unknown
> Source)
> at GeneratedMetadataHandler_NonCumulativeCost.getNonCumulativeCost(Unknown
> Source)
> at org.apache.calcite.rel.metadata.RelMetadataQuery.getNonCumulativeCost(
> RelMetadataQuery.java:258)
> at org.apache.calcite.plan.volcano.VolcanoPlanner.
> getCost(VolcanoPlanner.java:1128)
> at org.apache.calcite.plan.volcano.RelSubset.propagateCostImprovements0(
> RelSubset.java:336)
> at org.apache.calcite.plan.volcano.RelSubset.propagateCostImprovements(
> RelSubset.java:319)
> at org.apache.calcite.plan.volcano.RelSubset.propagateCostImprovements0(
> RelSubset.java:348)
> at org.apache.calcite.plan.volcano.RelSubset.propagateCostImprovements(
> RelSubset.java:319)
> at org.apache.calcite.plan.volcano.RelSubset.propagateCostImprovements0(
> RelSubset.java:348)
> at org.apache.calcite.plan.volcano.RelSubset.propagateCostImprovements(
> RelSubset.java:319)
> at org.apache.calcite.plan.volcano.VolcanoPlanner.
> addRelToSet(VolcanoPlanner.java:1830)
> at org.apache.calcite.plan.volcano.VolcanoPlanner.
> registerImpl(VolcanoPlanner.java:1766)
> at org.apache.calcite.plan.volcano.VolcanoPlanner.
> register(VolcanoPlanner.java:1032)
> at org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(
> VolcanoPlanner.java:1052)
> at org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(
> VolcanoPlanner.java:1942)
> at org.apache.calcite.plan.volcano.VolcanoRuleCall.
> transformTo(VolcanoRuleCall.java:136)
> ... 7 more
>
> I define the Statistics in the tables I use like this:
> public Statistic getStatistic() {
> int rowCount = rows.size();
> return Statistics.of(rowCount, null); //add List<ImmutableBitSet>
> }
>
>
> Thanks,
> George
>
> 2016-10-16 7:28 GMT+03:00 Jungtaek Lim <kabh...@gmail.com>:
>
>> Hi George,
>>
>> This patch is ported version (with small fixes) of Milinda's samza-sql
>> implementation for Storm SQL.
>> https://github.com/apache/storm/pull/1736
>>
>> In this patch I removed adding HepPlanner and just rely on Volcano Planner
>> (so the patch may be the closer thing what you want).
>> For now I also remove code regarding metadata since I'm not clear on how
>> it
>> works and what it helps, but I'll re-address once I can find its usage and
>> benefits.
>>
>> Hope this helps.
>>
>> Thanks,
>> Jungtaek Lim (HeartSaVioR)
>>
>> 2016년 10월 4일 (화) 오후 7:08, Γιώργος Θεοδωράκης <giwrgosrth...@gmail.com>님이
>> 작성:
>>
>> > I think I did as you said:
>> >
>> > https://github.com/giwrgostheod/Calcite-Saber/blob/master/
>> src/main/java/calcite/VolcanoTester.java
>> >
>> > and I get for every query I use:
>> > Exception in thread "main"
>> > org.apache.calcite.plan.RelOptPlanner$CannotPlanException: Node
>> > [rel#10:Subset#2.NONE.[]] could not be implemented; planner state:
>> > Root: rel#10:Subset#2.NONE.[]
>> > Original rel:
>> > ....
>> > at
>> >
>> > org.apache.calcite.plan.volcano.RelSubset$CheapestPlanReplac
>> er.visit(RelSubset.java:443)
>> > at
>> >
>> > org.apache.calcite.plan.volcano.RelSubset.buildCheapestPlan(
>> RelSubset.java:293)
>> > at
>> >
>> > org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(
>> VolcanoPlanner.java:835)
>> > at org.apache.calcite.tools.Programs$RuleSetProgram.run(Program
>> s.java:334)
>> > at org.apache.calcite.prepare.PlannerImpl.transform(PlannerImpl
>> .java:308)
>> > at calcite.VolcanoTester.main(VolcanoTester.java:77)
>> >
>> > My table's is defined here :
>> >
>> > https://github.com/giwrgostheod/Calcite-Saber/blob/master/
>> src/main/java/calcite/utils/OrdersTableFactory.java
>> >
>> >
>> > Thank you for your time,
>> > George
>> >
>> >
>> > 2016-10-04 2:38 GMT+03:00 Jordan Halterman <jordan.halter...@gmail.com
>> >:
>> >
>> > > The link you provided is a pretty good example. Build a
>> FrameworkConfig
>> > > with your schema, parser config, and other information, and use it to
>> > > create a Planner. That Planner uses a VolcanoPlanner internally.
>> What's
>> > > missing from that particular example is just the addition of programs.
>> > > Programs are effectively sets of rules you will use to optimize your
>> > query.
>> > > So, to add your FilterProjectTransposeRule to the planner, call this
>> when
>> > > building your FrameworkConfig:
>> > >
>> > > .programs(Programs.ofRules(FilterProjectTransposeRule.INSTANCE))
>> > >
>> > > That adds your program(s) to the set of programs in the planner, and
>> > those
>> > > programs can be accessed to optimize the query. Use the planner to
>> > parse()
>> > > your query, validate() your query, and then convert() your query into
>> a
>> > > logical plan. Then call...
>> > >
>> > > RelTraitSet traitSet = planner.emptyTraitSet().replac
>> e(Convention.NONE);
>> > > planner.transform(0. traitSet, logicalPlan);
>> > >
>> > > to apply the rules you added to the configuration. That should use the
>> > > VolcanoPlanner to apply the rules you added in your Program. The trait
>> > set
>> > > that's passed to that method is the required output trait set. So, if
>> you
>> > > wanted to convert the logical plan into some physical convention,
>> you'd
>> > > pass your physical convention instead of Convention.NONE. I can
>> respond
>> > > with a full example if you need it in a bit. I just don't have the
>> > capacity
>> > > to write it ATM.
>> > >
>> > > On Mon, Oct 3, 2016 at 8:51 AM, Γιώργος Θεοδωράκης <
>> > > giwrgosrth...@gmail.com>
>> > > wrote:
>> > >
>> > > > Hi,
>> > > >
>> > > > I want to parse an Sql query and transform it to an optimized
>> > relational
>> > > > plan (not convert it to physical !!) using calcite rules based on my
>> > > > database schema and metadata. Right now, the only helpful example I
>> > have
>> > > > found for my purpose is taken from
>> > > > https://github.com/milinda/samza-sql/blob/master/samza-
>> > > >
>> > sql-planner/src/main/java/org/apache/samza/sql/planner/Query
>> Planner.java
>> > > > ,
>> > > > in which a simple Planner is used for parsing and validating Sql
>> and a
>> > > > HepPlanner is used for searching for an optimized plan based on
>> > imported
>> > > > rules.
>> > > >
>> > > > Is there any way to use in my case the VolcanoPlanner? The only
>> > examples
>> > > I
>> > > > have seen so far from the test classes suggest that it should be
>> used
>> > for
>> > > > converting relational expressions to physical ones. How can I make
>> the
>> > > > Volcano Planner "see" my SchemaPlus schema ,when I can only define
>> > > > RelOptSchema? Can someone provide me with a complete example of
>> using
>> > > > Volcano Planner and adding rules, such
>> > > > as FilterProjectTransposeRule.INSTANCE?
>> > > >
>> > > > Thanks in advance,
>> > > > George
>> > > >
>> > >
>> >
>>
>
>
