Is there anymore information I can give to help understand the missing rule?
Thank you! Eric On Tue, Aug 20, 2024 at 18:59 Eric Berryman <eric.berry...@gmail.com> wrote: > Is there something I’m missing about > EnumerableLimit? It seems like I get an infinite cost anytime it’s in my > path? > > Thank you again, > Eric > > On Mon, Aug 19, 2024 at 18:00 Eric Berryman <eric.berry...@gmail.com> > wrote: > >> I see the red circles in the svg query plan I added to the gist. I’m just >> not sure what it’s supposed to be telling me. >> >> Thank you! >> Eric >> >> On Mon, Aug 19, 2024 at 11:35 Eric Berryman <eric.berry...@gmail.com> >> wrote: >> >>> 1.37.0 >>> >>> On Mon, Aug 19, 2024 at 11:28 Ruben Q L <rube...@gmail.com> wrote: >>> >>>> Which Calcite version are you using? >>>> >>>> >>>> On Mon, Aug 19, 2024 at 4:22 PM Eric Berryman <eric.berry...@gmail.com> >>>> wrote: >>>> >>>> > I’ve noticed if I remove the limit, the EnumerableUnion works fine. >>>> So it >>>> > seems the infinite cost is coming from the EnumerableLimit outside of >>>> the >>>> > union. I have an LDAPSortLimit which is pushed past the union, and an >>>> > EnumerableLimit >>>> > is left outside the union, as you can see in the gist. >>>> > >>>> > On Mon, Aug 19, 2024 at 11:10 Eric Berryman <eric.berry...@gmail.com> >>>> > wrote: >>>> > >>>> > > Is there anymore information that would be helpful for me to give >>>> to aide >>>> > > in troubleshooting this? >>>> > > >>>> > > Thank you! >>>> > > Eric >>>> > > >>>> > > On Thu, Aug 15, 2024 at 16:51 Eric Berryman < >>>> eric.berry...@gmail.com> >>>> > > wrote: >>>> > > >>>> > >> >>>> > >> I seem to be struggling with unions still. Here is a gist with the >>>> > output >>>> > >> and svg of the plan. >>>> > >> >>>> > >> https://gist.github.com/berryma4/c4870dd4e292e0509d1c85c308b52e67 >>>> > >> >>>> > >> I need a little help on direction of what the svg is telling me. >>>> I see >>>> > >> the cost is infinite, but I don’t see where I’m missing a rule. >>>> > >> >>>> > >> >>>> > >> Thank you for your help, >>>> > >> Eric >>>> > >> >>>> > >> >>>> > >> >>>> > >> On Mon, Jul 29, 2024 at 18:28 Eric Berryman < >>>> eric.berry...@gmail.com> >>>> > >> wrote: >>>> > >> >>>> > >>> That didn't paste well. Here is a gist: >>>> > >>> https://gist.github.com/berryma4/c6c09da050f273295edd23c045c63403 >>>> > >>> >>>> > >>> >>>> > >>> >>>> > >>> On Mon, Jul 29, 2024 at 4:58 PM Eric Berryman < >>>> eric.berry...@gmail.com >>>> > > >>>> > >>> wrote: >>>> > >>> >>>> > >>>> I’m back from a long holiday, and seem stuck still on this >>>> scenario. >>>> > >>>> The plan gets created without any issues, but I get this error >>>> while >>>> > >>>> executing queries with unions. >>>> > >>>> >>>> > >>>> Thank you again! >>>> > >>>> Eric >>>> > >>>> >>>> > >>>> Here is the plan dump and error after running >>>> > >>>> relRunner.prepareStatement. >>>> > >>>> >>>> > >>>> EnumerableProject(UserObject=[$0], id=[$3]): rowcount = 30.0, >>>> > >>>> cumulative cost = {146.0 rows, 1393.2 cpu, 0.0 io}, id = 213 >>>> > >>>> EnumerableLimit(fetch=[100]): rowcount = 30.0, cumulative cost = >>>> > {116.0 >>>> > >>>> rows, 1333.2 cpu, 0.0 io}, id = 212 EnumerableUnion(all=[true]): >>>> > rowcount = >>>> > >>>> 30.0, cumulative cost = {86.0 rows, 1303.2 cpu, 0.0 io}, id = 211 >>>> > >>>> EnumerableProject(UserObject=[$0], id=[$3]): rowcount = 15.0, >>>> > cumulative >>>> > >>>> cost = {28.0 rows, 636.6 cpu, 0.0 io}, id = 207 >>>> > LDAPToEnumerableConverter: >>>> > >>>> rowcount = 15.0, cumulative cost = {13.0 rows, 21.6 cpu, 0.0 >>>> io}, id >>>> > = 206 >>>> > >>>> LDAPFilter(condition=[=(UPPER($5), 'RUDD')]): rowcount = 15.0, >>>> > cumulative >>>> > >>>> cost = {11.5 rows, 20.1 cpu, 0.0 io}, id = 205 >>>> > >>>> LDAPTableScan(table=[[SCIMUserSchema, cap_internet_ddpmildap]]): >>>> > rowcount = >>>> > >>>> 100.0, cumulative cost = {10.0 rows, 10.100000000000001 cpu, 0.0 >>>> io}, >>>> > id = >>>> > >>>> 0 EnumerableProject(UserObject=[$0], id=[$3]): rowcount = 15.0, >>>> > cumulative >>>> > >>>> cost = {28.0 rows, 636.6 cpu, 0.0 io}, id = 210 >>>> > LDAPToEnumerableConverter: >>>> > >>>> rowcount = 15.0, cumulative cost = {13.0 rows, 21.6 cpu, 0.0 >>>> io}, id >>>> > = 209 >>>> > >>>> LDAPFilter(condition=[=(UPPER($5), 'RUDD')]): rowcount = 15.0, >>>> > cumulative >>>> > >>>> cost = {11.5 rows, 20.1 cpu, 0.0 io}, id = 208 >>>> > >>>> LDAPTableScan(table=[[SCIMUserSchema, prov_internet_ddpmildap]]): >>>> > rowcount >>>> > >>>> = 100.0, cumulative cost = {10.0 rows, 10.100000000000001 cpu, >>>> 0.0 >>>> > io}, id >>>> > >>>> = 2 >>>> > >>>> >>>> > >>>> java.lang.AssertionError: null at >>>> > >>>> >>>> > >>>> org.apache.calcite.adapter.enumerable.EnumerableLimit.<init>(EnumerableLimit.java:60) >>>> > >>>> at >>>> > >>>> >>>> > >>>> org.apache.calcite.adapter.enumerable.EnumerableLimit.copy(EnumerableLimit.java:84) >>>> > >>>> at >>>> > >>>> >>>> > >>>> org.apache.calcite.adapter.enumerable.EnumerableLimit.copy(EnumerableLimit.java:43) >>>> > >>>> at >>>> > >>>> >>>> > >>>> org.apache.calcite.sql2rel.RelFieldTrimmer.trimFields(RelFieldTrimmer.java:378) >>>> > >>>> at >>>> > >>>> >>>> > >>>> java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) >>>> > >>>> at java.base/java.lang.reflect.Method.invoke(Method.java:580) at >>>> > >>>> >>>> org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:533) at >>>> > >>>> >>>> > >>>> org.apache.calcite.sql2rel.RelFieldTrimmer.dispatchTrimFields(RelFieldTrimmer.java:286) >>>> > >>>> at >>>> > >>>> >>>> > >>>> org.apache.calcite.sql2rel.RelFieldTrimmer.trimChild(RelFieldTrimmer.java:228) >>>> > >>>> at >>>> > >>>> >>>> > >>>> org.apache.calcite.sql2rel.RelFieldTrimmer.trimFields(RelFieldTrimmer.java:514) >>>> > >>>> at >>>> > >>>> >>>> > >>>> java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) >>>> > >>>> at java.base/java.lang.reflect.Method.invoke(Method.java:580) at >>>> > >>>> >>>> org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:533) at >>>> > >>>> >>>> > >>>> org.apache.calcite.sql2rel.RelFieldTrimmer.dispatchTrimFields(RelFieldTrimmer.java:286) >>>> > >>>> at >>>> > >>>> >>>> > >>>> org.apache.calcite.sql2rel.RelFieldTrimmer.trim(RelFieldTrimmer.java:173) >>>> > >>>> at >>>> > >>>> >>>> > >>>> org.apache.calcite.sql2rel.SqlToRelConverter.trimUnusedFields(SqlToRelConverter.java:567) >>>> > >>>> at >>>> > org.apache.calcite.prepare.Prepare.trimUnusedFields(Prepare.java:386) >>>> at >>>> > >>>> >>>> > >>>> org.apache.calcite.prepare.CalcitePrepareImpl$CalcitePreparingStmt.prepare_(CalcitePrepareImpl.java:1061) >>>> > >>>> at >>>> > >>>> >>>> > >>>> org.apache.calcite.prepare.CalcitePrepareImpl$CalcitePreparingStmt.prepareRel(CalcitePrepareImpl.java:1026) >>>> > >>>> at >>>> > >>>> >>>> > >>>> org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:687) >>>> > >>>> at >>>> > >>>> >>>> > >>>> org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:519) >>>> > >>>> at >>>> > >>>> >>>> > >>>> org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:487) >>>> > >>>> at >>>> > >>>> >>>> > >>>> org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:237) >>>> > >>>> at >>>> > >>>> >>>> > >>>> org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement_(CalciteConnectionImpl.java:217) >>>> > >>>> at >>>> > >>>> >>>> > >>>> org.apache.calcite.jdbc.CalciteConnectionImpl.lambda$unwrap$0(CalciteConnectionImpl.java:187) >>>> > >>>> >>>> > >>>> >>>> > >>>> On Fri, Jul 19, 2024 at 11:07 Ruben Q L <rube...@gmail.com> >>>> wrote: >>>> > >>>> >>>> > >>>>> Eric, if you want to use EnumerableUnion instead of >>>> implementing your >>>> > >>>>> own >>>> > >>>>> Union, then I think you'll need to implement the appropriate >>>> > Converter >>>> > >>>>> [1] >>>> > >>>>> to transform between conventions LDAP <=> ENUMERABLE >>>> > >>>>> >>>> > >>>>> Best, >>>> > >>>>> Ruben >>>> > >>>>> >>>> > >>>>> [1] >>>> > >>>>> >>>> > >>>>> >>>> > >>>> https://github.com/apache/calcite/blob/main/core/src/main/java/org/apache/calcite/rel/convert/Converter.java >>>> > >>>>> >>>> > >>>>> >>>> > >>>>> On Fri, Jul 19, 2024 at 3:56 PM Eric Berryman < >>>> > eric.berry...@gmail.com >>>> > >>>>> > >>>> > >>>>> wrote: >>>> > >>>>> >>>> > >>>>> > I implement an LDAPUnion and rule to convert from >>>> logicalunion, and >>>> > >>>>> of >>>> > >>>>> > course, the error goes away. But I didn’t implement the union >>>> > >>>>> correctly, so >>>> > >>>>> > nothing is returned in my query. Is there a way to have my >>>> > LDAPUnion >>>> > >>>>> > use EnumerableUnion, >>>> > >>>>> > or a way to not have LDAPUnion and take care of this with a >>>> rule? >>>> > >>>>> > >>>> > >>>>> > Thank you again! >>>> > >>>>> > Eric >>>> > >>>>> > >>>> > >>>>> > On Tue, Jul 16, 2024 at 14:24 Eric Berryman < >>>> > eric.berry...@gmail.com >>>> > >>>>> > >>>> > >>>>> > wrote: >>>> > >>>>> > >>>> > >>>>> > > Well, I was wrong. That didn’t solve my problem with : >>>> > >>>>> > > >>>> > >>>>> > > Missing conversion is LogicalUnion[convention: NONE -> LDAP] >>>> > >>>>> > > >>>> > >>>>> > > Do I have to implement my own union? Is there a way I could >>>> just >>>> > >>>>> use the >>>> > >>>>> > > EnumerableUnion? >>>> > >>>>> > > ie. NONE -> Enumerable and skip implementing one for my ldap >>>> > >>>>> datastore? >>>> > >>>>> > > >>>> > >>>>> > > Thank you! >>>> > >>>>> > > Eric >>>> > >>>>> > > >>>> > >>>>> > > On Tue, Jul 16, 2024 at 09:00 Eric Berryman < >>>> > >>>>> eric.berry...@gmail.com> >>>> > >>>>> > > wrote: >>>> > >>>>> > > >>>> > >>>>> > >> When I register my tablescan object, and add my rules, I >>>> also >>>> > >>>>> added a >>>> > >>>>> > >> removeRule for >>>> EnumerableRules.ENUMERABLE_MERGE_UNION_RULE, and >>>> > >>>>> now >>>> > >>>>> > >> everything works with an offset also. >>>> > >>>>> > >> >>>> > >>>>> > >> Although, I don’t know why this is. I noticed the >>>> > >>>>> EnumerableMergeUnion >>>> > >>>>> > >> object in the plan, and thought I would try to remove it, >>>> > because >>>> > >>>>> it was >>>> > >>>>> > >> different. >>>> > >>>>> > >> >>>> > >>>>> > >> Any explanation is appreciated, thank you!!! >>>> > >>>>> > >> Eric >>>> > >>>>> > >> >>>> > >>>>> > >> >>>> > >>>>> > >> On Mon, Jul 15, 2024 at 18:02 Eric Berryman < >>>> > >>>>> eric.berry...@gmail.com> >>>> > >>>>> > >> wrote: >>>> > >>>>> > >> >>>> > >>>>> > >>> Hello! >>>> > >>>>> > >>> >>>> > >>>>> > >>> I seem to have an issue with my new limit rule which >>>> pushes >>>> > down >>>> > >>>>> to the >>>> > >>>>> > >>> datastore. It works fine, unless I add an offset to the >>>> fetch. >>>> > >>>>> Where I >>>> > >>>>> > end >>>> > >>>>> > >>> up with the following error: >>>> > >>>>> > >>> There are not enough rules… Missing conversion is >>>> > >>>>> > >>> LogicalUnion[convention: NONE -> LDAP] >>>> > >>>>> > >>> >>>> > >>>>> > >>> Why would this only come up when an offset value is added? >>>> > >>>>> > >>> >>>> > >>>>> > >>> Thank you! >>>> > >>>>> > >>> Eric >>>> > >>>>> > >>> >>>> > >>>>> > >>> >>>> > >>>>> > >>> On Fri, Jun 21, 2024 at 09:25 Eric Berryman < >>>> > >>>>> eric.berry...@gmail.com> >>>> > >>>>> > >>> wrote: >>>> > >>>>> > >>> >>>> > >>>>> > >>>> That’s perfect >>>> > >>>>> > >>>> >>>> > >>>>> > >>>> >>>> > >>>>> > >>>> > >>>>> >>>> > >>>> https://github.com/apache/calcite/blob/main/cassandra/src/main/java/org/apache/calcite/adapter/cassandra/CassandraRules.java#L401 >>>> > >>>>> > >>>> >>>> > >>>>> > >>>> Thank you for such a quick response! >>>> > >>>>> > >>>> Eric >>>> > >>>>> > >>>> >>>> > >>>>> > >>>> >>>> > >>>>> > >>>> On Fri, Jun 21, 2024 at 09:14 Michael Mior < >>>> mm...@apache.org> >>>> > >>>>> wrote: >>>> > >>>>> > >>>> >>>> > >>>>> > >>>>> Eric, >>>> > >>>>> > >>>>> >>>> > >>>>> > >>>>> Could you give a more specific example of the failure >>>> > scenario >>>> > >>>>> you're >>>> > >>>>> > >>>>> experiencing? >>>> > >>>>> > >>>>> >>>> > >>>>> > >>>>> For a simple example of how limits can be pushed down, >>>> this >>>> > is >>>> > >>>>> done >>>> > >>>>> > in >>>> > >>>>> > >>>>> the >>>> > >>>>> > >>>>> Cassandra adapter with CassandraLimitRule. It matches an >>>> > >>>>> > >>>>> EnumerableLimit on >>>> > >>>>> > >>>>> top of a CassandraToEnumerableConverter and then >>>> converts >>>> > that >>>> > >>>>> limit >>>> > >>>>> > >>>>> to a >>>> > >>>>> > >>>>> CassandraLimit which passes along the limit and offset >>>> > >>>>> information to >>>> > >>>>> > >>>>> CassandraToEnumerableConverter for when the query is >>>> > executed. >>>> > >>>>> The >>>> > >>>>> > >>>>> EnumerableLimit is then replaced with a CassandraLimit >>>> > >>>>> effectively >>>> > >>>>> > as a >>>> > >>>>> > >>>>> placeholder to signal that the limit has been handled. >>>> > >>>>> > >>>>> >>>> > >>>>> > >>>>> -- >>>> > >>>>> > >>>>> Michael Mior >>>> > >>>>> > >>>>> mm...@apache.org >>>> > >>>>> > >>>>> >>>> > >>>>> > >>>>> >>>> > >>>>> > >>>>> On Fri, Jun 21, 2024 at 8:44 AM Eric Berryman < >>>> > >>>>> > eric.berry...@gmail.com >>>> > >>>>> > >>>>> > >>>> > >>>>> > >>>>> wrote: >>>> > >>>>> > >>>>> >>>> > >>>>> > >>>>> > Hello! >>>> > >>>>> > >>>>> > >>>> > >>>>> > >>>>> > When I add limit to my relbuilder object, the planner >>>> gives >>>> > >>>>> up. But >>>> > >>>>> > >>>>> works >>>> > >>>>> > >>>>> > fine without it. >>>> > >>>>> > >>>>> > >>>> > >>>>> > >>>>> > I wasn’t able to find any examples of using limit and >>>> > >>>>> pushing the >>>> > >>>>> > >>>>> limit >>>> > >>>>> > >>>>> > values down to a data source. >>>> > >>>>> > >>>>> > >>>> > >>>>> > >>>>> > Could someone help with some links on this subject? >>>> > >>>>> > >>>>> > >>>> > >>>>> > >>>>> > Thank you! >>>> > >>>>> > >>>>> > Eric >>>> > >>>>> > >>>>> > >>>> > >>>>> > >>>>> >>>> > >>>>> > >>>> >>>> > >>>>> > >>>> > >>>>> >>>> > >>>> >>>> > >>>> >>>
