I seem to be struggling with unions still. Here is a gist with the output and svg of the plan.
https://gist.github.com/berryma4/c4870dd4e292e0509d1c85c308b52e67 I need a little help on direction of what the svg is telling me. I see the cost is infinite, but I don’t see where I’m missing a rule. Thank you for your help, Eric On Mon, Jul 29, 2024 at 18:28 Eric Berryman <eric.berry...@gmail.com> wrote: > That didn't paste well. Here is a gist: > https://gist.github.com/berryma4/c6c09da050f273295edd23c045c63403 > > > > On Mon, Jul 29, 2024 at 4:58 PM Eric Berryman <eric.berry...@gmail.com> > wrote: > >> I’m back from a long holiday, and seem stuck still on this scenario. The >> plan gets created without any issues, but I get this error while executing >> queries with unions. >> >> Thank you again! >> Eric >> >> Here is the plan dump and error after running relRunner.prepareStatement. >> >> EnumerableProject(UserObject=[$0], id=[$3]): rowcount = 30.0, cumulative >> cost = {146.0 rows, 1393.2 cpu, 0.0 io}, id = 213 >> EnumerableLimit(fetch=[100]): rowcount = 30.0, cumulative cost = {116.0 >> rows, 1333.2 cpu, 0.0 io}, id = 212 EnumerableUnion(all=[true]): rowcount = >> 30.0, cumulative cost = {86.0 rows, 1303.2 cpu, 0.0 io}, id = 211 >> EnumerableProject(UserObject=[$0], id=[$3]): rowcount = 15.0, cumulative >> cost = {28.0 rows, 636.6 cpu, 0.0 io}, id = 207 LDAPToEnumerableConverter: >> rowcount = 15.0, cumulative cost = {13.0 rows, 21.6 cpu, 0.0 io}, id = 206 >> LDAPFilter(condition=[=(UPPER($5), 'RUDD')]): rowcount = 15.0, cumulative >> cost = {11.5 rows, 20.1 cpu, 0.0 io}, id = 205 >> LDAPTableScan(table=[[SCIMUserSchema, cap_internet_ddpmildap]]): rowcount = >> 100.0, cumulative cost = {10.0 rows, 10.100000000000001 cpu, 0.0 io}, id = >> 0 EnumerableProject(UserObject=[$0], id=[$3]): rowcount = 15.0, cumulative >> cost = {28.0 rows, 636.6 cpu, 0.0 io}, id = 210 LDAPToEnumerableConverter: >> rowcount = 15.0, cumulative cost = {13.0 rows, 21.6 cpu, 0.0 io}, id = 209 >> LDAPFilter(condition=[=(UPPER($5), 'RUDD')]): rowcount = 15.0, cumulative >> cost = {11.5 rows, 20.1 cpu, 0.0 io}, id = 208 >> LDAPTableScan(table=[[SCIMUserSchema, prov_internet_ddpmildap]]): rowcount >> = 100.0, cumulative cost = {10.0 rows, 10.100000000000001 cpu, 0.0 io}, id >> = 2 >> >> java.lang.AssertionError: null at >> org.apache.calcite.adapter.enumerable.EnumerableLimit.<init>(EnumerableLimit.java:60) >> at >> org.apache.calcite.adapter.enumerable.EnumerableLimit.copy(EnumerableLimit.java:84) >> at >> org.apache.calcite.adapter.enumerable.EnumerableLimit.copy(EnumerableLimit.java:43) >> at >> org.apache.calcite.sql2rel.RelFieldTrimmer.trimFields(RelFieldTrimmer.java:378) >> at >> java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) >> at java.base/java.lang.reflect.Method.invoke(Method.java:580) at >> org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:533) at >> org.apache.calcite.sql2rel.RelFieldTrimmer.dispatchTrimFields(RelFieldTrimmer.java:286) >> at >> org.apache.calcite.sql2rel.RelFieldTrimmer.trimChild(RelFieldTrimmer.java:228) >> at >> org.apache.calcite.sql2rel.RelFieldTrimmer.trimFields(RelFieldTrimmer.java:514) >> at >> java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) >> at java.base/java.lang.reflect.Method.invoke(Method.java:580) at >> org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:533) at >> org.apache.calcite.sql2rel.RelFieldTrimmer.dispatchTrimFields(RelFieldTrimmer.java:286) >> at >> org.apache.calcite.sql2rel.RelFieldTrimmer.trim(RelFieldTrimmer.java:173) >> at >> org.apache.calcite.sql2rel.SqlToRelConverter.trimUnusedFields(SqlToRelConverter.java:567) >> at org.apache.calcite.prepare.Prepare.trimUnusedFields(Prepare.java:386) at >> org.apache.calcite.prepare.CalcitePrepareImpl$CalcitePreparingStmt.prepare_(CalcitePrepareImpl.java:1061) >> at >> org.apache.calcite.prepare.CalcitePrepareImpl$CalcitePreparingStmt.prepareRel(CalcitePrepareImpl.java:1026) >> at >> org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:687) >> at >> org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:519) >> at >> org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:487) >> at >> org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:237) >> at >> org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement_(CalciteConnectionImpl.java:217) >> at >> org.apache.calcite.jdbc.CalciteConnectionImpl.lambda$unwrap$0(CalciteConnectionImpl.java:187) >> >> >> On Fri, Jul 19, 2024 at 11:07 Ruben Q L <rube...@gmail.com> wrote: >> >>> Eric, if you want to use EnumerableUnion instead of implementing your own >>> Union, then I think you'll need to implement the appropriate Converter >>> [1] >>> to transform between conventions LDAP <=> ENUMERABLE >>> >>> Best, >>> Ruben >>> >>> [1] >>> >>> https://github.com/apache/calcite/blob/main/core/src/main/java/org/apache/calcite/rel/convert/Converter.java >>> >>> >>> On Fri, Jul 19, 2024 at 3:56 PM Eric Berryman <eric.berry...@gmail.com> >>> wrote: >>> >>> > I implement an LDAPUnion and rule to convert from logicalunion, and of >>> > course, the error goes away. But I didn’t implement the union >>> correctly, so >>> > nothing is returned in my query. Is there a way to have my LDAPUnion >>> > use EnumerableUnion, >>> > or a way to not have LDAPUnion and take care of this with a rule? >>> > >>> > Thank you again! >>> > Eric >>> > >>> > On Tue, Jul 16, 2024 at 14:24 Eric Berryman <eric.berry...@gmail.com> >>> > wrote: >>> > >>> > > Well, I was wrong. That didn’t solve my problem with : >>> > > >>> > > Missing conversion is LogicalUnion[convention: NONE -> LDAP] >>> > > >>> > > Do I have to implement my own union? Is there a way I could just use >>> the >>> > > EnumerableUnion? >>> > > ie. NONE -> Enumerable and skip implementing one for my ldap >>> datastore? >>> > > >>> > > Thank you! >>> > > Eric >>> > > >>> > > On Tue, Jul 16, 2024 at 09:00 Eric Berryman <eric.berry...@gmail.com >>> > >>> > > wrote: >>> > > >>> > >> When I register my tablescan object, and add my rules, I also added >>> a >>> > >> removeRule for EnumerableRules.ENUMERABLE_MERGE_UNION_RULE, and now >>> > >> everything works with an offset also. >>> > >> >>> > >> Although, I don’t know why this is. I noticed the >>> EnumerableMergeUnion >>> > >> object in the plan, and thought I would try to remove it, because >>> it was >>> > >> different. >>> > >> >>> > >> Any explanation is appreciated, thank you!!! >>> > >> Eric >>> > >> >>> > >> >>> > >> On Mon, Jul 15, 2024 at 18:02 Eric Berryman < >>> eric.berry...@gmail.com> >>> > >> wrote: >>> > >> >>> > >>> Hello! >>> > >>> >>> > >>> I seem to have an issue with my new limit rule which pushes down >>> to the >>> > >>> datastore. It works fine, unless I add an offset to the fetch. >>> Where I >>> > end >>> > >>> up with the following error: >>> > >>> There are not enough rules… Missing conversion is >>> > >>> LogicalUnion[convention: NONE -> LDAP] >>> > >>> >>> > >>> Why would this only come up when an offset value is added? >>> > >>> >>> > >>> Thank you! >>> > >>> Eric >>> > >>> >>> > >>> >>> > >>> On Fri, Jun 21, 2024 at 09:25 Eric Berryman < >>> eric.berry...@gmail.com> >>> > >>> wrote: >>> > >>> >>> > >>>> That’s perfect >>> > >>>> >>> > >>>> >>> > >>> https://github.com/apache/calcite/blob/main/cassandra/src/main/java/org/apache/calcite/adapter/cassandra/CassandraRules.java#L401 >>> > >>>> >>> > >>>> Thank you for such a quick response! >>> > >>>> Eric >>> > >>>> >>> > >>>> >>> > >>>> On Fri, Jun 21, 2024 at 09:14 Michael Mior <mm...@apache.org> >>> wrote: >>> > >>>> >>> > >>>>> Eric, >>> > >>>>> >>> > >>>>> Could you give a more specific example of the failure scenario >>> you're >>> > >>>>> experiencing? >>> > >>>>> >>> > >>>>> For a simple example of how limits can be pushed down, this is >>> done >>> > in >>> > >>>>> the >>> > >>>>> Cassandra adapter with CassandraLimitRule. It matches an >>> > >>>>> EnumerableLimit on >>> > >>>>> top of a CassandraToEnumerableConverter and then converts that >>> limit >>> > >>>>> to a >>> > >>>>> CassandraLimit which passes along the limit and offset >>> information to >>> > >>>>> CassandraToEnumerableConverter for when the query is executed. >>> The >>> > >>>>> EnumerableLimit is then replaced with a CassandraLimit >>> effectively >>> > as a >>> > >>>>> placeholder to signal that the limit has been handled. >>> > >>>>> >>> > >>>>> -- >>> > >>>>> Michael Mior >>> > >>>>> mm...@apache.org >>> > >>>>> >>> > >>>>> >>> > >>>>> On Fri, Jun 21, 2024 at 8:44 AM Eric Berryman < >>> > eric.berry...@gmail.com >>> > >>>>> > >>> > >>>>> wrote: >>> > >>>>> >>> > >>>>> > Hello! >>> > >>>>> > >>> > >>>>> > When I add limit to my relbuilder object, the planner gives >>> up. But >>> > >>>>> works >>> > >>>>> > fine without it. >>> > >>>>> > >>> > >>>>> > I wasn’t able to find any examples of using limit and pushing >>> the >>> > >>>>> limit >>> > >>>>> > values down to a data source. >>> > >>>>> > >>> > >>>>> > Could someone help with some links on this subject? >>> > >>>>> > >>> > >>>>> > Thank you! >>> > >>>>> > Eric >>> > >>>>> > >>> > >>>>> >>> > >>>> >>> > >>> >>
