I searched for mapPartitions and flatMapGroupsWithState, and it looks as if you are talking about Apache Spark operations. Can you give some examples of typical queries that would use these operations?
It’s possible that these operations accomplish things that are not possible in the relational model; but I think it’s more likely that these are algorithms that can implement relational operations such as windowed aggregate functions. If you give some examples, we can see whether they can be expressed in SQL or relational algebra. Julian > On Mar 3, 2021, at 10:54 PM, Rui Wang <amaliu...@apache.org> wrote: > > Well I think the expected approach is to translate other operations to > relational operators by yourself ;-) > > I think Calcite won't be the place to have extensions for such translation. > The main concern is that those non relational operations are "non-standard". > > -Rui > > On Wed, Mar 3, 2021 at 10:12 PM Debajyoti Roy <newroy...@gmail.com> wrote: > >> Hi All, >> >> For operators like filter, join, union, aggregate, window the >> logical RelNode choices are obvious within org.apache.calcite.rel.logical >> package. >> >> However, for more complex applications that require operations like >> mapPartitions, flatMapGroupsWithState, etc. what would be some choices for >> logical rel node and relational expression classes in Apache Calcite >> (independent of engine)? >> >> What is a good way to model operators that are not traditionally relational >> and hence do not exist in Calcite (looking for hooks / extension points / >> roadmaps)? >> >> Thanks in advance for any pointers, (disclaimer: I am new to Calcite) >> Debajyoti Roy >>
