Yes there is definitely some similarity to groupby What is this used for: https://calcite.apache.org/javadocAggregate/org/apache/calcite/rel/logical/LogicalTableFunctionScan.html Can i model a mapPartitions T -> U as ^ ?
On Thu, Mar 4, 2021 at 12:06 PM Rui Wang <amaliu...@apache.org> wrote: > I feel like the mapPartitions can be implemented as a SELECT + GROUP BY, > where GROUP BY is to partition the data, then per partition computation is > handled by the SELECT. > > -Rui > > On Thu, Mar 4, 2021 at 11:56 AM Debajyoti Roy <newroy...@gmail.com> wrote: > > > Thanks again Julian. > > > > Since, mapPartitions is really a specialized map would it be best to > model > > it as a SELECT (similar to functions inside an expression) ? > > Barring cases where h > h' and mapPartitions acts like a filter. > > > > On Thu, Mar 4, 2021 at 11:41 AM Julian Hyde <jhyde.apa...@gmail.com> > > wrote: > > > > > SQL has equivalents of many functional programming idioms: > > > > > > * map is SELECT > > > * filter is WHERE > > > * flatMap is similar to CROSS APPLY > > > > > > That said, SQL’s strength is that the operations are not optimized for > > any > > > particular physical organization of data (e.g. working on sorted or > > > partitioned data). mapPartitions is in this category. Of course a > > physical > > > implementation of one of SQL’s logical operators might use > mapPartitions. > > > > > > Julian > > > > > > > > > > > > > > > > > > > On Mar 4, 2021, at 10:44 AM, Debajyoti Roy <newroy...@gmail.com> > > wrote: > > > > > > > > Thanks for the responses, adding some more color below. > > > > > > > > Spark's API adopted concepts from the functional programming paradigm > > > (map, > > > > filter, flatmap,...) into data processing. Spark did add several > > > relational > > > > operators like join, union, select, etc. However, there are certain > > APIs > > > > that are really hard to model in terms of standard relational > > operators. > > > > Let me take one example of mapPartitions. > > > > > > > > mapPartitions( T -> U ): > > > > w columns and h rows can turn into totally different w' != w columns > > and > > > h' > > > > != h rows. Since processing happens per partition, this API is a > great > > > > choice for vectorized heavyweight initialization cost operations e.g. > > > batch > > > > inferencing. > > > > > > > > In terms of relational models, mapPartitions can be modeled just > like a > > > > function inside an expression operator. However, there can be > > interesting > > > > cases e.g. h > h' and mapMartitions starts to feel like a filter. Can > > > there > > > > be other challenges and opportunities in terms of planner and > optimizer > > > > because mapPartitions is definitely NOT like any other function > inside > > an > > > > expression as shown below: > > > > > > > > SELECT name, address, mapPartitions(id, tweet, '{threshold: 0.5}', > > > > 'sentiment_analysis_4', 10000) FROM my_twitter_data... > > > > > > > > So what is a better usage example for mapPartitions expressed as SQL > ? > > I > > > am > > > > really struggling with that part and I agree with Julian that is the > > key. > > > > > > > > Regards, > > > > Debajyoti Roy > > > > > > > > On Thu, Mar 4, 2021 at 12:01 AM Julian Hyde <jhyde.apa...@gmail.com> > > > wrote: > > > > > > > >> I searched for mapPartitions and flatMapGroupsWithState, and it > looks > > as > > > >> if you are talking about Apache Spark operations. Can you give some > > > >> examples of typical queries that would use these operations? > > > >> > > > >> It’s possible that these operations accomplish things that are not > > > >> possible in the relational model; but I think it’s more likely that > > > these > > > >> are algorithms that can implement relational operations such as > > windowed > > > >> aggregate functions. If you give some examples, we can see whether > > they > > > can > > > >> be expressed in SQL or relational algebra. > > > >> > > > >> Julian > > > >> > > > >> > > > >>> On Mar 3, 2021, at 10:54 PM, Rui Wang <amaliu...@apache.org> > wrote: > > > >>> > > > >>> Well I think the expected approach is to translate other operations > > to > > > >>> relational operators by yourself ;-) > > > >>> > > > >>> I think Calcite won't be the place to have extensions for such > > > >> translation. > > > >>> The main concern is that those non relational operations are > > > >> "non-standard". > > > >>> > > > >>> -Rui > > > >>> > > > >>> On Wed, Mar 3, 2021 at 10:12 PM Debajyoti Roy <newroy...@gmail.com > > > > > >> wrote: > > > >>> > > > >>>> Hi All, > > > >>>> > > > >>>> For operators like filter, join, union, aggregate, window the > > > >>>> logical RelNode choices are obvious within > > > >> org.apache.calcite.rel.logical > > > >>>> package. > > > >>>> > > > >>>> However, for more complex applications that require operations > like > > > >>>> mapPartitions, flatMapGroupsWithState, etc. what would be some > > choices > > > >> for > > > >>>> logical rel node and relational expression classes in Apache > Calcite > > > >>>> (independent of engine)? > > > >>>> > > > >>>> What is a good way to model operators that are not traditionally > > > >> relational > > > >>>> and hence do not exist in Calcite (looking for hooks / extension > > > points > > > >> / > > > >>>> roadmaps)? > > > >>>> > > > >>>> Thanks in advance for any pointers, (disclaimer: I am new to > > Calcite) > > > >>>> Debajyoti Roy > > > >>>> > > > >> > > > >> > > > > > > > > >