Github user koertkuipers commented on the issue:
https://github.com/apache/spark/pull/13526
see this conversation:
https://mail-archives.apache.org/mod_mbox/spark-user/201602.mbox/%3ccaaswr-7kqfmxd_cpr-_wdygafh+rarecm9olm5jkxfk14fc...@mail.gmail.com%3E
mapGroups is not a very interesting API, since without support for
secondary sort and hence no need for fold operations pushing all the value into
the reducer never really makes sense. so the interesting APIs are reduce (when
its fixed to be efficient and not use mapGroups) and agg.
how do you transform the values before they go into reduce? you can not do
this currently, which is why we need something like mapValues. with Aggregators
you can indeed do something similar inside the Aggregator (since the input type
is not equal to the buffer type), but this leads to all Aggregators currently
taking in some kind of input transform function, which hints at a suboptimal
API and a pattern that should be generalized and extracted.
i am curious to know why appending a column is inefficient? i am open to
different designs
about this being a rare case: i would argue the opposite. i expect to see a
lot of key-value datasets (```Dataset[(K, V)]```) in our codebase, and on those
a lot of operations like ```ds.groupByKey(_._1).mapValues.(_._2).reduce(...)```.
since this is the most natural translation of many RDD algos.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
