I finally got this compiling and working, I think, but since (as Reynold points out) it involves a little API refactoring, I was hoping to get some discussion about it going as soon as possible.
I have the changes necessary to give RDD, DStream, and DataFrame some level of common interface, in https://github.com/apache/spark/pull/5565, and would very much appreciate comments. Thanks, Nathan On Thu, Dec 19, 2013 at 12:42 AM, Reynold Xin <r...@apache.org> wrote: > > On Wed, Dec 18, 2013 at 12:17 PM, Nathan Kronenfeld < > nkronenf...@oculusinfo.com> wrote: > >> >> >> Since many of the functions exist in parallel between the two, I guess I >> would expect something like: >> >> trait BasicRDDFunctions { >> def map... >> def reduce... >> def filter... >> def foreach... >> } >> >> class RDD extends BasicRDDFunctions... >> class DStream extends BasicRDDFunctions... >> > > I like this idea. We should discuss more about it on the dev list. It > would require refactoring some APIs, but does lead to better unification. >
