Yes - that sounds good Anton, I can work on documenting the window functions.
From: Anton Okolnychyi <[email protected]> Date: Thursday, December 15, 2016 at 4:34 PM To: Conversant <[email protected]> Cc: Michael Armbrust <[email protected]>, Jim Hughes <[email protected]>, "[email protected]" <[email protected]> Subject: Re: Expand the Spark SQL programming guide? I think it will make sense to show a sample implementation of UserDefinedAggregateFunction for DataFrames, and an example of the Aggregator API for typed Datasets. Jim, what if I submit a PR and you join the review process? I also do not mind to split this if you want, but it seems to be an overkill for this part. Jayesh, shall I skip the window functions part since you are going to work on that? 2016-12-15 22:48 GMT+01:00 Thakrar, Jayesh <[email protected]<mailto:[email protected]>>: I too am interested in expanding the documentation for Spark SQL. For my work I needed to get some info/examples/guidance on window functions and have been using https://databricks.com/blog/2015/07/15/introducing-window-functions-in-spark-sql.html . How about divide and conquer? From: Michael Armbrust <[email protected]<mailto:[email protected]>> Date: Thursday, December 15, 2016 at 3:21 PM To: Jim Hughes <[email protected]<mailto:[email protected]>> Cc: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: Expand the Spark SQL programming guide? Pull requests would be welcome for any major missing features in the guide: https://github.com/apache/spark/blob/master/docs/sql-programming-guide.md On Thu, Dec 15, 2016 at 11:48 AM, Jim Hughes <[email protected]<mailto:[email protected]>> wrote: Hi Anton, I'd like to see this as well. I've been working on implementing geospatial user-defined types and functions. Having examples of aggregations and window functions would be awesome! I did test out implementing a distributed convex hull as a UserDefinedAggregateFunction, and that seemed to work sensibly. Cheers, Jim On 12/15/2016 03:28 AM, Anton Okolnychyi wrote: Hi, I am wondering whether it makes sense to expand the Spark SQL programming guide with examples of aggregations (including user-defined via the Aggregator API) and window functions. For instance, there might be a separate subsection under "Getting Started" for each functionality. SPARK-16046 seems to be related but there is no activity for more than 4 months. Best regards, Anton
