May be you could try something like this using sparkSQL 1.4 and dataframes
student.join(Grade, Grade(student_id) === student(student_id), left)
.groupBy(id)
.agg(sum(grade(Marks)), avg(grade(Marks)))
You could refer to the following document :
https://spark.apache.org/docs/1.4.0/api/scala/index.html#org.apache.spark.sql.DataFrame
On Thu, Jun 25, 2015 at 5:16 PM, Richard Catlin richard.m.cat...@gmail.com
wrote:
I am looking to do something similar to this Postgres query in HiveQL. If
I have a DataFrame student and a DataFrame grade, is this possible?
I read in Learning Spark: Lightning-Fast Big Data Analysis that it should
be possible. It says in Chapter 9
SchemaRDDs can store several basic types, as well as structures and
arrays of these types. They use the HiveQL syntax for type definitions.
Table-9-1 shown
and goes on to say
The last type, structures, is simply represented as other Rows in Spark
SQL. All of these types can also be nested within each other; for example,
you can have arrays of structs, or maps that contain structs
Here is the url of the page that describes the Postgres:
http://stackoverflow.com/questions/10928210/postgresql-aggregate-array
SELECT s.name, array_agg(g.Mark) as marks
FROM student s
LEFT JOIN Grade g ON g.Student_id = s.Id
GROUP BY s.Id
Thank you.
Richard Catlin
