I'll start off by saying that the DSL is pretty experimental, and we are still figuring out exactly how to expose all of it to end users. Right now you are going to get more full featured functionality from SQL. Under the covers, using these two representations results in the same execution plans.
That said, you can create expressions using implicit conversions that are provided when you import a SQLContext. The leaves of these expression trees will be either `Attributes` (columns of the tables), or `Literals` (constant values). You can represent an attribute by creating a Scala Symbol<http://daily-scala.blogspot.com/2010/01/symbols.html> (an identifier that is prefixed with a '). When you perform an operation on a Symbol it gets converted into an attribute, and an expression is returned. scala> 'a + 1 res7: org.apache.spark.sql.catalyst.expressions.Add = ('a + 1) You can also construct more complex expressions: scala> 'b + 1 === 3 && 'c === 2 res9: org.apache.spark.sql.catalyst.expressions.And = ((('b + 1) = 3) && ('c = 2)) Note that sometimes we have to use special operators to avoid falling back on the standard JVM implementation. For example since symbols already have a method ==, doing the following does not give us what we want. scala> 'a == 'b res13: Boolean = false In constrast, since === is not defined on a symbol, the compiler is forced to use an implicit conversion from the SQLContext. This gives us the desired result. scala> 'a === 'b res14: org.apache.spark.sql.catalyst.expressions.Equals = ('a = 'b) Let me know if you have further questions! Michael On Wed, Apr 2, 2014 at 9:52 PM, All In A Days Work <allinadays...@gmail.com>wrote: > For various schemaRDD functions like select, where, orderby, groupby etc. > I would like to create expression objects and pass these to the methods for > execution. > > Can someone show some examples of how to create expressions for case class > and execute ? E.g., how to create expressions for select, order by, group > by etc. and execute methods using the expressions ? > > Regards, >
