Hi folks, having some seemingly noob issues with the dataframe API. I have a DF which came from the csv package.
1. What would be an easy way to cast a column to a given type -- my DF columns are all typed as strings coming from a csv. I see a schema getter but not setter on DF 2. I am trying to use the syntax used in various blog posts but can't figure out how to reference a column by name: scala> df.filter("customer_id"!="") <console>:23: error: overloaded method value filter with alternatives: (conditionExpr: String)org.apache.spark.sql.DataFrame <and> (condition: org.apache.spark.sql.Column)org.apache.spark.sql.DataFrame cannot be applied to (Boolean) df.filter("customer_id"!="") 3. what would be the recommended way to drop a row containing a null value -- is it possible to do this: scala> df.filter("customer_id" IS NOT NULL)