For cast, you can use selectExpr method. For example,
df.selectExpr(cast(col1 as int) as col1, cast(col2 as bigint) as col2).
Or, df.select(df(colA).cast(int), ...)
On Thu, Apr 2, 2015 at 8:33 PM, Michael Armbrust mich...@databricks.com
wrote:
val df = Seq((test, 1)).toDF(col1, col2)
You can use SQL style expressions as a string:
df.filter(col1 IS NOT NULL).collect()
res1: Array[org.apache.spark.sql.Row] = Array([test,1])
Or you can also reference columns using df(colName) or quot;colName or
col(colName)
df.filter(df(col1) === test).collect()
res2: Array[org.apache.spark.sql.Row] = Array([test,1])
On Thu, Apr 2, 2015 at 7:45 PM, Yana Kadiyska yana.kadiy...@gmail.com
wrote:
Hi folks, having some seemingly noob issues with the dataframe API.
I have a DF which came from the csv package.
1. What would be an easy way to cast a column to a given type -- my DF
columns are all typed as strings coming from a csv. I see a schema getter
but not setter on DF
2. I am trying to use the syntax used in various blog posts but can't
figure out how to reference a column by name:
scala df.filter(customer_id!=)
console:23: error: overloaded method value filter with alternatives:
(conditionExpr: String)org.apache.spark.sql.DataFrame and
(condition: org.apache.spark.sql.Column)org.apache.spark.sql.DataFrame
cannot be applied to (Boolean)
df.filter(customer_id!=)
3. what would be the recommended way to drop a row containing a null
value -- is it possible to do this:
scala df.filter(customer_id IS NOT NULL)
