Is there some place I can read more about it ? I can't find any reference.
I actully want to flatten these structures and not return them from the UDF.
Thanks,
Daniel
On Tue, Nov 25, 2014 at 8:44 PM, Michael Armbrust mich...@databricks.com
wrote:
Maps should just be scala maps, structs are
Hi,
I'm selecting columns from a json file, transform some of them and would
like to store the result as a parquet file but I'm failing.
This is what I'm doing:
val jsonFiles=sqlContext.jsonFile(/requests.loading)
jsonFiles.registerTempTable(jRequests)
val clean_jRequests=sqlContext.sql(select
Probably the easiest/closest way to do this would be with a UDF, something
like:
registerFunction(makeString, (s: Seq[String]) = s.mkString(,))
sql(SELECT *, makeString(c8) AS newC8 FROM jRequests)
Although this does not modify a column, but instead appends a new column.
Another more
Thank you.
How can I address more complex columns like maps and structs?
Thanks again!
Daniel
On 25 בנוב׳ 2014, at 19:43, Michael Armbrust mich...@databricks.com wrote:
Probably the easiest/closest way to do this would be with a UDF, something
like:
registerFunction(makeString, (s:
Maps should just be scala maps, structs are rows inside of rows. If you
wan to return a struct from a UDF you can do that with a case class.
On Tue, Nov 25, 2014 at 10:25 AM, Daniel Haviv danielru...@gmail.com
wrote:
Thank you.
How can I address more complex columns like maps and structs?
