Re: udf StructField to JSON String

So I think in your case you’d do something more like: val jsontrans = new JsonSerializationTransformer[StructType].setInputCol(“event").setOutputCol(“eventJSON") > On Mar 11, 2016, at 3:51 PM, Tristan Nixon wrote: > > val jsontrans = new >

Re: udf StructField to JSON String

It’s pretty simple, really: import com.fasterxml.jackson.databind.ObjectMapper import org.apache.spark.ml.UnaryTransformer import org.apache.spark.ml.util.Identifiable import org.apache.spark.sql.types.{DataType, StringType} /** * A SparkML Transformer that will transform an * entity of type T

Re: udf StructField to JSON String

I would like to see the code as well Tristan! On Fri, Mar 11, 2016 at 1:53 PM Tristan Nixon wrote: > Have you looked at DataFrame.write.json( path )? > > https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.DataFrameWriter > > > On Mar 11, 2016,

Re: udf StructField to JSON String

df.select("event").toJSON On Fri, Mar 11, 2016 at 9:53 AM, Caires Vinicius wrote: > Hmm. I think my problem is a little more complex. I'm using > https://github.com/databricks/spark-redshift and when I read from JSON > file I got this schema. > > root > > |-- app: string

Re: udf StructField to JSON String

Hi Tristan, Mind sharing the relevant code? I'd like to learn the way you use Transformer to do so. Thanks! Jacek 11.03.2016 7:07 PM "Tristan Nixon" napisał(a): > I have a similar situation in an app of mine. I implemented a custom ML > Transformer that wraps the Jackson

Re: udf StructField to JSON String

I have a similar situation in an app of mine. I implemented a custom ML Transformer that wraps the Jackson ObjectMapper - this gives you full control over how your custom entities / structs are serialized. > On Mar 11, 2016, at 11:53 AM, Caires Vinicius wrote: > > Hmm. I

Re: udf StructField to JSON String

Hmm. I think my problem is a little more complex. I'm using https://github.com/databricks/spark-redshift and when I read from JSON file I got this schema. root |-- app: string (nullable = true) |-- ct: long (nullable = true) |-- event: struct (nullable = true) ||-- attributes: struct

Re: udf StructField to JSON String

Have you looked at DataFrame.write.json( path )? https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.DataFrameWriter > On Mar 11, 2016, at 7:15 AM, Caires Vinicius wrote: > > I have one DataFrame with nested StructField and I want to convert to

udf StructField to JSON String

I have one DataFrame with nested StructField and I want to convert to JSON String. There is anyway to accomplish this?