Ok when I look at the schema it looks like KRYO makes one column is there a way to do a custom encoder with my own columns On Jan 25, 2016 1:30 PM, "Michael Armbrust" <mich...@databricks.com> wrote:
> The encoder is responsible for mapping your class onto some set of > columns. Try running: datasetMyType.printSchema() > > On Mon, Jan 25, 2016 at 1:16 PM, Steve Lewis <lordjoe2...@gmail.com> > wrote: > >> assume I have the following code >> >> SparkConf sparkConf = new SparkConf(); >> >> JavaSparkContext sqlCtx= new JavaSparkContext(sparkConf); >> >> JavaRDD<MyType> rddMyType= generateRDD(); // some code >> >> Encoder<MyType> evidence = Encoders.kryo(MyType.class); >> Dataset<MyType> datasetMyType= sqlCtx.createDataset( rddMyType.rdd(), >> evidence); >> >> Now I have a Dataset of MyType and assume there is some data. >> >> Assume MyType has bean fields with getters and setters as well as some >> internal collections and other data. What can I say about datasetMyType?? >> >> Does datasetMyType have columns and if so what? >> >> If not are there other ways to maka a DataSet with columns and if so what >> are they >> >> >> >
