Re: [Spark SQL]: Convert SchemaRDD back to RDD
This is not currently supported. Right now you can only get RDD[Row] as
Ted suggested.
On Sun, Feb 22, 2015 at 2:52 PM, Ted Yu yuzhih...@gmail.com wrote:
Haven't found the method in
http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.SchemaRDD
The new DataFrame has this method:
/**
* Returns the content of the [[DataFrame]] as an [[RDD]] of [[Row]]s.
* @group rdd
*/
def rdd: RDD[Row] = {
FYI
On Sun, Feb 22, 2015 at 11:51 AM, stephane.collot
stephane.col...@gmail.com wrote:
Hi Michael,
I think that the feature (convert a SchemaRDD to a structured class RDD)
is
now available. But I didn't understand in the PR how exactly to do this.
Can
you give an example or doc links?
Best regards
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Spark-SQL-Convert-SchemaRDD-back-to-RDD-tp9071p21753.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
Re: [Spark SQL]: Convert SchemaRDD back to RDD
Haven't found the method in
http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.SchemaRDD
The new DataFrame has this method:
/**
* Returns the content of the [[DataFrame]] as an [[RDD]] of [[Row]]s.
* @group rdd
*/
def rdd: RDD[Row] = {
FYI
On Sun, Feb 22, 2015 at 11:51 AM, stephane.collot stephane.col...@gmail.com
wrote:
Hi Michael,
I think that the feature (convert a SchemaRDD to a structured class RDD) is
now available. But I didn't understand in the PR how exactly to do this.
Can
you give an example or doc links?
Best regards
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Spark-SQL-Convert-SchemaRDD-back-to-RDD-tp9071p21753.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
Re: [Spark SQL]: Convert SchemaRDD back to RDD
Hi Michael, I think that the feature (convert a SchemaRDD to a structured class RDD) is now available. But I didn't understand in the PR how exactly to do this. Can you give an example or doc links? Best regards -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-SQL-Convert-SchemaRDD-back-to-RDD-tp9071p21753.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: [Spark SQL]: Convert SchemaRDD back to RDD
On Tue, Jul 8, 2014 at 12:43 PM, Pierre B pierre.borckm...@realimpactanalytics.com wrote: 1/ Is there a way to convert a SchemaRDD (for instance loaded from a parquet file) back to a RDD of a given case class? There may be someday, but doing so will either require a lot of reflection or a bunch of macro magic. So while I think this would be cool, it will probably be a while before we can implement it, and it'll likely be experimental. 2/ Even better, is there a way to get the schema information from a SchemaRDD ? I am trying to figure out how to properly get the various fields of the Rows of a SchemaRDD. Knowing the schema (in the form of a Map?), I guess I could nicely use getInt, getString, ..., on each row. We are actively working on this (SPARK-2179 https://issues.apache.org/jira/browse/SPARK-2179). Hopefully there will be a PR soon, and we are targeting the 1.1 release.
Re: [Spark SQL]: Convert SchemaRDD back to RDD
Cool Thanks Michael! Message sent from a mobile device - excuse typos and abbreviations Le 8 juil. 2014 à 22:17, Michael Armbrust [via Apache Spark User List] ml-node+s1001560n9084...@n3.nabble.com a écrit : On Tue, Jul 8, 2014 at 12:43 PM, Pierre B [hidden email] wrote: 1/ Is there a way to convert a SchemaRDD (for instance loaded from a parquet file) back to a RDD of a given case class? There may be someday, but doing so will either require a lot of reflection or a bunch of macro magic. So while I think this would be cool, it will probably be a while before we can implement it, and it'll likely be experimental. 2/ Even better, is there a way to get the schema information from a SchemaRDD ? I am trying to figure out how to properly get the various fields of the Rows of a SchemaRDD. Knowing the schema (in the form of a Map?), I guess I could nicely use getInt, getString, ..., on each row. We are actively working on this (SPARK-2179). Hopefully there will be a PR soon, and we are targeting the 1.1 release. If you reply to this email, your message will be added to the discussion below: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-SQL-Convert-SchemaRDD-back-to-RDD-tp9071p9084.html To unsubscribe from [Spark SQL]: Convert SchemaRDD back to RDD, click here. NAML -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-SQL-Convert-SchemaRDD-back-to-RDD-tp9071p9090.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
