[jira] [Commented] (SPARK-10735) CatalystTypeConverters MatchError converting RDD with custom object to dataframe

2016-01-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15103076#comment-15103076 ] Apache Spark commented on SPARK-10735: -- User 'davies' has created a pull request for this issue:

[jira] [Commented] (SPARK-10735) CatalystTypeConverters MatchError converting RDD with custom object to dataframe

2015-09-24 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14906862#comment-14906862 ] Thomas Graves commented on SPARK-10735: --- The issue here appears to be that in spark 1.4.1 it would

[jira] [Commented] (SPARK-10735) CatalystTypeConverters MatchError converting RDD with custom object to dataframe

2015-09-24 Thread Glenn Strycker (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14906877#comment-14906877 ] Glenn Strycker commented on SPARK-10735: This appears very similar to a problem I had earlier

[jira] [Commented] (SPARK-10735) CatalystTypeConverters MatchError converting RDD with custom object to dataframe

2015-09-24 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14906894#comment-14906894 ] Josh Rosen commented on SPARK-10735: [~tgraves], Spark 1.5.0 is stricter in its enforcement of types

[jira] [Commented] (SPARK-10735) CatalystTypeConverters MatchError converting RDD with custom object to dataframe

2015-09-24 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14906870#comment-14906870 ] Thomas Graves commented on SPARK-10735: --- [~joshrosen] it appears you did some of the refactoring

[jira] [Commented] (SPARK-10735) CatalystTypeConverters MatchError converting RDD with custom object to dataframe

2015-09-24 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14906931#comment-14906931 ] Thomas Graves commented on SPARK-10735: --- They are using SQLContext.createDataFrame(RDD, Class)

[jira] [Commented] (SPARK-10735) CatalystTypeConverters MatchError converting RDD with custom object to dataframe

2015-09-24 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14906936#comment-14906936 ] Josh Rosen commented on SPARK-10735: While UserDefinedType is currently exposed as {{@DeveloperAPI}}

[jira] [Commented] (SPARK-10735) CatalystTypeConverters MatchError converting RDD with custom object to dataframe

2015-09-24 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14907011#comment-14907011 ] Josh Rosen commented on SPARK-10735: I would _not_ recommend using UserDefinedType. UDT was intended

[jira] [Commented] (SPARK-10735) CatalystTypeConverters MatchError converting RDD with custom object to dataframe

2015-09-24 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14907072#comment-14907072 ] Thomas Graves commented on SPARK-10735: --- print schema for this shows: |-- beaconIds: array

[jira] [Commented] (SPARK-10735) CatalystTypeConverters MatchError converting RDD with custom object to dataframe

2015-09-24 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14906962#comment-14906962 ] Thomas Graves commented on SPARK-10735: --- It looks like there code created the dataframe but after