[
https://issues.apache.org/jira/browse/SPARK-7768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16592171#comment-16592171
]
Alexander commented on SPARK-7768:
----------------------------------
Ah, looking at this SO post what I am asking about (i.e. accessing the object
fields) it's not actually possible because the sqlType in the UserDefinedType
does not actually expose the schema fields externally.
[https://stackoverflow.com/questions/32440461/how-to-define-schema-for-custom-type-in-spark-sql/32454596#32454596]
This is good to know, expecially since the comments in Spark's MatrixUDT are
highly misleading:
{code:java}
/**
* User-defined type for [[Matrix]] in [[mllib-local]] which allows easy
interaction with SQL
* via [[org.apache.spark.sql.Dataset]].
*/
{code}
Based on that comment, you would think the whole reason UserDefinedType.sqlType
exists is to use these sub-fields for querying which is clearly not the case!
I also came across this SO post that basically says the same thing:
[https://stackoverflow.com/questions/32440461/how-to-define-schema-for-custom-type-in-spark-sql/32454596#32454596]
Spark Maintainers, what are your plans for UDTs in release 2.4? Care to share
with us confused, bewildered, unwashed masses?
> Make user-defined type (UDT) API public
> ---------------------------------------
>
> Key: SPARK-7768
> URL: https://issues.apache.org/jira/browse/SPARK-7768
> Project: Spark
> Issue Type: New Feature
> Components: SQL
> Reporter: Xiangrui Meng
> Priority: Critical
>
> As the demand for UDTs increases beyond sparse/dense vectors in MLlib, it
> would be nice to make the UDT API public in 1.5.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
