[jira] [Commented] (SPARK-14155) Hide UserDefinedType in Spark 2.0
[ https://issues.apache.org/jira/browse/SPARK-14155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16155811#comment-16155811 ] Maciej Szymkiewicz commented on SPARK-14155: [~barrybecker4] Nope: SPARK-7768 > Hide UserDefinedType in Spark 2.0 > - > > Key: SPARK-14155 > URL: https://issues.apache.org/jira/browse/SPARK-14155 > Project: Spark > Issue Type: Improvement > Components: SQL >Reporter: Reynold Xin >Assignee: Reynold Xin > Fix For: 2.0.0 > > > UserDefinedType is a developer API in Spark 1.x. > With very high probability we will create a new API for user-defined type > that also works well with column batches as well as encoders (datasets). In > Spark 2.0, let's make UserDefinedType private[spark] first. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-14155) Hide UserDefinedType in Spark 2.0
[ https://issues.apache.org/jira/browse/SPARK-14155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16155790#comment-16155790 ] Barry Becker commented on SPARK-14155: -- Does it work with datasets now in 2.1? > Hide UserDefinedType in Spark 2.0 > - > > Key: SPARK-14155 > URL: https://issues.apache.org/jira/browse/SPARK-14155 > Project: Spark > Issue Type: Improvement > Components: SQL >Reporter: Reynold Xin >Assignee: Reynold Xin > Fix For: 2.0.0 > > > UserDefinedType is a developer API in Spark 1.x. > With very high probability we will create a new API for user-defined type > that also works well with column batches as well as encoders (datasets). In > Spark 2.0, let's make UserDefinedType private[spark] first. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-14155) Hide UserDefinedType in Spark 2.0
[ https://issues.apache.org/jira/browse/SPARK-14155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677052#comment-15677052 ] Raghu Ganti commented on SPARK-14155: - Is there an update re this? Any timeline as to when UDTs will be working for Spark 2.0.x? > Hide UserDefinedType in Spark 2.0 > - > > Key: SPARK-14155 > URL: https://issues.apache.org/jira/browse/SPARK-14155 > Project: Spark > Issue Type: Improvement > Components: SQL >Reporter: Reynold Xin >Assignee: Reynold Xin > Fix For: 2.0.0 > > > UserDefinedType is a developer API in Spark 1.x. > With very high probability we will create a new API for user-defined type > that also works well with column batches as well as encoders (datasets). In > Spark 2.0, let's make UserDefinedType private[spark] first. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-14155) Hide UserDefinedType in Spark 2.0
[ https://issues.apache.org/jira/browse/SPARK-14155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15450612#comment-15450612 ] Robert Conrad commented on SPARK-14155: --- [~r...@databricks.com] echoing Maciej above - is there any progress on UDTs for datasets or a jira ticket we can follow? > Hide UserDefinedType in Spark 2.0 > - > > Key: SPARK-14155 > URL: https://issues.apache.org/jira/browse/SPARK-14155 > Project: Spark > Issue Type: Improvement > Components: SQL >Reporter: Reynold Xin >Assignee: Reynold Xin > Fix For: 2.0.0 > > > UserDefinedType is a developer API in Spark 1.x. > With very high probability we will create a new API for user-defined type > that also works well with column batches as well as encoders (datasets). In > Spark 2.0, let's make UserDefinedType private[spark] first. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-14155) Hide UserDefinedType in Spark 2.0
[ https://issues.apache.org/jira/browse/SPARK-14155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15401156#comment-15401156 ] Maciej Szymkiewicz commented on SPARK-14155: [~rxin] Is there any progress on that or some opened JIRA? > Hide UserDefinedType in Spark 2.0 > - > > Key: SPARK-14155 > URL: https://issues.apache.org/jira/browse/SPARK-14155 > Project: Spark > Issue Type: Improvement > Components: SQL >Reporter: Reynold Xin >Assignee: Reynold Xin > Fix For: 2.0.0 > > > UserDefinedType is a developer API in Spark 1.x. > With very high probability we will create a new API for user-defined type > that also works well with column batches as well as encoders (datasets). In > Spark 2.0, let's make UserDefinedType private[spark] first. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-14155) Hide UserDefinedType in Spark 2.0
[ https://issues.apache.org/jira/browse/SPARK-14155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15225248#comment-15225248 ] Reynold Xin commented on SPARK-14155: - Yes let's close that one. Can you link it to this one when you close? > Hide UserDefinedType in Spark 2.0 > - > > Key: SPARK-14155 > URL: https://issues.apache.org/jira/browse/SPARK-14155 > Project: Spark > Issue Type: Improvement > Components: SQL >Reporter: Reynold Xin >Assignee: Reynold Xin > Fix For: 2.0.0 > > > UserDefinedType is a developer API in Spark 1.x. > With very high probability we will create a new API for user-defined type > that also works well with column batches as well as encoders (datasets). In > Spark 2.0, let's make UserDefinedType private[spark] first. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-14155) Hide UserDefinedType in Spark 2.0
[ https://issues.apache.org/jira/browse/SPARK-14155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15225210#comment-15225210 ] Koert Kuipers commented on SPARK-14155: --- got it. i was running into issues with UDTs and Datasets as well. see https://issues.apache.org/jira/browse/SPARK-13326 close that one? > Hide UserDefinedType in Spark 2.0 > - > > Key: SPARK-14155 > URL: https://issues.apache.org/jira/browse/SPARK-14155 > Project: Spark > Issue Type: Improvement > Components: SQL >Reporter: Reynold Xin >Assignee: Reynold Xin > Fix For: 2.0.0 > > > UserDefinedType is a developer API in Spark 1.x. > With very high probability we will create a new API for user-defined type > that also works well with column batches as well as encoders (datasets). In > Spark 2.0, let's make UserDefinedType private[spark] first. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-14155) Hide UserDefinedType in Spark 2.0
[ https://issues.apache.org/jira/browse/SPARK-14155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15225091#comment-15225091 ] Reynold Xin commented on SPARK-14155: - We want to make UDTs actually work with Datasets, and we haven't had enough time to think about it yet. So the proposal here is to hide it and then open something up that could work with Datasets in 2.1. > Hide UserDefinedType in Spark 2.0 > - > > Key: SPARK-14155 > URL: https://issues.apache.org/jira/browse/SPARK-14155 > Project: Spark > Issue Type: Improvement > Components: SQL >Reporter: Reynold Xin >Assignee: Reynold Xin > Fix For: 2.0.0 > > > UserDefinedType is a developer API in Spark 1.x. > With very high probability we will create a new API for user-defined type > that also works well with column batches as well as encoders (datasets). In > Spark 2.0, let's make UserDefinedType private[spark] first. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-14155) Hide UserDefinedType in Spark 2.0
[ https://issues.apache.org/jira/browse/SPARK-14155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15219935#comment-15219935 ] koert kuipers commented on SPARK-14155: --- we are depending on these to use our own custom types in Dataset[Row] (basically treat them as byte arrays from Dataset perspective). also not sure what the point is of a user defined type if the user can not define it. is it used somewhere else in spark? > Hide UserDefinedType in Spark 2.0 > - > > Key: SPARK-14155 > URL: https://issues.apache.org/jira/browse/SPARK-14155 > Project: Spark > Issue Type: Improvement > Components: SQL >Reporter: Reynold Xin >Assignee: Reynold Xin > Fix For: 2.0.0 > > > UserDefinedType is a developer API in Spark 1.x. > With very high probability we will create a new API for user-defined type > that also works well with column batches as well as encoders (datasets). In > Spark 2.0, let's make UserDefinedType private[spark] first. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-14155) Hide UserDefinedType in Spark 2.0
[ https://issues.apache.org/jira/browse/SPARK-14155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15211654#comment-15211654 ] Jacek Laskowski commented on SPARK-14155: - Right! I'm saved. Thanks. > Hide UserDefinedType in Spark 2.0 > - > > Key: SPARK-14155 > URL: https://issues.apache.org/jira/browse/SPARK-14155 > Project: Spark > Issue Type: Improvement > Components: SQL >Reporter: Reynold Xin >Assignee: Reynold Xin > > UserDefinedType is a developer API in Spark 1.x. > With very high probability we will create a new API for user-defined type > that also works well with column batches as well as encoders (datasets). In > Spark 2.0, let's make UserDefinedType private[spark] first. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-14155) Hide UserDefinedType in Spark 2.0
[ https://issues.apache.org/jira/browse/SPARK-14155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15211625#comment-15211625 ] Reynold Xin commented on SPARK-14155: - This is not UDF. > Hide UserDefinedType in Spark 2.0 > - > > Key: SPARK-14155 > URL: https://issues.apache.org/jira/browse/SPARK-14155 > Project: Spark > Issue Type: Improvement > Components: SQL >Reporter: Reynold Xin >Assignee: Reynold Xin > > UserDefinedType is a developer API in Spark 1.x. > With very high probability we will create a new API for user-defined type > that also works well with column batches as well as encoders (datasets). In > Spark 2.0, let's make UserDefinedType private[spark] first. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-14155) Hide UserDefinedType in Spark 2.0
[ https://issues.apache.org/jira/browse/SPARK-14155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15211619#comment-15211619 ] Jacek Laskowski commented on SPARK-14155: - Doh, now when I've finally grasped the concept of UDFs and even taught few people how to use them you're moving it away from me :) A never-ending story it is to master Spark it seems...Lots to learn. > Hide UserDefinedType in Spark 2.0 > - > > Key: SPARK-14155 > URL: https://issues.apache.org/jira/browse/SPARK-14155 > Project: Spark > Issue Type: Improvement > Components: SQL >Reporter: Reynold Xin >Assignee: Reynold Xin > > UserDefinedType is a developer API in Spark 1.x. > With very high probability we will create a new API for user-defined type > that also works well with column batches as well as encoders (datasets). In > Spark 2.0, let's make UserDefinedType private[spark] first. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-14155) Hide UserDefinedType in Spark 2.0
[ https://issues.apache.org/jira/browse/SPARK-14155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15211601#comment-15211601 ] Apache Spark commented on SPARK-14155: -- User 'rxin' has created a pull request for this issue: https://github.com/apache/spark/pull/11955 > Hide UserDefinedType in Spark 2.0 > - > > Key: SPARK-14155 > URL: https://issues.apache.org/jira/browse/SPARK-14155 > Project: Spark > Issue Type: Improvement > Components: SQL >Reporter: Reynold Xin >Assignee: Reynold Xin > > UserDefinedType is a developer API in Spark 1.x. > With very high probability we will create a new API for user-defined type > that also works well with column batches as well as encoders (datasets). In > Spark 2.0, let's make UserDefinedType private[spark] first. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
