[jira] [Commented] (SPARK-15140) ensure input object of encoder is not null
[
https://issues.apache.org/jira/browse/SPARK-15140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15301654#comment-15301654
]
Apache Spark commented on SPARK-15140:
--
User 'cloud-fan' has created a pull request for this issue:
https://github.com/apache/spark/pull/13322
> ensure input object of encoder is not null
> --
>
> Key: SPARK-15140
> URL: https://issues.apache.org/jira/browse/SPARK-15140
> Project: Spark
> Issue Type: Improvement
>Reporter: Wenchen Fan
>
> Current we assume the input object for encoder won't be null, but we don't
> check it. For example, in 1.6 `Seq("a", null).toDS.collect` will throw NPE,
> in 2.0 this will return Array("a", null).
> We should define this behaviour more clearly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15140) ensure input object of encoder is not null
[
https://issues.apache.org/jira/browse/SPARK-15140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15296805#comment-15296805
]
Michael Armbrust commented on SPARK-15140:
--
I don't think you should ever get a null row back. So I think all the fields
being null makes sense.
> ensure input object of encoder is not null
> --
>
> Key: SPARK-15140
> URL: https://issues.apache.org/jira/browse/SPARK-15140
> Project: Spark
> Issue Type: Improvement
>Reporter: Wenchen Fan
>
> Current we assume the input object for encoder won't be null, but we don't
> check it. For example, in 1.6 `Seq("a", null).toDS.collect` will throw NPE,
> in 2.0 this will return Array("a", null).
> We should define this behaviour more clearly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15140) ensure input object of encoder is not null
[
https://issues.apache.org/jira/browse/SPARK-15140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15295313#comment-15295313
]
Wenchen Fan commented on SPARK-15140:
-
I'm a little worried about supporting null input object here, what should the
serialized row be? a null row or a row with all fields null?
> ensure input object of encoder is not null
> --
>
> Key: SPARK-15140
> URL: https://issues.apache.org/jira/browse/SPARK-15140
> Project: Spark
> Issue Type: Improvement
>Reporter: Wenchen Fan
>
> Current we assume the input object for encoder won't be null, but we don't
> check it. For example, in 1.6 `Seq("a", null).toDS.collect` will throw NPE,
> in 2.0 this will return Array("a", null).
> We should define this behaviour more clearly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15140) ensure input object of encoder is not null
[
https://issues.apache.org/jira/browse/SPARK-15140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15273100#comment-15273100
]
Michael Armbrust commented on SPARK-15140:
--
The 2.0 behavior seems correct. Ideally .toDS().collect() will always
round-trip the data without change.
> ensure input object of encoder is not null
> --
>
> Key: SPARK-15140
> URL: https://issues.apache.org/jira/browse/SPARK-15140
> Project: Spark
> Issue Type: Improvement
>Reporter: Wenchen Fan
>
> Current we assume the input object for encoder won't be null, but we don't
> check it. For example, in 1.6 `Seq("a", null).toDS.collect` will throw NPE,
> in 2.0 this will return Array("a", null).
> We should define this behaviour more clearly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15140) ensure input object of encoder is not null
[
https://issues.apache.org/jira/browse/SPARK-15140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15271885#comment-15271885
]
Wenchen Fan commented on SPARK-15140:
-
cc [~marmbrus] [~lian cheng]
> ensure input object of encoder is not null
> --
>
> Key: SPARK-15140
> URL: https://issues.apache.org/jira/browse/SPARK-15140
> Project: Spark
> Issue Type: Improvement
>Reporter: Wenchen Fan
>
> Current we assume the input object for encoder won't be null, but we don't
> check it. For example, in 1.6 `Seq("a", null).toDS.collect` will throw NPE,
> in 2.0 this will return Array("a", null).
> We should define this behaviour more clearly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
