[jira] [Commented] (SPARK-19614) add type-preserving null function
[
https://issues.apache.org/jira/browse/SPARK-19614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870525#comment-15870525
]
Nick Dimiduk commented on SPARK-19614:
--
{{lit(null).cast(type)}} does exactly what I needed. Thanks fellas.
> add type-preserving null function
> -
>
> Key: SPARK-19614
> URL: https://issues.apache.org/jira/browse/SPARK-19614
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 2.1.0
>Reporter: Nick Dimiduk
>Priority: Trivial
>
> There's currently no easy way to extend the columns of a DataFrame with null
> columns that also preserves the type. {{lit(null)}} evaluates to
> {{Literal(null, NullType)}}, despite any subsequent hinting, for instance
> with {{Column.as(String, Metadata)}}. This comes up when programmatically
> munging data from disparate sources. A function such as {{null(DataType)}}
> would be nice.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19614) add type-preserving null function
[
https://issues.apache.org/jira/browse/SPARK-19614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870082#comment-15870082
]
Hyukjin Kwon commented on SPARK-19614:
--
[~ndimiduk]] Or maybe you are referring {{lit(null).cast(IntegerType)}}?
> add type-preserving null function
> -
>
> Key: SPARK-19614
> URL: https://issues.apache.org/jira/browse/SPARK-19614
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 2.1.0
>Reporter: Nick Dimiduk
>Priority: Trivial
>
> There's currently no easy way to extend the columns of a DataFrame with null
> columns that also preserves the type. {{lit(null)}} evaluates to
> {{Literal(null, NullType)}}, despite any subsequent hinting, for instance
> with {{Column.as(String, Metadata)}}. This comes up when programmatically
> munging data from disparate sources. A function such as {{null(DataType)}}
> would be nice.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19614) add type-preserving null function
[
https://issues.apache.org/jira/browse/SPARK-19614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869593#comment-15869593
]
Takeshi Yamamuro commented on SPARK-19614:
--
Is it much common for users to explicitly set NULL?
Anyway, you can set NULL with types;
{code}
Seq((1, 0), (2, 0)).toDF("a", "b").selectExpr("a", "b", "CAST(NULL AS INT)")
{code}
Is this not easy enough for your case?
> add type-preserving null function
> -
>
> Key: SPARK-19614
> URL: https://issues.apache.org/jira/browse/SPARK-19614
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 2.1.0
>Reporter: Nick Dimiduk
>Priority: Trivial
>
> There's currently no easy way to extend the columns of a DataFrame with null
> columns that also preserves the type. {{lit(null)}} evaluates to
> {{Literal(null, NullType)}}, despite any subsequent hinting, for instance
> with {{Column.as(String, Metadata)}}. This comes up when programmatically
> munging data from disparate sources. A function such as {{null(DataType)}}
> would be nice.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
