CalendarInterval is definitely externally visible.
E.g. sql("select interval 1 day").dtypes would return "Array[(String,
String)] = Array((interval 1 days,CalendarIntervalType))"
However, I'm not sure what it means to support casting. What are the
semantics for casting from any other data type to calendar interval? I can
see string casting and casting from itself, but not any other data types.
On Fri, Jul 27, 2018 at 10:34 AM Daniel Mateus Pires <dmate...@gmail.com>
wrote:
> Hi Sparkers! (maybe Sparkles ?)
>
> I just wanted to bring up the apparently ?controversial? Calendar Interval
> topic.
>
> I worked on: https://issues.apache.org/jira/browse/SPARK-24702,
> https://github.com/apache/spark/pull/21706
>
> The user was reporting an unexpected behaviour where he/she wasn’t able to
> cast to a Calendar Interval type.
>
> In the current version of Spark the following code works:
>
> scala> spark.sql("SELECT 'interval 1 hour' as
> a").select(col("a").cast("calendarinterval")).show()+----------------+|
> a|+----------------+|interval 1 hours|+----------------+
>
>
> While the following doesn’t:
> spark.sql("SELECT CALENDARINTERVAL('interval 1 hour') as a").show()
>
>
> Since the DataFrame API equivalent of the SQL worked, I thought adding it
> would be an easy decision to make (to make it consistent)
>
> However, I got push-back on the PR on the basis that “*we do not plan to
> expose Calendar Interval as a public type*”
> Should there be a consensus on either cleaning up the public DataFrame API
> out of CalendarIntervalType OR making it consistent with the SQL ?
>
> --
> Best regards,
> Daniel Mateus Pires
> Data Engineer @ Hudson's Bay Company
>
