[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user adrian-wang commented on a diff in the pull request: https://github.com/apache/spark/pull/7589#discussion_r35281284 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeFunctions.scala --- @@ -258,3 +305,59 @@ case class DateFormatClass(left: Expression, right: Expression) extends BinaryEx }) } } + +/** + * Time Adds Interval. + */ +case class TimeAdd(start: Expression, interval: Expression) --- End diff -- @rxin I do implemented the expression for add date/timestamp and interval --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/7589#issuecomment-123847318 Let's cast string to "date". not "timestamp" then. We should probably also add an expression for date/timestamp + interval. In that case, I don't think we want to do implicit type cast. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/7589#discussion_r35257029
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeFunctions.scala
---
@@ -62,6 +62,53 @@ case class CurrentTimestamp() extends LeafExpression
with CodegenFallback {
}
}
+/**
+ * Adds a number of days to startdate.
+ */
+case class DateAdd(startDate: Expression, days: Expression)
+ extends BinaryExpression with ImplicitCastInputTypes {
+
+ override def left: Expression = startDate
+ override def right: Expression = days
+
+ override def inputTypes: Seq[AbstractDataType] = Seq(DateType,
IntegerType)
--- End diff --
OK that makes sense. Let's use date for string then.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7589#issuecomment-123811600 [Test build #1163 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1163/console) for PR 7589 at commit [`1a68e03`](https://github.com/apache/spark/commit/1a68e0334caee8c2fff822261fd728bd6209b4b6). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class DateAdd(startDate: Expression, days: Expression)` * `case class DateSub(startDate: Expression, days: Expression)` * `case class TimeAdd(start: Expression, interval: Expression)` * `case class TimeSub(start: Expression, interval: Expression)` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7589#issuecomment-123782708 [Test build #1163 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1163/consoleFull) for PR 7589 at commit [`1a68e03`](https://github.com/apache/spark/commit/1a68e0334caee8c2fff822261fd728bd6209b4b6). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user adrian-wang commented on the pull request: https://github.com/apache/spark/pull/7589#issuecomment-123745566 @rxin On my second thought, I think we should keep date_add and date_sub as simple as it should be. When it comes to Datetime IntervalType computation, we need another node specially for this. Here are my considerations: 1. hive 1.2 supports interval type, but still keep date_add and date_sub simple. 2, use another node can keep the registered function the same as most database systems. 3. for strings, it is hard to decide whether it is a date or timestamp, this would be troublesome. For most databases, we use `date '2015-01-01' + interval ` to do this calculation. so the sql engine could know how we should take care of the string here. And I have several other TODOS for the simple POC shown here, as follows: 1. we need to let us be aware of the precision of the interval type. For the calculation here, we would return DateType if we are add a Date and an Year-Month Interval, otherwise we should return TimestampType. For POC shown here, I just use TimestampType as the general returning type. Maybe we need two interval types here. 2. I need to get add support in parser. to translate the `shiptime + interval ` into corresponding logical node. 2-1. the left operand of `timestamp '2015-01-01 11:22:33' + interval ` should be translated into a Cast that cast string into corresponding type. 2-2. Then we translate the calculation to a TimeAdd or a TimeSub. I think the whole stuff deserves another PR. cc @chenghao-intel --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7589#issuecomment-123743328 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7589#issuecomment-123743297 [Test build #38078 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/38078/consoleFull) for PR 7589 at commit [`1a68e03`](https://github.com/apache/spark/commit/1a68e0334caee8c2fff822261fd728bd6209b4b6). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7589#issuecomment-123743323 [Test build #38078 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/38078/console) for PR 7589 at commit [`1a68e03`](https://github.com/apache/spark/commit/1a68e0334caee8c2fff822261fd728bd6209b4b6). * This patch **fails to build**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class DateAdd(startDate: Expression, days: Expression)` * `case class DateSub(startDate: Expression, days: Expression)` * `case class TimeAdd(start: Expression, interval: Expression)` * `case class TimeSub(start: Expression, interval: Expression)` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7589#issuecomment-123742233 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7589#issuecomment-123742195 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user adrian-wang commented on a diff in the pull request:
https://github.com/apache/spark/pull/7589#discussion_r35196236
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeFunctions.scala
---
@@ -62,6 +62,53 @@ case class CurrentTimestamp() extends LeafExpression
with CodegenFallback {
}
}
+/**
+ * Adds a number of days to startdate.
+ */
+case class DateAdd(startDate: Expression, days: Expression)
+ extends BinaryExpression with ImplicitCastInputTypes {
+
+ override def left: Expression = startDate
+ override def right: Expression = days
+
+ override def inputTypes: Seq[AbstractDataType] = Seq(DateType,
IntegerType)
--- End diff --
hive and oracle use + sign
select birthday + interval 3 days from xxx;
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/7589#discussion_r35194124
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeFunctions.scala
---
@@ -62,6 +62,53 @@ case class CurrentTimestamp() extends LeafExpression
with CodegenFallback {
}
}
+/**
+ * Adds a number of days to startdate.
+ */
+case class DateAdd(startDate: Expression, days: Expression)
+ extends BinaryExpression with ImplicitCastInputTypes {
+
+ override def left: Expression = startDate
+ override def right: Expression = days
+
+ override def inputTypes: Seq[AbstractDataType] = Seq(DateType,
IntegerType)
--- End diff --
`date_add` is used for adding days, where does hive support `timestamp +
interval`? in `+` or another special operator?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user adrian-wang commented on a diff in the pull request:
https://github.com/apache/spark/pull/7589#discussion_r35189328
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeFunctions.scala
---
@@ -62,6 +62,53 @@ case class CurrentTimestamp() extends LeafExpression
with CodegenFallback {
}
}
+/**
+ * Adds a number of days to startdate.
+ */
+case class DateAdd(startDate: Expression, days: Expression)
+ extends BinaryExpression with ImplicitCastInputTypes {
+
+ override def left: Expression = startDate
+ override def right: Expression = days
+
+ override def inputTypes: Seq[AbstractDataType] = Seq(DateType,
IntegerType)
--- End diff --
I just checked with hive 1.2.
hive> select date_add("2015-01-01 00:11:22", 2);
2015-01-03
hive> select date_add(cast("2015-01-01 00:11:22" as timestamp), 2);
2015-01-03
shall we use the same pattern?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user adrian-wang commented on a diff in the pull request:
https://github.com/apache/spark/pull/7589#discussion_r35187996
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeFunctions.scala
---
@@ -62,6 +62,53 @@ case class CurrentTimestamp() extends LeafExpression
with CodegenFallback {
}
}
+/**
+ * Adds a number of days to startdate.
+ */
+case class DateAdd(startDate: Expression, days: Expression)
+ extends BinaryExpression with ImplicitCastInputTypes {
+
+ override def left: Expression = startDate
+ override def right: Expression = days
+
+ override def inputTypes: Seq[AbstractDataType] = Seq(DateType,
IntegerType)
--- End diff --
I think date_add('2015-07-22', 1) return '2015-07-23 00:00:00' is not so
natural...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/7589#discussion_r35187647
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeFunctions.scala
---
@@ -62,6 +62,53 @@ case class CurrentTimestamp() extends LeafExpression
with CodegenFallback {
}
}
+/**
+ * Adds a number of days to startdate.
+ */
+case class DateAdd(startDate: Expression, days: Expression)
+ extends BinaryExpression with ImplicitCastInputTypes {
+
+ override def left: Expression = startDate
+ override def right: Expression = days
+
+ override def inputTypes: Seq[AbstractDataType] = Seq(DateType,
IntegerType)
--- End diff --
i think we can just put timestamp first, and as a result implicitly cast
string to timestamp.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user adrian-wang commented on a diff in the pull request:
https://github.com/apache/spark/pull/7589#discussion_r35187201
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeFunctions.scala
---
@@ -62,6 +62,53 @@ case class CurrentTimestamp() extends LeafExpression
with CodegenFallback {
}
}
+/**
+ * Adds a number of days to startdate.
+ */
+case class DateAdd(startDate: Expression, days: Expression)
+ extends BinaryExpression with ImplicitCastInputTypes {
+
+ override def left: Expression = startDate
+ override def right: Expression = days
+
+ override def inputTypes: Seq[AbstractDataType] = Seq(DateType,
IntegerType)
--- End diff --
Yes, but then we have to decide it is date or timestamp, and that need to
be done at run time.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/7589#discussion_r35187060
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeFunctions.scala
---
@@ -62,6 +62,53 @@ case class CurrentTimestamp() extends LeafExpression
with CodegenFallback {
}
}
+/**
+ * Adds a number of days to startdate.
+ */
+case class DateAdd(startDate: Expression, days: Expression)
+ extends BinaryExpression with ImplicitCastInputTypes {
+
+ override def left: Expression = startDate
+ override def right: Expression = days
+
+ override def inputTypes: Seq[AbstractDataType] = Seq(DateType,
IntegerType)
--- End diff --
string can get implicitly cast to date or timestamp, can't it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user adrian-wang commented on a diff in the pull request:
https://github.com/apache/spark/pull/7589#discussion_r35186862
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeFunctions.scala
---
@@ -62,6 +62,53 @@ case class CurrentTimestamp() extends LeafExpression
with CodegenFallback {
}
}
+/**
+ * Adds a number of days to startdate.
+ */
+case class DateAdd(startDate: Expression, days: Expression)
+ extends BinaryExpression with ImplicitCastInputTypes {
+
+ override def left: Expression = startDate
+ override def right: Expression = days
+
+ override def inputTypes: Seq[AbstractDataType] = Seq(DateType,
IntegerType)
--- End diff --
OK, will update today. BTW, How do we tell the difference if we are doing
string + interval/int,
should we just take strings as date?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7589#issuecomment-123585577 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7589#issuecomment-123585560 [Test build #38051 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/38051/consoleFull) for PR 7589 at commit [`c506661`](https://github.com/apache/spark/commit/c50666195d43fa652ea1cb8a7eef3d1fb3504659). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7589#issuecomment-123585576 [Test build #38051 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/38051/console) for PR 7589 at commit [`c506661`](https://github.com/apache/spark/commit/c50666195d43fa652ea1cb8a7eef3d1fb3504659). * This patch **fails to build**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class DateAdd(startDate: Expression, days: Expression)` * `case class DateSub(startDate: Expression, days: Expression)` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/7589#discussion_r35186500
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeFunctions.scala
---
@@ -62,6 +62,53 @@ case class CurrentTimestamp() extends LeafExpression
with CodegenFallback {
}
}
+/**
+ * Adds a number of days to startdate.
+ */
+case class DateAdd(startDate: Expression, days: Expression)
+ extends BinaryExpression with ImplicitCastInputTypes {
+
+ override def left: Expression = startDate
+ override def right: Expression = days
+
+ override def inputTypes: Seq[AbstractDataType] = Seq(DateType,
IntegerType)
--- End diff --
i think we want to support:
timestamp + interval returns timestamp
date + interval returns date
date + int returns date ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7589#issuecomment-123584836 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7589#issuecomment-123584769 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
GitHub user adrian-wang opened a pull request: https://github.com/apache/spark/pull/7589 [SPARK-8186] [SPARK-8187] [SQL] datetime function: date_add, date_sub This subsumes #6782 You can merge this pull request into a Git repository by running: $ git pull https://github.com/adrian-wang/spark udfdatecal Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/7589.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #7589 commit c50666195d43fa652ea1cb8a7eef3d1fb3504659 Author: Daoyuan Wang Date: 2015-07-22T07:04:53Z function date_add , date_sub --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user adrian-wang commented on the pull request: https://github.com/apache/spark/pull/7589#issuecomment-123584116 cc @chenghao-intel --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user adrian-wang closed the pull request at: https://github.com/apache/spark/pull/6782 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user adrian-wang commented on the pull request: https://github.com/apache/spark/pull/6782#issuecomment-116949501 OK, I'll close these for now. Could you assign them all to me in case duplicated work here? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/6782#issuecomment-116888516 @adrian-wang I thought about this more -- I think we need to go back to the drawing board to specify the behavior of various date functions, return types, before we can merge these expressions. Some open questions include: - what data type should these expressions accept? - what data type should these expressions return? - should we support intervals? ... I will make sure the design happens next week. Meantime, can you close the expressions related to dates? We can reopen them once we flush out the design details. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user brkyvz commented on a diff in the pull request:
https://github.com/apache/spark/pull/6782#discussion_r33526888
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -946,6 +947,38 @@ object functions {
def cosh(columnName: String): Column = cosh(Column(columnName))
/**
+ * Adds a number of days to startDate, given values for startDate and
days
+ *
+ * @group datetime_funcs
+ * @since 1.5.0
+ */
+ def date_add(startDate: Column, days: Column): Column =
DateAdd(startDate.expr, days.expr)
+
+ /**
+ * Adds a number of days to startDate, given column names for startDate
and days
+ *
+ * @group datetime_funcs
+ * @since 1.5.0
+ */
+ def date_add(startDate: String, days: String): Column =
date_add(Column(startDate), Column(days))
--- End diff --
@davies In those, you have to accept any combination, because they are math
functions. I may want to take the power of 2 with the values in my DataFrame
column. I also should be able to take the power of my Column by 2. If we can
have a case where `f(a, b) == f(b, a)`, we wouldn't need all combinations, but
those functions (pow, atan2) don't necessarily have `f(a, b) == f(b, a)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/6782#discussion_r33526479
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeFunctions.scala
---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions
+
+import org.apache.spark.sql.catalyst.analysis.TypeCheckResult
+import
org.apache.spark.sql.catalyst.expressions.codegen.{GeneratedExpressionCode,
CodeGenContext}
+import org.apache.spark.sql.catalyst.util.DateUtils
+import org.apache.spark.sql.types._
+import org.apache.spark.unsafe.types.UTF8String
+
+/**
+ * Adds a number of days to startdate: date_add('2008-12-31', 1) =
'2009-01-01'.
+ */
+case class DateAdd(startDate: Expression, days: Expression) extends
Expression {
+ override def children: Seq[Expression] = startDate :: days :: Nil
+
+ override def foldable: Boolean = startDate.foldable && days.foldable
+ override def nullable: Boolean = startDate.nullable || days.nullable
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+val supportedLeftType = Seq(StringType, DateType, TimestampType,
NullType)
+if (!supportedLeftType.contains(startDate.dataType)) {
+ TypeCheckResult.TypeCheckFailure(
+s"type of startdate expression in DateAdd should be
string/timestamp/date," +
+ s" not ${startDate.dataType}")
+} else if (days.dataType != IntegerType && days.dataType != NullType) {
+ TypeCheckResult.TypeCheckFailure(
+s"type of days expression in DateAdd should be int, not
${days.dataType}.")
+} else {
+ TypeCheckResult.TypeCheckSuccess
--- End diff --
@davies agree that 100% compatibility is not a goal and mysql's behavior
seems reasonable here. It would be good though to more holistically design our
data handling API and make sure we aren't missing anything. For example, today
we can handle queries like: `WHERE timestamp > "2014-01"` but only because we
convert the timestamp to a string and the strings compare lexicographically as
you would hope.
However, this feels like a hack to me and it would be much more efficient
if we could avoid creating strings for every date and instead could covert
"2014-01" into an `Int` internally. How does mysql handle partial dates like
this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/6782#discussion_r4134
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeFunctions.scala
---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions
+
+import org.apache.spark.sql.catalyst.analysis.TypeCheckResult
+import
org.apache.spark.sql.catalyst.expressions.codegen.{GeneratedExpressionCode,
CodeGenContext}
+import org.apache.spark.sql.catalyst.util.DateUtils
+import org.apache.spark.sql.types._
+import org.apache.spark.unsafe.types.UTF8String
+
+/**
+ * Adds a number of days to startdate: date_add('2008-12-31', 1) =
'2009-01-01'.
+ */
+case class DateAdd(startDate: Expression, days: Expression) extends
Expression {
+ override def children: Seq[Expression] = startDate :: days :: Nil
+
+ override def foldable: Boolean = startDate.foldable && days.foldable
+ override def nullable: Boolean = startDate.nullable || days.nullable
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+val supportedLeftType = Seq(StringType, DateType, TimestampType,
NullType)
+if (!supportedLeftType.contains(startDate.dataType)) {
+ TypeCheckResult.TypeCheckFailure(
+s"type of startdate expression in DateAdd should be
string/timestamp/date," +
+ s" not ${startDate.dataType}")
+} else if (days.dataType != IntegerType && days.dataType != NullType) {
+ TypeCheckResult.TypeCheckFailure(
+s"type of days expression in DateAdd should be int, not
${days.dataType}.")
+} else {
+ TypeCheckResult.TypeCheckSuccess
--- End diff --
I see. I checked MySQL, it can cast '2012-01-01 12:21:12' to Date:
```
mysql> select cast('2012-01-01 12:21:12' as date);
+-+
| cast('2012-01-01 12:21:12' as date) |
+-+
| 2012-01-01 |
+-+
1 row in set (0.00 sec)
```
This sounds reasonable to me, but not sure making it different than Hive
will break real workload or not.
Because of all kinds of wired behavior in Hive, we cannot have 100%
compatibility with Hive already, see many hive tests are skipped. @marmbrus,
what do yo think?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user adrian-wang commented on a diff in the pull request:
https://github.com/apache/spark/pull/6782#discussion_r3522
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeFunctions.scala
---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions
+
+import org.apache.spark.sql.catalyst.analysis.TypeCheckResult
+import
org.apache.spark.sql.catalyst.expressions.codegen.{GeneratedExpressionCode,
CodeGenContext}
+import org.apache.spark.sql.catalyst.util.DateUtils
+import org.apache.spark.sql.types._
+import org.apache.spark.unsafe.types.UTF8String
+
+/**
+ * Adds a number of days to startdate: date_add('2008-12-31', 1) =
'2009-01-01'.
+ */
+case class DateAdd(startDate: Expression, days: Expression) extends
Expression {
+ override def children: Seq[Expression] = startDate :: days :: Nil
+
+ override def foldable: Boolean = startDate.foldable && days.foldable
+ override def nullable: Boolean = startDate.nullable || days.nullable
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+val supportedLeftType = Seq(StringType, DateType, TimestampType,
NullType)
+if (!supportedLeftType.contains(startDate.dataType)) {
+ TypeCheckResult.TypeCheckFailure(
+s"type of startdate expression in DateAdd should be
string/timestamp/date," +
+ s" not ${startDate.dataType}")
+} else if (days.dataType != IntegerType && days.dataType != NullType) {
+ TypeCheckResult.TypeCheckFailure(
+s"type of days expression in DateAdd should be int, not
${days.dataType}.")
+} else {
+ TypeCheckResult.TypeCheckSuccess
--- End diff --
for string type of time, the tests are in
https://github.com/apache/hive/blob/master/ql/src/test/queries/clientpositive/timestamp_udf.q
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user adrian-wang commented on a diff in the pull request:
https://github.com/apache/spark/pull/6782#discussion_r3367
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeFunctions.scala
---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions
+
+import org.apache.spark.sql.catalyst.analysis.TypeCheckResult
+import
org.apache.spark.sql.catalyst.expressions.codegen.{GeneratedExpressionCode,
CodeGenContext}
+import org.apache.spark.sql.catalyst.util.DateUtils
+import org.apache.spark.sql.types._
+import org.apache.spark.unsafe.types.UTF8String
+
+/**
+ * Adds a number of days to startdate: date_add('2008-12-31', 1) =
'2009-01-01'.
+ */
+case class DateAdd(startDate: Expression, days: Expression) extends
Expression {
+ override def children: Seq[Expression] = startDate :: days :: Nil
+
+ override def foldable: Boolean = startDate.foldable && days.foldable
+ override def nullable: Boolean = startDate.nullable || days.nullable
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+val supportedLeftType = Seq(StringType, DateType, TimestampType,
NullType)
+if (!supportedLeftType.contains(startDate.dataType)) {
+ TypeCheckResult.TypeCheckFailure(
+s"type of startdate expression in DateAdd should be
string/timestamp/date," +
+ s" not ${startDate.dataType}")
+} else if (days.dataType != IntegerType && days.dataType != NullType) {
+ TypeCheckResult.TypeCheckFailure(
+s"type of days expression in DateAdd should be int, not
${days.dataType}.")
+} else {
+ TypeCheckResult.TypeCheckSuccess
--- End diff --
@davies you can find more test in
https://github.com/apache/hive/blob/master/ql/src/test/queries/clientpositive/date_udf.q
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/6782#discussion_r2371
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeFunctions.scala
---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions
+
+import org.apache.spark.sql.catalyst.analysis.TypeCheckResult
+import
org.apache.spark.sql.catalyst.expressions.codegen.{GeneratedExpressionCode,
CodeGenContext}
+import org.apache.spark.sql.catalyst.util.DateUtils
+import org.apache.spark.sql.types._
+import org.apache.spark.unsafe.types.UTF8String
+
+/**
+ * Adds a number of days to startdate: date_add('2008-12-31', 1) =
'2009-01-01'.
+ */
+case class DateAdd(startDate: Expression, days: Expression) extends
Expression {
+ override def children: Seq[Expression] = startDate :: days :: Nil
+
+ override def foldable: Boolean = startDate.foldable && days.foldable
+ override def nullable: Boolean = startDate.nullable || days.nullable
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+val supportedLeftType = Seq(StringType, DateType, TimestampType,
NullType)
+if (!supportedLeftType.contains(startDate.dataType)) {
+ TypeCheckResult.TypeCheckFailure(
+s"type of startdate expression in DateAdd should be
string/timestamp/date," +
+ s" not ${startDate.dataType}")
+} else if (days.dataType != IntegerType && days.dataType != NullType) {
+ TypeCheckResult.TypeCheckFailure(
+s"type of days expression in DateAdd should be int, not
${days.dataType}.")
+} else {
+ TypeCheckResult.TypeCheckSuccess
--- End diff --
@adrian-wang Thanks for the confirm. From the HIVE test [1], I see date_add
works with tinyint, smallint, int, but there is no case for "2010-01-01
01:02:03", is it not tested side behavior?
[1]
https://github.com/apache/hive/blob/master/ql/src/test/queries/clientpositive/udf_date_add.q
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user adrian-wang commented on a diff in the pull request:
https://github.com/apache/spark/pull/6782#discussion_r1698
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeFunctions.scala
---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions
+
+import org.apache.spark.sql.catalyst.analysis.TypeCheckResult
+import
org.apache.spark.sql.catalyst.expressions.codegen.{GeneratedExpressionCode,
CodeGenContext}
+import org.apache.spark.sql.catalyst.util.DateUtils
+import org.apache.spark.sql.types._
+import org.apache.spark.unsafe.types.UTF8String
+
+/**
+ * Adds a number of days to startdate: date_add('2008-12-31', 1) =
'2009-01-01'.
+ */
+case class DateAdd(startDate: Expression, days: Expression) extends
Expression {
+ override def children: Seq[Expression] = startDate :: days :: Nil
+
+ override def foldable: Boolean = startDate.foldable && days.foldable
+ override def nullable: Boolean = startDate.nullable || days.nullable
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+val supportedLeftType = Seq(StringType, DateType, TimestampType,
NullType)
+if (!supportedLeftType.contains(startDate.dataType)) {
+ TypeCheckResult.TypeCheckFailure(
+s"type of startdate expression in DateAdd should be
string/timestamp/date," +
+ s" not ${startDate.dataType}")
+} else if (days.dataType != IntegerType && days.dataType != NullType) {
+ TypeCheckResult.TypeCheckFailure(
+s"type of days expression in DateAdd should be int, not
${days.dataType}.")
+} else {
+ TypeCheckResult.TypeCheckSuccess
--- End diff --
Actually I'll abstract this part as a trait to handle this, but not in this
one. This PR has to create these files in upstream first, so that I can get my
work continued.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/6782#discussion_r1507
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -946,6 +947,38 @@ object functions {
def cosh(columnName: String): Column = cosh(Column(columnName))
/**
+ * Adds a number of days to startDate, given values for startDate and
days
+ *
+ * @group datetime_funcs
+ * @since 1.5.0
+ */
+ def date_add(startDate: Column, days: Column): Column =
DateAdd(startDate.expr, days.expr)
+
+ /**
+ * Adds a number of days to startDate, given column names for startDate
and days
+ *
+ * @group datetime_funcs
+ * @since 1.5.0
+ */
+ def date_add(startDate: String, days: String): Column =
date_add(Column(startDate), Column(days))
--- End diff --
Good point, we have many functions which take two or more parameters, all
the parameters could Column or Literal, we usually take only a few combination
of them as DataFrame function. It's hard to have a rule to follow, but we
usually take the first one as Column, and others as Literal.
There are a few cases that we have all the combinations (for example,
atan2, pow), that's bad, we should revisit them, cc @rxin @brkyvz
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/6782#discussion_r1514
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeFunctions.scala
---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions
+
+import org.apache.spark.sql.catalyst.analysis.TypeCheckResult
+import
org.apache.spark.sql.catalyst.expressions.codegen.{GeneratedExpressionCode,
CodeGenContext}
+import org.apache.spark.sql.catalyst.util.DateUtils
+import org.apache.spark.sql.types._
+import org.apache.spark.unsafe.types.UTF8String
+
+/**
+ * Adds a number of days to startdate: date_add('2008-12-31', 1) =
'2009-01-01'.
+ */
+case class DateAdd(startDate: Expression, days: Expression) extends
Expression {
+ override def children: Seq[Expression] = startDate :: days :: Nil
+
+ override def foldable: Boolean = startDate.foldable && days.foldable
+ override def nullable: Boolean = startDate.nullable || days.nullable
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+val supportedLeftType = Seq(StringType, DateType, TimestampType,
NullType)
+if (!supportedLeftType.contains(startDate.dataType)) {
+ TypeCheckResult.TypeCheckFailure(
+s"type of startdate expression in DateAdd should be
string/timestamp/date," +
+ s" not ${startDate.dataType}")
+} else if (days.dataType != IntegerType && days.dataType != NullType) {
+ TypeCheckResult.TypeCheckFailure(
+s"type of days expression in DateAdd should be int, not
${days.dataType}.")
+} else {
+ TypeCheckResult.TypeCheckSuccess
--- End diff --
How about we create a `StringToDate` Expression that cast date or timestamp
string to Date? If a lot of date functions need to support string, then it
would be nasty to write code to handle `StringType` in all of them. It makes
more sense to me that we can handle String at `HiveTypeCoercion`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user adrian-wang commented on a diff in the pull request:
https://github.com/apache/spark/pull/6782#discussion_r1281
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeFunctions.scala
---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions
+
+import org.apache.spark.sql.catalyst.analysis.TypeCheckResult
+import
org.apache.spark.sql.catalyst.expressions.codegen.{GeneratedExpressionCode,
CodeGenContext}
+import org.apache.spark.sql.catalyst.util.DateUtils
+import org.apache.spark.sql.types._
+import org.apache.spark.unsafe.types.UTF8String
+
+/**
+ * Adds a number of days to startdate: date_add('2008-12-31', 1) =
'2009-01-01'.
+ */
+case class DateAdd(startDate: Expression, days: Expression) extends
Expression {
+ override def children: Seq[Expression] = startDate :: days :: Nil
+
+ override def foldable: Boolean = startDate.foldable && days.foldable
+ override def nullable: Boolean = startDate.nullable || days.nullable
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+val supportedLeftType = Seq(StringType, DateType, TimestampType,
NullType)
+if (!supportedLeftType.contains(startDate.dataType)) {
+ TypeCheckResult.TypeCheckFailure(
+s"type of startdate expression in DateAdd should be
string/timestamp/date," +
+ s" not ${startDate.dataType}")
+} else if (days.dataType != IntegerType && days.dataType != NullType) {
+ TypeCheckResult.TypeCheckFailure(
+s"type of days expression in DateAdd should be int, not
${days.dataType}.")
+} else {
+ TypeCheckResult.TypeCheckSuccess
--- End diff --
sure,
hql> select cast ("2010-01-01 00:00:00" as date) ;
NULL
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/6782#discussion_r1122
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeFunctions.scala
---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions
+
+import org.apache.spark.sql.catalyst.analysis.TypeCheckResult
+import
org.apache.spark.sql.catalyst.expressions.codegen.{GeneratedExpressionCode,
CodeGenContext}
+import org.apache.spark.sql.catalyst.util.DateUtils
+import org.apache.spark.sql.types._
+import org.apache.spark.unsafe.types.UTF8String
+
+/**
+ * Adds a number of days to startdate: date_add('2008-12-31', 1) =
'2009-01-01'.
+ */
+case class DateAdd(startDate: Expression, days: Expression) extends
Expression {
+ override def children: Seq[Expression] = startDate :: days :: Nil
+
+ override def foldable: Boolean = startDate.foldable && days.foldable
+ override def nullable: Boolean = startDate.nullable || days.nullable
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+val supportedLeftType = Seq(StringType, DateType, TimestampType,
NullType)
+if (!supportedLeftType.contains(startDate.dataType)) {
+ TypeCheckResult.TypeCheckFailure(
+s"type of startdate expression in DateAdd should be
string/timestamp/date," +
+ s" not ${startDate.dataType}")
+} else if (days.dataType != IntegerType && days.dataType != NullType) {
+ TypeCheckResult.TypeCheckFailure(
+s"type of days expression in DateAdd should be int, not
${days.dataType}.")
+} else {
+ TypeCheckResult.TypeCheckSuccess
--- End diff --
It does not make sense to me that date_add() can treat "2010-01-01
01:02:03" as `Date(2010-01-01)` but `CAST` cannot.
Does Hive have this kind of behavior?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user adrian-wang commented on a diff in the pull request:
https://github.com/apache/spark/pull/6782#discussion_r0601
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeFunctions.scala
---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions
+
+import org.apache.spark.sql.catalyst.analysis.TypeCheckResult
+import
org.apache.spark.sql.catalyst.expressions.codegen.{GeneratedExpressionCode,
CodeGenContext}
+import org.apache.spark.sql.catalyst.util.DateUtils
+import org.apache.spark.sql.types._
+import org.apache.spark.unsafe.types.UTF8String
+
+/**
+ * Adds a number of days to startdate: date_add('2008-12-31', 1) =
'2009-01-01'.
+ */
+case class DateAdd(startDate: Expression, days: Expression) extends
Expression {
+ override def children: Seq[Expression] = startDate :: days :: Nil
+
+ override def foldable: Boolean = startDate.foldable && days.foldable
+ override def nullable: Boolean = startDate.nullable || days.nullable
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+val supportedLeftType = Seq(StringType, DateType, TimestampType,
NullType)
+if (!supportedLeftType.contains(startDate.dataType)) {
+ TypeCheckResult.TypeCheckFailure(
+s"type of startdate expression in DateAdd should be
string/timestamp/date," +
+ s" not ${startDate.dataType}")
+} else if (days.dataType != IntegerType && days.dataType != NullType) {
+ TypeCheckResult.TypeCheckFailure(
+s"type of days expression in DateAdd should be int, not
${days.dataType}.")
+} else {
+ TypeCheckResult.TypeCheckSuccess
--- End diff --
I think change the default behavior of cast is not a good choice.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/6782#discussion_r0538
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeFunctions.scala
---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions
+
+import org.apache.spark.sql.catalyst.analysis.TypeCheckResult
+import
org.apache.spark.sql.catalyst.expressions.codegen.{GeneratedExpressionCode,
CodeGenContext}
+import org.apache.spark.sql.catalyst.util.DateUtils
+import org.apache.spark.sql.types._
+import org.apache.spark.unsafe.types.UTF8String
+
+/**
+ * Adds a number of days to startdate: date_add('2008-12-31', 1) =
'2009-01-01'.
+ */
+case class DateAdd(startDate: Expression, days: Expression) extends
Expression {
+ override def children: Seq[Expression] = startDate :: days :: Nil
+
+ override def foldable: Boolean = startDate.foldable && days.foldable
+ override def nullable: Boolean = startDate.nullable || days.nullable
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+val supportedLeftType = Seq(StringType, DateType, TimestampType,
NullType)
+if (!supportedLeftType.contains(startDate.dataType)) {
+ TypeCheckResult.TypeCheckFailure(
+s"type of startdate expression in DateAdd should be
string/timestamp/date," +
+ s" not ${startDate.dataType}")
+} else if (days.dataType != IntegerType && days.dataType != NullType) {
+ TypeCheckResult.TypeCheckFailure(
+s"type of days expression in DateAdd should be int, not
${days.dataType}.")
+} else {
+ TypeCheckResult.TypeCheckSuccess
--- End diff --
How about make it could cast "2010-01-01" or "2010-01-01 00:00:00" into
Date without runtime error?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user adrian-wang commented on a diff in the pull request:
https://github.com/apache/spark/pull/6782#discussion_r33214991
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeFunctions.scala
---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions
+
+import org.apache.spark.sql.catalyst.analysis.TypeCheckResult
+import
org.apache.spark.sql.catalyst.expressions.codegen.{GeneratedExpressionCode,
CodeGenContext}
+import org.apache.spark.sql.catalyst.util.DateUtils
+import org.apache.spark.sql.types._
+import org.apache.spark.unsafe.types.UTF8String
+
+/**
+ * Adds a number of days to startdate: date_add('2008-12-31', 1) =
'2009-01-01'.
+ */
+case class DateAdd(startDate: Expression, days: Expression) extends
Expression {
+ override def children: Seq[Expression] = startDate :: days :: Nil
+
+ override def foldable: Boolean = startDate.foldable && days.foldable
+ override def nullable: Boolean = startDate.nullable || days.nullable
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+val supportedLeftType = Seq(StringType, DateType, TimestampType,
NullType)
+if (!supportedLeftType.contains(startDate.dataType)) {
+ TypeCheckResult.TypeCheckFailure(
+s"type of startdate expression in DateAdd should be
string/timestamp/date," +
+ s" not ${startDate.dataType}")
+} else if (days.dataType != IntegerType && days.dataType != NullType) {
+ TypeCheckResult.TypeCheckFailure(
+s"type of days expression in DateAdd should be int, not
${days.dataType}.")
+} else {
+ TypeCheckResult.TypeCheckSuccess
--- End diff --
That's not right here. A string could be "2010-01-01" or "2010-01-01
00:00:00". cast this into a date or a timestamp would cause runtime error here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/6782#discussion_r33170413
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeFunctions.scala
---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions
+
+import org.apache.spark.sql.catalyst.analysis.TypeCheckResult
+import
org.apache.spark.sql.catalyst.expressions.codegen.{GeneratedExpressionCode,
CodeGenContext}
+import org.apache.spark.sql.catalyst.util.DateUtils
+import org.apache.spark.sql.types._
+import org.apache.spark.unsafe.types.UTF8String
+
+/**
+ * Adds a number of days to startdate: date_add('2008-12-31', 1) =
'2009-01-01'.
+ */
+case class DateAdd(startDate: Expression, days: Expression) extends
Expression {
+ override def children: Seq[Expression] = startDate :: days :: Nil
+
+ override def foldable: Boolean = startDate.foldable && days.foldable
+ override def nullable: Boolean = startDate.nullable || days.nullable
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+val supportedLeftType = Seq(StringType, DateType, TimestampType,
NullType)
+if (!supportedLeftType.contains(startDate.dataType)) {
+ TypeCheckResult.TypeCheckFailure(
+s"type of startdate expression in DateAdd should be
string/timestamp/date," +
+ s" not ${startDate.dataType}")
+} else if (days.dataType != IntegerType && days.dataType != NullType) {
+ TypeCheckResult.TypeCheckFailure(
+s"type of days expression in DateAdd should be int, not
${days.dataType}.")
+} else {
+ TypeCheckResult.TypeCheckSuccess
--- End diff --
@adrian-wang ExpectsInputTypes can help a lot, with `expectedChildTypes` as
`Seq(DateType, IntegerType)`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6782#issuecomment-114792119 [Test build #35655 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35655/console) for PR 6782 at commit [`95b3b58`](https://github.com/apache/spark/commit/95b3b582b770c1fd3e3df46176192151d55bda32). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class DateAdd(startDate: Expression, days: Expression) extends Expression ` * `case class DateSub(startDate: Expression, days: Expression) extends Expression ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6782#issuecomment-114792162 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6782#issuecomment-114752627 [Test build #35655 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35655/consoleFull) for PR 6782 at commit [`95b3b58`](https://github.com/apache/spark/commit/95b3b582b770c1fd3e3df46176192151d55bda32). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6782#issuecomment-114752522 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6782#issuecomment-114752510 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6782#issuecomment-114749202 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6782#issuecomment-114749148 [Test build #35646 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35646/console) for PR 6782 at commit [`0944f03`](https://github.com/apache/spark/commit/0944f038bff4c1f0d17d6727e75ca2c2167e509d). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class DateAdd(startDate: Expression, days: Expression) extends Expression ` * `case class DateSub(startDate: Expression, days: Expression) extends Expression ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6782#issuecomment-114742867 [Test build #35646 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35646/consoleFull) for PR 6782 at commit [`0944f03`](https://github.com/apache/spark/commit/0944f038bff4c1f0d17d6727e75ca2c2167e509d). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6782#issuecomment-114741824 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6782#issuecomment-114741789 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/6782#discussion_r33118435
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -946,6 +947,38 @@ object functions {
def cosh(columnName: String): Column = cosh(Column(columnName))
/**
+ * Adds a number of days to startDate, given values for startDate and
days
+ *
+ * @group datetime_funcs
+ * @since 1.5.0
+ */
+ def date_add(startDate: Column, days: Column): Column =
DateAdd(startDate.expr, days.expr)
+
+ /**
+ * Adds a number of days to startDate, given column names for startDate
and days
+ *
+ * @group datetime_funcs
+ * @since 1.5.0
+ */
+ def date_add(startDate: String, days: String): Column =
date_add(Column(startDate), Column(days))
--- End diff --
I think we probably need to think about what we need to provide for DF api.
Otherwise the similar will happens again. E.g. Should we also provide the
version like:
```
def date_add(startDate: Date, days: String): Column
def date_add(startDate: Timestamp, days: String): Column
...
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/6782#discussion_r33118210
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -946,6 +947,38 @@ object functions {
def cosh(columnName: String): Column = cosh(Column(columnName))
/**
+ * Adds a number of days to startDate, given values for startDate and
days
+ *
+ * @group datetime_funcs
+ * @since 1.5.0
+ */
+ def date_add(startDate: Column, days: Column): Column =
DateAdd(startDate.expr, days.expr)
+
+ /**
+ * Adds a number of days to startDate, given column names for startDate
and days
+ *
+ * @group datetime_funcs
+ * @since 1.5.0
+ */
+ def date_add(startDate: String, days: String): Column =
date_add(Column(startDate), Column(days))
--- End diff --
Not agreed, cc @rxin
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user adrian-wang commented on a diff in the pull request:
https://github.com/apache/spark/pull/6782#discussion_r33118085
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -946,6 +947,38 @@ object functions {
def cosh(columnName: String): Column = cosh(Column(columnName))
/**
+ * Adds a number of days to startDate, given values for startDate and
days
+ *
+ * @group datetime_funcs
+ * @since 1.5.0
+ */
+ def date_add(startDate: Column, days: Column): Column =
DateAdd(startDate.expr, days.expr)
+
+ /**
+ * Adds a number of days to startDate, given column names for startDate
and days
+ *
+ * @group datetime_funcs
+ * @since 1.5.0
+ */
+ def date_add(startDate: String, days: String): Column =
date_add(Column(startDate), Column(days))
--- End diff --
I think you should use `df.selectExpr("date_add(my_date, 7)")` in this
case. It is necessary to keep API small.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/6782#discussion_r33117976
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -946,6 +947,38 @@ object functions {
def cosh(columnName: String): Column = cosh(Column(columnName))
/**
+ * Adds a number of days to startDate, given values for startDate and
days
+ *
+ * @group datetime_funcs
+ * @since 1.5.0
+ */
+ def date_add(startDate: Column, days: Column): Column =
DateAdd(startDate.expr, days.expr)
+
+ /**
+ * Adds a number of days to startDate, given column names for startDate
and days
+ *
+ * @group datetime_funcs
+ * @since 1.5.0
+ */
+ def date_add(startDate: String, days: String): Column =
date_add(Column(startDate), Column(days))
--- End diff --
I means `df.select(date_add('my_date', 7))`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user adrian-wang commented on a diff in the pull request:
https://github.com/apache/spark/pull/6782#discussion_r33117423
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeFunctions.scala
---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions
+
+import org.apache.spark.sql.catalyst.analysis.TypeCheckResult
+import
org.apache.spark.sql.catalyst.expressions.codegen.{GeneratedExpressionCode,
CodeGenContext}
+import org.apache.spark.sql.catalyst.util.DateUtils
+import org.apache.spark.sql.types._
+import org.apache.spark.unsafe.types.UTF8String
+
+/**
+ * Adds a number of days to startdate: date_add('2008-12-31', 1) =
'2009-01-01'.
+ */
+case class DateAdd(startDate: Expression, days: Expression) extends
Expression {
+ override def children: Seq[Expression] = startDate :: days :: Nil
+
+ override def foldable: Boolean = startDate.foldable && days.foldable
+ override def nullable: Boolean = startDate.nullable || days.nullable
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+val supportedLeftType = Seq(StringType, DateType, TimestampType,
NullType)
+if (!supportedLeftType.contains(startDate.dataType)) {
+ TypeCheckResult.TypeCheckFailure(
+s"type of startdate expression in DateAdd should be
string/timestamp/date," +
+ s" not ${startDate.dataType}")
+} else if (days.dataType != IntegerType && days.dataType != NullType) {
+ TypeCheckResult.TypeCheckFailure(
+s"type of days expression in DateAdd should be int, not
${days.dataType}.")
+} else {
+ TypeCheckResult.TypeCheckSuccess
--- End diff --
To do cast in HiveTypeCoercion for about 30 different Datetime Functions?
That would be nasty to maintain.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user adrian-wang commented on a diff in the pull request:
https://github.com/apache/spark/pull/6782#discussion_r33117362
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -946,6 +947,38 @@ object functions {
def cosh(columnName: String): Column = cosh(Column(columnName))
/**
+ * Adds a number of days to startDate, given values for startDate and
days
+ *
+ * @group datetime_funcs
+ * @since 1.5.0
+ */
+ def date_add(startDate: Column, days: Column): Column =
DateAdd(startDate.expr, days.expr)
+
+ /**
+ * Adds a number of days to startDate, given column names for startDate
and days
+ *
+ * @group datetime_funcs
+ * @since 1.5.0
+ */
+ def date_add(startDate: String, days: String): Column =
date_add(Column(startDate), Column(days))
--- End diff --
This string is for columnName. We'd never expose we are using `Int` as
internal Date type, right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/6782#discussion_r33117317
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeFunctions.scala
---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions
+
+import org.apache.spark.sql.catalyst.analysis.TypeCheckResult
+import
org.apache.spark.sql.catalyst.expressions.codegen.{GeneratedExpressionCode,
CodeGenContext}
+import org.apache.spark.sql.catalyst.util.DateUtils
+import org.apache.spark.sql.types._
+import org.apache.spark.unsafe.types.UTF8String
+
+/**
+ * Adds a number of days to startdate: date_add('2008-12-31', 1) =
'2009-01-01'.
+ */
+case class DateAdd(startDate: Expression, days: Expression) extends
Expression {
+ override def children: Seq[Expression] = startDate :: days :: Nil
+
+ override def foldable: Boolean = startDate.foldable && days.foldable
+ override def nullable: Boolean = startDate.nullable || days.nullable
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+val supportedLeftType = Seq(StringType, DateType, TimestampType,
NullType)
+if (!supportedLeftType.contains(startDate.dataType)) {
+ TypeCheckResult.TypeCheckFailure(
+s"type of startdate expression in DateAdd should be
string/timestamp/date," +
+ s" not ${startDate.dataType}")
+} else if (days.dataType != IntegerType && days.dataType != NullType) {
+ TypeCheckResult.TypeCheckFailure(
+s"type of days expression in DateAdd should be int, not
${days.dataType}.")
+} else {
+ TypeCheckResult.TypeCheckSuccess
+}
+ }
+
+ override def dataType: DataType = StringType
--- End diff --
+1 for DateType.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/6782#discussion_r33117283
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeFunctions.scala
---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions
+
+import org.apache.spark.sql.catalyst.analysis.TypeCheckResult
+import
org.apache.spark.sql.catalyst.expressions.codegen.{GeneratedExpressionCode,
CodeGenContext}
+import org.apache.spark.sql.catalyst.util.DateUtils
+import org.apache.spark.sql.types._
+import org.apache.spark.unsafe.types.UTF8String
+
+/**
+ * Adds a number of days to startdate: date_add('2008-12-31', 1) =
'2009-01-01'.
+ */
+case class DateAdd(startDate: Expression, days: Expression) extends
Expression {
+ override def children: Seq[Expression] = startDate :: days :: Nil
+
+ override def foldable: Boolean = startDate.foldable && days.foldable
+ override def nullable: Boolean = startDate.nullable || days.nullable
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+val supportedLeftType = Seq(StringType, DateType, TimestampType,
NullType)
+if (!supportedLeftType.contains(startDate.dataType)) {
+ TypeCheckResult.TypeCheckFailure(
+s"type of startdate expression in DateAdd should be
string/timestamp/date," +
+ s" not ${startDate.dataType}")
+} else if (days.dataType != IntegerType && days.dataType != NullType) {
+ TypeCheckResult.TypeCheckFailure(
+s"type of days expression in DateAdd should be int, not
${days.dataType}.")
+} else {
+ TypeCheckResult.TypeCheckSuccess
--- End diff --
I agreed with @cloud-fan, that also helps to simplify the implementations.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/6782#discussion_r33117247
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -946,6 +947,38 @@ object functions {
def cosh(columnName: String): Column = cosh(Column(columnName))
/**
+ * Adds a number of days to startDate, given values for startDate and
days
+ *
+ * @group datetime_funcs
+ * @since 1.5.0
+ */
+ def date_add(startDate: Column, days: Column): Column =
DateAdd(startDate.expr, days.expr)
+
+ /**
+ * Adds a number of days to startDate, given column names for startDate
and days
+ *
+ * @group datetime_funcs
+ * @since 1.5.0
+ */
+ def date_add(startDate: String, days: String): Column =
date_add(Column(startDate), Column(days))
--- End diff --
Should we also support `days` as `Int`? Which is more useful than `days` as
`Column`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/6782#discussion_r33117014
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeFunctions.scala
---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions
+
+import org.apache.spark.sql.catalyst.analysis.TypeCheckResult
+import
org.apache.spark.sql.catalyst.expressions.codegen.{GeneratedExpressionCode,
CodeGenContext}
+import org.apache.spark.sql.catalyst.util.DateUtils
+import org.apache.spark.sql.types._
+import org.apache.spark.unsafe.types.UTF8String
+
+/**
+ * Adds a number of days to startdate: date_add('2008-12-31', 1) =
'2009-01-01'.
+ */
+case class DateAdd(startDate: Expression, days: Expression) extends
Expression {
+ override def children: Seq[Expression] = startDate :: days :: Nil
+
+ override def foldable: Boolean = startDate.foldable && days.foldable
+ override def nullable: Boolean = startDate.nullable || days.nullable
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+val supportedLeftType = Seq(StringType, DateType, TimestampType,
NullType)
+if (!supportedLeftType.contains(startDate.dataType)) {
+ TypeCheckResult.TypeCheckFailure(
+s"type of startdate expression in DateAdd should be
string/timestamp/date," +
+ s" not ${startDate.dataType}")
+} else if (days.dataType != IntegerType && days.dataType != NullType) {
--- End diff --
Could we also support integral types here for `days`?
For example, we always infer integer from Python and Json as LongType.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user chenghao-intel commented on the pull request: https://github.com/apache/spark/pull/6782#issuecomment-114706311 Can you remove the WIP from the title? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/6782#discussion_r33113212
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeFunctions.scala
---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions
+
+import org.apache.spark.sql.catalyst.analysis.TypeCheckResult
+import
org.apache.spark.sql.catalyst.expressions.codegen.{GeneratedExpressionCode,
CodeGenContext}
+import org.apache.spark.sql.catalyst.util.DateUtils
+import org.apache.spark.sql.types._
+import org.apache.spark.unsafe.types.UTF8String
+
+/**
+ * Adds a number of days to startdate: date_add('2008-12-31', 1) =
'2009-01-01'.
+ */
+case class DateAdd(startDate: Expression, days: Expression) extends
Expression {
+ override def children: Seq[Expression] = startDate :: days :: Nil
+
+ override def foldable: Boolean = startDate.foldable && days.foldable
+ override def nullable: Boolean = startDate.nullable || days.nullable
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+val supportedLeftType = Seq(StringType, DateType, TimestampType,
NullType)
+if (!supportedLeftType.contains(startDate.dataType)) {
+ TypeCheckResult.TypeCheckFailure(
+s"type of startdate expression in DateAdd should be
string/timestamp/date," +
+ s" not ${startDate.dataType}")
+} else if (days.dataType != IntegerType && days.dataType != NullType) {
+ TypeCheckResult.TypeCheckFailure(
+s"type of days expression in DateAdd should be int, not
${days.dataType}.")
+} else {
+ TypeCheckResult.TypeCheckSuccess
+}
+ }
+
+ override def dataType: DataType = StringType
+
+ override def toString: String = s"DateAdd($startDate, $days)"
+
+ override def eval(input: InternalRow): Any = {
+val start = startDate.eval(input)
+if (start == null) {
+ null
+} else {
+ val d = days.eval(input)
+ if (d == null) {
+null
+ } else {
+val offset = d.asInstanceOf[Int]
+val resultDays = startDate.dataType match {
+ case StringType =>
+DateUtils.millisToDays(DateUtils.stringToTime(
+ start.asInstanceOf[UTF8String].toString).getTime) + offset
+ case TimestampType =>
+DateUtils.millisToDays(DateUtils.toJavaTimestamp(
+ start.asInstanceOf[Long]).getTime) + offset
+ case DateType => start.asInstanceOf[Int] + offset
+}
+UTF8String.fromString(DateUtils.toString(resultDays))
+ }
+}
+ }
+
+ override def genCode(ctx: CodeGenContext, ev: GeneratedExpressionCode):
String = {
+val evalStart = startDate.gen(ctx)
+val evalDays = days.gen(ctx)
+val dateUtils = "org.apache.spark.sql.catalyst.util.DateUtils"
+val uTF8StringClass = "org.apache.spark.unsafe.types.UTF8String"
+val startToDay: String = startDate.dataType match {
+ case StringType =>
+s"""$dateUtils.millisToDays(
+
$dateUtils.stringToTime(${evalStart.primitive}.toString()).getTime())"""
+ case TimestampType =>
+
s"$dateUtils.millisToDays($dateUtils.toJavaTimestamp(${evalStart.primitive}).getTime())"
+ case DateType => evalStart.primitive
+ case _ => "" // for NullType
+}
+evalStart.code + evalDays.code + s"""
+ boolean ${ev.isNull} = ${evalStart.isNull} || ${evalDays.isNull};
+ ${ctx.javaType(dataType)} ${ev.primitive} =
${ctx.defaultValue(dataType)};
+ if (!${ev.isNull}) {
+${ev.primitive} = $uTF8StringClass.fromString(
+ $dateUtils.toString($startToDay + ${evalDays.primitive}));
+ }
+"""
+ }
+}
+
+/**
+ * Subtracts a number of days to startdate: date_sub('2008-12-31', 1) =
'2
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/6782#discussion_r33113199
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeFunctions.scala
---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions
+
+import org.apache.spark.sql.catalyst.analysis.TypeCheckResult
+import
org.apache.spark.sql.catalyst.expressions.codegen.{GeneratedExpressionCode,
CodeGenContext}
+import org.apache.spark.sql.catalyst.util.DateUtils
+import org.apache.spark.sql.types._
+import org.apache.spark.unsafe.types.UTF8String
+
+/**
+ * Adds a number of days to startdate: date_add('2008-12-31', 1) =
'2009-01-01'.
+ */
+case class DateAdd(startDate: Expression, days: Expression) extends
Expression {
+ override def children: Seq[Expression] = startDate :: days :: Nil
+
+ override def foldable: Boolean = startDate.foldable && days.foldable
+ override def nullable: Boolean = startDate.nullable || days.nullable
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+val supportedLeftType = Seq(StringType, DateType, TimestampType,
NullType)
+if (!supportedLeftType.contains(startDate.dataType)) {
+ TypeCheckResult.TypeCheckFailure(
+s"type of startdate expression in DateAdd should be
string/timestamp/date," +
+ s" not ${startDate.dataType}")
+} else if (days.dataType != IntegerType && days.dataType != NullType) {
+ TypeCheckResult.TypeCheckFailure(
+s"type of days expression in DateAdd should be int, not
${days.dataType}.")
+} else {
+ TypeCheckResult.TypeCheckSuccess
+}
+ }
+
+ override def dataType: DataType = StringType
+
+ override def toString: String = s"DateAdd($startDate, $days)"
--- End diff --
Let's remove this as the case class will handle that pretty well.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/6782#discussion_r32884986
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeFunctions.scala
---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions
+
+import org.apache.spark.sql.catalyst.analysis.TypeCheckResult
+import
org.apache.spark.sql.catalyst.expressions.codegen.{GeneratedExpressionCode,
CodeGenContext}
+import org.apache.spark.sql.catalyst.util.DateUtils
+import org.apache.spark.sql.types._
+import org.apache.spark.unsafe.types.UTF8String
+
+/**
+ * Adds a number of days to startdate: date_add('2008-12-31', 1) =
'2009-01-01'.
+ */
+case class DateAdd(startDate: Expression, days: Expression) extends
Expression {
+ override def children: Seq[Expression] = startDate :: days :: Nil
+
+ override def foldable: Boolean = startDate.foldable && days.foldable
+ override def nullable: Boolean = startDate.nullable || days.nullable
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+val supportedLeftType = Seq(StringType, DateType, TimestampType,
NullType)
+if (!supportedLeftType.contains(startDate.dataType)) {
+ TypeCheckResult.TypeCheckFailure(
+s"type of startdate expression in DateAdd should be
string/timestamp/date," +
+ s" not ${startDate.dataType}")
+} else if (days.dataType != IntegerType && days.dataType != NullType) {
+ TypeCheckResult.TypeCheckFailure(
+s"type of days expression in DateAdd should be int, not
${days.dataType}.")
+} else {
+ TypeCheckResult.TypeCheckSuccess
+}
+ }
+
+ override def dataType: DataType = StringType
+
+ override def toString: String = s"DateAdd($startDate, $days)"
+
+ override def eval(input: InternalRow): Any = {
+val start = startDate.eval(input)
+if (start == null) {
+ null
+} else {
+ val d = days.eval(input)
+ if (d == null) {
+null
+ } else {
+val offset = d.asInstanceOf[Int]
+val resultDays = startDate.dataType match {
+ case StringType =>
+DateUtils.millisToDays(DateUtils.stringToTime(
+ start.asInstanceOf[UTF8String].toString).getTime) + offset
--- End diff --
Ah I see, you handle both date and timestamp string here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user cloud-fan commented on the pull request: https://github.com/apache/spark/pull/6782#issuecomment-113775652 `DateAdd` and `DateSub` are very similar , can we abstract some common pattern to reduce code size? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/6782#discussion_r32884919
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeFunctions.scala
---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions
+
+import org.apache.spark.sql.catalyst.analysis.TypeCheckResult
+import
org.apache.spark.sql.catalyst.expressions.codegen.{GeneratedExpressionCode,
CodeGenContext}
+import org.apache.spark.sql.catalyst.util.DateUtils
+import org.apache.spark.sql.types._
+import org.apache.spark.unsafe.types.UTF8String
+
+/**
+ * Adds a number of days to startdate: date_add('2008-12-31', 1) =
'2009-01-01'.
+ */
+case class DateAdd(startDate: Expression, days: Expression) extends
Expression {
+ override def children: Seq[Expression] = startDate :: days :: Nil
+
+ override def foldable: Boolean = startDate.foldable && days.foldable
+ override def nullable: Boolean = startDate.nullable || days.nullable
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+val supportedLeftType = Seq(StringType, DateType, TimestampType,
NullType)
+if (!supportedLeftType.contains(startDate.dataType)) {
+ TypeCheckResult.TypeCheckFailure(
+s"type of startdate expression in DateAdd should be
string/timestamp/date," +
+ s" not ${startDate.dataType}")
+} else if (days.dataType != IntegerType && days.dataType != NullType) {
+ TypeCheckResult.TypeCheckFailure(
+s"type of days expression in DateAdd should be int, not
${days.dataType}.")
+} else {
+ TypeCheckResult.TypeCheckSuccess
+}
+ }
+
+ override def dataType: DataType = StringType
+
+ override def toString: String = s"DateAdd($startDate, $days)"
+
+ override def eval(input: InternalRow): Any = {
+val start = startDate.eval(input)
+if (start == null) {
+ null
+} else {
+ val d = days.eval(input)
+ if (d == null) {
+null
+ } else {
+val offset = d.asInstanceOf[Int]
+val resultDays = startDate.dataType match {
+ case StringType =>
+DateUtils.millisToDays(DateUtils.stringToTime(
+ start.asInstanceOf[UTF8String].toString).getTime) + offset
+ case TimestampType =>
+DateUtils.millisToDays(DateUtils.toJavaTimestamp(
+ start.asInstanceOf[Long]).getTime) + offset
--- End diff --
`DateUtils.millisToDays(t / 1L))` in `Cast`, which way is more
efficient?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/6782#discussion_r32884906
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeFunctions.scala
---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions
+
+import org.apache.spark.sql.catalyst.analysis.TypeCheckResult
+import
org.apache.spark.sql.catalyst.expressions.codegen.{GeneratedExpressionCode,
CodeGenContext}
+import org.apache.spark.sql.catalyst.util.DateUtils
+import org.apache.spark.sql.types._
+import org.apache.spark.unsafe.types.UTF8String
+
+/**
+ * Adds a number of days to startdate: date_add('2008-12-31', 1) =
'2009-01-01'.
+ */
+case class DateAdd(startDate: Expression, days: Expression) extends
Expression {
+ override def children: Seq[Expression] = startDate :: days :: Nil
+
+ override def foldable: Boolean = startDate.foldable && days.foldable
+ override def nullable: Boolean = startDate.nullable || days.nullable
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+val supportedLeftType = Seq(StringType, DateType, TimestampType,
NullType)
+if (!supportedLeftType.contains(startDate.dataType)) {
+ TypeCheckResult.TypeCheckFailure(
+s"type of startdate expression in DateAdd should be
string/timestamp/date," +
+ s" not ${startDate.dataType}")
+} else if (days.dataType != IntegerType && days.dataType != NullType) {
+ TypeCheckResult.TypeCheckFailure(
+s"type of days expression in DateAdd should be int, not
${days.dataType}.")
+} else {
+ TypeCheckResult.TypeCheckSuccess
+}
+ }
+
+ override def dataType: DataType = StringType
+
+ override def toString: String = s"DateAdd($startDate, $days)"
+
+ override def eval(input: InternalRow): Any = {
+val start = startDate.eval(input)
+if (start == null) {
+ null
+} else {
+ val d = days.eval(input)
+ if (d == null) {
+null
+ } else {
+val offset = d.asInstanceOf[Int]
+val resultDays = startDate.dataType match {
+ case StringType =>
+DateUtils.millisToDays(DateUtils.stringToTime(
+ start.asInstanceOf[UTF8String].toString).getTime) + offset
--- End diff --
In `Cast` we using `DateUtils.fromJavaDate(Date.valueOf(s.toString))` to
cast string to date. Which way is more efficient?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/6782#discussion_r32867754
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeFunctions.scala
---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions
+
+import org.apache.spark.sql.catalyst.analysis.TypeCheckResult
+import
org.apache.spark.sql.catalyst.expressions.codegen.{GeneratedExpressionCode,
CodeGenContext}
+import org.apache.spark.sql.catalyst.util.DateUtils
+import org.apache.spark.sql.types._
+import org.apache.spark.unsafe.types.UTF8String
+
+/**
+ * Adds a number of days to startdate: date_add('2008-12-31', 1) =
'2009-01-01'.
+ */
+case class DateAdd(startDate: Expression, days: Expression) extends
Expression {
+ override def children: Seq[Expression] = startDate :: days :: Nil
+
+ override def foldable: Boolean = startDate.foldable && days.foldable
+ override def nullable: Boolean = startDate.nullable || days.nullable
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+val supportedLeftType = Seq(StringType, DateType, TimestampType,
NullType)
+if (!supportedLeftType.contains(startDate.dataType)) {
+ TypeCheckResult.TypeCheckFailure(
+s"type of startdate expression in DateAdd should be
string/timestamp/date," +
+ s" not ${startDate.dataType}")
+} else if (days.dataType != IntegerType && days.dataType != NullType) {
+ TypeCheckResult.TypeCheckFailure(
+s"type of days expression in DateAdd should be int, not
${days.dataType}.")
+} else {
+ TypeCheckResult.TypeCheckSuccess
+}
+ }
+
+ override def dataType: DataType = StringType
--- End diff --
In general though we cast back to a string whenever you need to. From an
efficiency stand point it seems much better to keep it a date.
/cc @rxin
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user adrian-wang commented on a diff in the pull request:
https://github.com/apache/spark/pull/6782#discussion_r32807518
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeFunctions.scala
---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions
+
+import org.apache.spark.sql.catalyst.analysis.TypeCheckResult
+import
org.apache.spark.sql.catalyst.expressions.codegen.{GeneratedExpressionCode,
CodeGenContext}
+import org.apache.spark.sql.catalyst.util.DateUtils
+import org.apache.spark.sql.types._
+import org.apache.spark.unsafe.types.UTF8String
+
+/**
+ * Adds a number of days to startdate: date_add('2008-12-31', 1) =
'2009-01-01'.
+ */
+case class DateAdd(startDate: Expression, days: Expression) extends
Expression {
+ override def children: Seq[Expression] = startDate :: days :: Nil
+
+ override def foldable: Boolean = startDate.foldable && days.foldable
+ override def nullable: Boolean = startDate.nullable || days.nullable
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+val supportedLeftType = Seq(StringType, DateType, TimestampType,
NullType)
+if (!supportedLeftType.contains(startDate.dataType)) {
+ TypeCheckResult.TypeCheckFailure(
+s"type of startdate expression in DateAdd should be
string/timestamp/date," +
+ s" not ${startDate.dataType}")
+} else if (days.dataType != IntegerType && days.dataType != NullType) {
+ TypeCheckResult.TypeCheckFailure(
+s"type of days expression in DateAdd should be int, not
${days.dataType}.")
+} else {
+ TypeCheckResult.TypeCheckSuccess
--- End diff --
After #6759 gets in, I probably will refract DateUtils to offer some more
direct interfaces, but let's skip it now to avoid huge conflicts.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
