[GitHub] spark pull request #18080: [Spark-20771][SQL] Make weekofyear more intuitive
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/18080 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18080: [Spark-20771][SQL] Make weekofyear more intuitive
GitHub user setjet opened a pull request: https://github.com/apache/spark/pull/18080 [Spark-20771][SQL] Make weekofyear more intuitive ## What changes were proposed in this pull request? The current implementation of weekofyear implements ISO8601, which results in the following unintuitive behaviour: weekofyear("2017-01-01") returns 52 In MySQL, this would return 1 (https://dev.mysql.com/doc/refman/5.5/en/date-and-time-functions.html#function_weekofyear), although it could return 52 if specified specifically (https://dev.mysql.com/doc/refman/5.5/en/date-and-time-functions.html#function_week). I therefore think instead of only changing the behavior as specified in the JIRA, it would be better to support both. Hence I've added an additional function. ## How was this patch tested? Added some unit tests You can merge this pull request into a Git repository by running: $ git pull https://github.com/setjet/spark SPARK-20771 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/18080.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #18080 commit 7235f4a731f83a3a81fd65846179efaf38354bfa Author: setjet Date: 2017-05-24T00:20:30Z added additional weekofyear function commit 057ede5b68cc7980987ae181156f376f84c41809 Author: setjet Date: 2017-05-24T00:22:54Z updated desc --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18080: [Spark-20771][SQL] Make weekofyear more intuitive
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/18080#discussion_r118803850 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala --- @@ -402,23 +402,40 @@ case class DayOfMonth(child: Expression) extends UnaryExpression with ImplicitCa } } +// scalastyle:off line.size.limit @ExpressionDescription( - usage = "_FUNC_(date) - Returns the week of the year of the given date.", + usage = "_FUNC_(date[, format]) - Returns the week of the year of the given date. Defaults to ISO 8601 standard, but can be gregorian specific", extended = """ Examples: > SELECT _FUNC_('2008-02-20'); 8 + > SELECT _FUNC_('2017-01-01', 'gregorian'); + 1 + > SELECT _FUNC_('2017-01-01', 'iso'); + 52 + > SELECT _FUNC_('2017-01-01'); + 52 """) -case class WeekOfYear(child: Expression) extends UnaryExpression with ImplicitCastInputTypes { +// scalastyle:on line.size.limit +case class WeekOfYear(child: Expression, format: Expression) extends + UnaryExpression with ImplicitCastInputTypes { + + def this(child: Expression) = { +this(child, Literal("iso")) + } override def inputTypes: Seq[AbstractDataType] = Seq(DateType) override def dataType: DataType = IntegerType + @transient private lazy val minimalDays = { +if ("gregorian".equalsIgnoreCase(format.toString)) 1 else 4 --- End diff -- How many formats the other DB/systems allow? Could you do a search? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18080: [Spark-20771][SQL] Make weekofyear more intuitive
Github user setjet commented on a diff in the pull request: https://github.com/apache/spark/pull/18080#discussion_r118820456 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala --- @@ -402,23 +402,40 @@ case class DayOfMonth(child: Expression) extends UnaryExpression with ImplicitCa } } +// scalastyle:off line.size.limit @ExpressionDescription( - usage = "_FUNC_(date) - Returns the week of the year of the given date.", + usage = "_FUNC_(date[, format]) - Returns the week of the year of the given date. Defaults to ISO 8601 standard, but can be gregorian specific", extended = """ Examples: > SELECT _FUNC_('2008-02-20'); 8 + > SELECT _FUNC_('2017-01-01', 'gregorian'); + 1 + > SELECT _FUNC_('2017-01-01', 'iso'); + 52 + > SELECT _FUNC_('2017-01-01'); + 52 """) -case class WeekOfYear(child: Expression) extends UnaryExpression with ImplicitCastInputTypes { +// scalastyle:on line.size.limit +case class WeekOfYear(child: Expression, format: Expression) extends + UnaryExpression with ImplicitCastInputTypes { + + def this(child: Expression) = { +this(child, Literal("iso")) + } override def inputTypes: Seq[AbstractDataType] = Seq(DateType) override def dataType: DataType = IntegerType + @transient private lazy val minimalDays = { +if ("gregorian".equalsIgnoreCase(format.toString)) 1 else 4 --- End diff -- I did a bit of research, and there seem to be no other formats. However, some systems (such as MySQL and Java), allow the first day of the week to be defined as well. Some countries in the middle east have a week on Friday/Saturday, or even Thursday/Friday. I will update the PR to allow users to override the first day of the week, as well as specify how the first week is defined (1 iso standard: week with more than half of the days, i.e. Thursday in a Monday-Sunday week. 2 gregorian: week with first day of the new year) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18080: [Spark-20771][SQL] Make weekofyear more intuitive
Github user setjet commented on a diff in the pull request: https://github.com/apache/spark/pull/18080#discussion_r118820467 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala --- @@ -402,23 +402,40 @@ case class DayOfMonth(child: Expression) extends UnaryExpression with ImplicitCa } } +// scalastyle:off line.size.limit @ExpressionDescription( - usage = "_FUNC_(date) - Returns the week of the year of the given date.", + usage = "_FUNC_(date[, format]) - Returns the week of the year of the given date. Defaults to ISO 8601 standard, but can be gregorian specific", extended = """ Examples: > SELECT _FUNC_('2008-02-20'); 8 + > SELECT _FUNC_('2017-01-01', 'gregorian'); + 1 + > SELECT _FUNC_('2017-01-01', 'iso'); + 52 + > SELECT _FUNC_('2017-01-01'); + 52 """) -case class WeekOfYear(child: Expression) extends UnaryExpression with ImplicitCastInputTypes { +// scalastyle:on line.size.limit +case class WeekOfYear(child: Expression, format: Expression) extends + UnaryExpression with ImplicitCastInputTypes { + + def this(child: Expression) = { +this(child, Literal("iso")) + } override def inputTypes: Seq[AbstractDataType] = Seq(DateType) override def dataType: DataType = IntegerType + @transient private lazy val minimalDays = { +if ("gregorian".equalsIgnoreCase(format.toString)) 1 else 4 --- End diff -- It will still default to ISO stanards with Monday-Sunday week of course, but now users can override it in any way they would like --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org