[GitHub] spark pull request #14036: [SPARK-16323] [SQL] Add IntegerDivide to avoid un...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/14036 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14036: [SPARK-16323] [SQL] Add IntegerDivide to avoid un...
Github user lianhuiwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/14036#discussion_r70746279
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala
---
@@ -234,6 +234,7 @@ object FunctionRegistry {
expression[Subtract]("-"),
expression[Multiply]("*"),
expression[Divide]("/"),
+expression[IntegerDivide]("div"),
--- End diff --
'select 4 div 2' is the right code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14036: [SPARK-16323] [SQL] Add IntegerDivide to avoid un...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/14036#discussion_r70665534
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala
---
@@ -207,20 +207,12 @@ case class Multiply(left: Expression, right:
Expression)
protected override def nullSafeEval(input1: Any, input2: Any): Any =
numeric.times(input1, input2)
}
-@ExpressionDescription(
- usage = "a _FUNC_ b - Divides a by b.",
- extended = "> SELECT 3 _FUNC_ 2;\n 1.5")
-case class Divide(left: Expression, right: Expression)
-extends BinaryArithmetic with NullIntolerant {
-
- override def inputType: AbstractDataType = TypeCollection(DoubleType,
DecimalType)
-
- override def symbol: String = "/"
- override def decimalMethod: String = "$div"
+abstract class DivideBase extends BinaryArithmetic with NullIntolerant {
override def nullable: Boolean = true
private lazy val div: (Any, Any) => Any = dataType match {
case ft: FractionalType =>
ft.fractional.asInstanceOf[Fractional[Any]].div
+case i: IntegralType => i.integral.asInstanceOf[Integral[Any]].quot
--- End diff --
The `DivideBase` implement codegen for all types, so I think it's fine for
it to implement `eval` for all types.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14036: [SPARK-16323] [SQL] Add IntegerDivide to avoid un...
Github user lianhuiwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/14036#discussion_r70660136
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala
---
@@ -207,20 +207,12 @@ case class Multiply(left: Expression, right:
Expression)
protected override def nullSafeEval(input1: Any, input2: Any): Any =
numeric.times(input1, input2)
}
-@ExpressionDescription(
- usage = "a _FUNC_ b - Divides a by b.",
- extended = "> SELECT 3 _FUNC_ 2;\n 1.5")
-case class Divide(left: Expression, right: Expression)
-extends BinaryArithmetic with NullIntolerant {
-
- override def inputType: AbstractDataType = TypeCollection(DoubleType,
DecimalType)
-
- override def symbol: String = "/"
- override def decimalMethod: String = "$div"
+abstract class DivideBase extends BinaryArithmetic with NullIntolerant {
override def nullable: Boolean = true
private lazy val div: (Any, Any) => Any = dataType match {
case ft: FractionalType =>
ft.fractional.asInstanceOf[Fractional[Any]].div
+case i: IntegralType => i.integral.asInstanceOf[Integral[Any]].quot
--- End diff --
@cloud-fan how about make this line in IntegralDivide that can be more
readable?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14036: [SPARK-16323] [SQL] Add IntegerDivide to avoid un...
Github user lianhuiwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/14036#discussion_r70660445
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala
---
@@ -207,20 +207,12 @@ case class Multiply(left: Expression, right:
Expression)
protected override def nullSafeEval(input1: Any, input2: Any): Any =
numeric.times(input1, input2)
}
-@ExpressionDescription(
- usage = "a _FUNC_ b - Divides a by b.",
- extended = "> SELECT 3 _FUNC_ 2;\n 1.5")
-case class Divide(left: Expression, right: Expression)
-extends BinaryArithmetic with NullIntolerant {
-
- override def inputType: AbstractDataType = TypeCollection(DoubleType,
DecimalType)
-
- override def symbol: String = "/"
--- End diff --
we need to keep symbol because it is very useful for BinaryOperator'
sqlOperator() and sql()
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14036: [SPARK-16323] [SQL] Add IntegerDivide to avoid un...
Github user lianhuiwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/14036#discussion_r70659855
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
---
@@ -957,7 +957,7 @@ class AstBuilder extends SqlBaseBaseVisitor[AnyRef]
with Logging {
case SqlBaseParser.PERCENT =>
Remainder(left, right)
case SqlBaseParser.DIV =>
-Cast(Divide(left, right), LongType)
+IntegralDivide(left, right)
--- End diff --
That's ok because I find SparkSQL has SqlBaseParser.SLASH for '/' .
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14036: [SPARK-16323] [SQL] Add IntegerDivide to avoid un...
Github user lianhuiwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/14036#discussion_r70656424
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
---
@@ -957,7 +957,7 @@ class AstBuilder extends SqlBaseBaseVisitor[AnyRef]
with Logging {
case SqlBaseParser.PERCENT =>
Remainder(left, right)
case SqlBaseParser.DIV =>
-Cast(Divide(left, right), LongType)
+IntegralDivide(left, right)
--- End diff --
I think we needs to add SqlBaseParser.DIVIDE for '/'.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14036: [SPARK-16323] [SQL] Add IntegerDivide to avoid un...
Github user techaddict commented on a diff in the pull request:
https://github.com/apache/spark/pull/14036#discussion_r70639783
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala
---
@@ -234,6 +234,7 @@ object FunctionRegistry {
expression[Subtract]("-"),
expression[Multiply]("*"),
expression[Divide]("/"),
+expression[IntegerDivide]("div"),
--- End diff --
@lianhuiwang doing ```div(4,2)``` gives
```
hive> div(4, 2);
NoViableAltException(14@[])
at
org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1099)
at
org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:204)
at
org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:440)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:319)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1249)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1295)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1178)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1166)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:236)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:187)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
at
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:782)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:721)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:648)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
FAILED: ParseException line 1:0 cannot recognize input near 'div' '(' '4'
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14036: [SPARK-16323] [SQL] Add IntegerDivide to avoid un...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/14036#discussion_r70631504
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala
---
@@ -285,6 +278,26 @@ case class Divide(left: Expression, right: Expression)
}
@ExpressionDescription(
+ usage = "a _FUNC_ b - Fraction Division a by b.",
+ extended = "> SELECT 3 _FUNC_ 2;\n 1.5")
+case class Divide(left: Expression, right: Expression) extends DivideBase {
+
+ override def inputType: AbstractDataType = TypeCollection(DoubleType,
DecimalType)
+
+ override def symbol: String = "/"
+}
+
+@ExpressionDescription(
+ usage = "a _FUNC_ b - Divides a by b.",
--- End diff --
how about Divides a by b of integral type?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14036: [SPARK-16323] [SQL] Add IntegerDivide to avoid un...
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/14036#discussion_r70631462 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala --- @@ -285,6 +278,26 @@ case class Divide(left: Expression, right: Expression) } @ExpressionDescription( + usage = "a _FUNC_ b - Fraction Division a by b.", --- End diff -- how about `Divides a by b of fraction type`? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14036: [SPARK-16323] [SQL] Add IntegerDivide to avoid un...
Github user lianhuiwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/14036#discussion_r70628892
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala
---
@@ -234,6 +234,7 @@ object FunctionRegistry {
expression[Subtract]("-"),
expression[Multiply]("*"),
expression[Divide]("/"),
+expression[IntegerDivide]("div"),
--- End diff --
@cloud-fan yes, hive support div and / .
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14036: [SPARK-16323] [SQL] Add IntegerDivide to avoid un...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/14036#discussion_r70580188
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala
---
@@ -207,20 +207,12 @@ case class Multiply(left: Expression, right:
Expression)
protected override def nullSafeEval(input1: Any, input2: Any): Any =
numeric.times(input1, input2)
}
-@ExpressionDescription(
- usage = "a _FUNC_ b - Divides a by b.",
- extended = "> SELECT 3 _FUNC_ 2;\n 1.5")
-case class Divide(left: Expression, right: Expression)
-extends BinaryArithmetic with NullIntolerant {
-
- override def inputType: AbstractDataType = TypeCollection(DoubleType,
DecimalType)
-
- override def symbol: String = "/"
- override def decimalMethod: String = "$div"
+abstract class DivisionArithmetic extends BinaryArithmetic with
NullIntolerant {
--- End diff --
how about `DivideBase`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14036: [SPARK-16323] [SQL] Add IntegerDivide to avoid un...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/14036#discussion_r70580079
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala
---
@@ -285,6 +278,28 @@ case class Divide(left: Expression, right: Expression)
}
@ExpressionDescription(
+ usage = "a _FUNC_ b - Fraction Division a by b.",
+ extended = "> SELECT 3 _FUNC_ 2;\n 1.5")
+case class Divide(left: Expression, right: Expression)
+extends DivisionArithmetic {
+
+ override def inputType: AbstractDataType = TypeCollection(DoubleType,
DecimalType)
+
+ override def symbol: String = "/"
+}
+
+@ExpressionDescription(
+ usage = "a _FUNC_ b - Divides a by b.",
+ extended = "> SELECT 3 _FUNC_ 2;\n 1")
+case class IntegerDivide(left: Expression, right: Expression)
--- End diff --
`IntegralDivide`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14036: [SPARK-16323] [SQL] Add IntegerDivide to avoid un...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/14036#discussion_r70564289
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala
---
@@ -249,11 +241,12 @@ case class Divide(left: Expression, right: Expression)
s"${eval2.value} == 0"
}
val javaType = ctx.javaType(dataType)
-val divide = if (dataType.isInstanceOf[DecimalType]) {
+val divide = if (dataType.isInstanceOf[DecimalType] ||
dataType.isInstanceOf[DoubleType]) {
s"${eval1.value}.$decimalMethod(${eval2.value})"
} else {
- s"($javaType)(${eval1.value} $symbol ${eval2.value})"
+ s"($javaType)(${eval1.value} $decimalMethod ${eval2.value})"
--- End diff --
just follow the previous behaviour, inline "$div"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14036: [SPARK-16323] [SQL] Add IntegerDivide to avoid un...
Github user techaddict commented on a diff in the pull request:
https://github.com/apache/spark/pull/14036#discussion_r70563932
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala
---
@@ -249,11 +241,12 @@ case class Divide(left: Expression, right: Expression)
s"${eval2.value} == 0"
}
val javaType = ctx.javaType(dataType)
-val divide = if (dataType.isInstanceOf[DecimalType]) {
+val divide = if (dataType.isInstanceOf[DecimalType] ||
dataType.isInstanceOf[DoubleType]) {
s"${eval1.value}.$decimalMethod(${eval2.value})"
} else {
- s"($javaType)(${eval1.value} $symbol ${eval2.value})"
+ s"($javaType)(${eval1.value} $decimalMethod ${eval2.value})"
--- End diff --
but what about the decimalMethod used in line 245 ? if we inline the
operator `/` there too it gives
```Binary numeric promotion not possible on types
"org.apache.spark.sql.types.Decimal" and "org.apache.spark.sql.types.Decimal"```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14036: [SPARK-16323] [SQL] Add IntegerDivide to avoid un...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/14036#discussion_r70562757
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala
---
@@ -249,11 +241,12 @@ case class Divide(left: Expression, right: Expression)
s"${eval2.value} == 0"
}
val javaType = ctx.javaType(dataType)
-val divide = if (dataType.isInstanceOf[DecimalType]) {
+val divide = if (dataType.isInstanceOf[DecimalType] ||
dataType.isInstanceOf[DoubleType]) {
s"${eval1.value}.$decimalMethod(${eval2.value})"
} else {
- s"($javaType)(${eval1.value} $symbol ${eval2.value})"
+ s"($javaType)(${eval1.value} $decimalMethod ${eval2.value})"
--- End diff --
ok I see the problem. Since `symbol` is also used in the string format of
arithmetic expression, `IntegerDivide` has to override it.
I think we can:
1. inline the operator in generated code, e.g. `($javaType)(${eval1.value}
/ ${eval2.value})` instead of `($javaType)(${eval1.value} $symbol
${eval2.value})`
2. don't override `decimalMethod` as it's useless in the 2 division
expressions.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14036: [SPARK-16323] [SQL] Add IntegerDivide to avoid un...
Github user techaddict commented on a diff in the pull request: https://github.com/apache/spark/pull/14036#discussion_r70471772 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala --- @@ -237,6 +229,9 @@ case class Divide(left: Expression, right: Expression) } } + // Used by doGenCode + protected def divide(eval1: ExprCode, eval2: ExprCode, javaType: String): String --- End diff -- @cloud-fan yes but getting ```A method named "$div" is not declared in any enclosing class nor any supertype, nor through a static import``` in the updated pr for `Code generation of (2.0 / 1.0)` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14036: [SPARK-16323] [SQL] Add IntegerDivide to avoid un...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/14036#discussion_r70459222
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala
---
@@ -237,6 +229,9 @@ case class Divide(left: Expression, right: Expression)
}
}
+ // Used by doGenCode
+ protected def divide(eval1: ExprCode, eval2: ExprCode, javaType:
String): String
--- End diff --
you can see from [this
test](https://github.com/apache/spark/pull/14036/files#diff-f3fc3b985a8877ee493f1a12ddfb98d6L143)
that the previous `Divide` expression can support all integral types and
fraction types.
BTW looks there is no special handling for integral type in your PR, for
fraction type:
```
if (dataType.isInstanceOf[DecimalType]) {
s"${eval1.value}.$decimalMethod(${eval2.value})"
} else {
s"($javaType)(${eval1.value} $symbol ${eval2.value})"
}
```
while the `symbol` is `/`, so we generate `($javaType)(${eval1.value} /
${eval2.value})` for double and float.
for integral type:
```
($javaType)(${eval1.value} $decimalMethod (${eval2.value}))
```
while the `decimalMethod` is `/`, so it's still generates
`($javaType)(${eval1.value} / ${eval2.value})`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14036: [SPARK-16323] [SQL] Add IntegerDivide to avoid un...
Github user techaddict commented on a diff in the pull request: https://github.com/apache/spark/pull/14036#discussion_r70437720 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala --- @@ -237,6 +229,9 @@ case class Divide(left: Expression, right: Expression) } } + // Used by doGenCode + protected def divide(eval1: ExprCode, eval2: ExprCode, javaType: String): String --- End diff -- I did it on purpose. we can't call `$div` on `byte's` and plus if I try to call `value = value1 / value2;` for decimals, I get ```Binary numeric promotion not possible on types "org.apache.spark.sql.types.Decimal" and "org.apache.spark.sql.types.Decimal"```. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14036: [SPARK-16323] [SQL] Add IntegerDivide to avoid un...
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/14036#discussion_r70381897 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala --- @@ -237,6 +229,9 @@ case class Divide(left: Expression, right: Expression) } } + // Used by doGenCode + protected def divide(eval1: ExprCode, eval2: ExprCode, javaType: String): String --- End diff -- I don't think we need this abstraction. [this one](https://github.com/apache/spark/pull/14036/files#diff-1516b10738479bbe190fb4e239258473L252) already covers both fraction and integral --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14036: [SPARK-16323] [SQL] Add IntegerDivide to avoid un...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/14036#discussion_r69848182
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala
---
@@ -277,14 +268,52 @@ case class Divide(left: Expression, right: Expression)
if (${eval1.isNull}) {
${ev.isNull} = true;
} else {
-${ev.value} = $divide;
+${ev.value} = $division;
}
}""")
}
}
}
@ExpressionDescription(
+ usage = "a _FUNC_ b - Divides a by b.",
--- End diff --
we should mention this is a fraction division, i.e. the parameter must be
fraction type and the result is also fraction.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14036: [SPARK-16323] [SQL] Add IntegerDivide to avoid un...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/14036#discussion_r69848124
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala
---
@@ -249,11 +244,7 @@ case class Divide(left: Expression, right: Expression)
s"${eval2.value} == 0"
}
val javaType = ctx.javaType(dataType)
-val divide = if (dataType.isInstanceOf[DecimalType]) {
--- End diff --
Does it already cover both fraction and integral division?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14036: [SPARK-16323] [SQL] Add IntegerDivide to avoid un...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14036#discussion_r69619115
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Column.scala ---
@@ -732,6 +732,21 @@ class Column(protected[sql] val expr: Expression)
extends Logging {
def / (other: Any): Column = withExpr { Divide(expr, lit(other).expr) }
/**
+ * Integer Division this expression by another expression.
+ * {{{
+ * // Scala: The following divides a person's height by their weight.
+ * people.select( people("height") div people("weight") )
+ *
+ * // Java:
+ * people.select( people("height").div(people("weight")) );
+ * }}}
+ *
+ * @group expr_ops
+ * @since 2.1.0
+ */
+ def div (other: Any): Column = withExpr { IntegerDivide(expr,
lit(other).expr) }
--- End diff --
why can't we just use the normal / ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14036: [SPARK-16323] [SQL] Add IntegerDivide to avoid un...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/14036#discussion_r69511190
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala
---
@@ -237,54 +229,100 @@ case class Divide(left: Expression, right:
Expression)
}
}
+ // Used by doGenCode
+ def divide(eval1: ExprCode, eval2: ExprCode, javaType: String): String
+ def isZero(eval2: ExprCode): String
+
/**
* Special case handling due to division by 0 => null.
*/
override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
val eval1 = left.genCode(ctx)
val eval2 = right.genCode(ctx)
-val isZero = if (dataType.isInstanceOf[DecimalType]) {
--- End diff --
why not just keep it? Then the 2 implementations can share the same codegen.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14036: [SPARK-16323] [SQL] Add IntegerDivide to avoid un...
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/14036#discussion_r69511063 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala --- @@ -237,54 +229,100 @@ case class Divide(left: Expression, right: Expression) } } + // Used by doGenCode + def divide(eval1: ExprCode, eval2: ExprCode, javaType: String): String --- End diff -- mark as `protected`? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14036: [SPARK-16323] [SQL] Add IntegerDivide to avoid un...
Github user techaddict commented on a diff in the pull request: https://github.com/apache/spark/pull/14036#discussion_r69407742 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala --- @@ -285,6 +284,75 @@ case class Divide(left: Expression, right: Expression) } @ExpressionDescription( + usage = "a _FUNC_ b - Divides a by b.", + extended = "> SELECT 3 _FUNC_ 2;\n 1") +case class IntegerDivide(left: Expression, right: Expression) --- End diff -- Done --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14036: [SPARK-16323] [SQL] Add IntegerDivide to avoid un...
Github user techaddict commented on a diff in the pull request: https://github.com/apache/spark/pull/14036#discussion_r69392406 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala --- @@ -285,6 +284,75 @@ case class Divide(left: Expression, right: Expression) } @ExpressionDescription( + usage = "a _FUNC_ b - Divides a by b.", + extended = "> SELECT 3 _FUNC_ 2;\n 1") +case class IntegerDivide(left: Expression, right: Expression) --- End diff -- Let me try doing that ð --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14036: [SPARK-16323] [SQL] Add IntegerDivide to avoid un...
Github user techaddict commented on a diff in the pull request:
https://github.com/apache/spark/pull/14036#discussion_r69392402
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala
---
@@ -234,6 +234,7 @@ object FunctionRegistry {
expression[Subtract]("-"),
expression[Multiply]("*"),
expression[Divide]("/"),
+expression[IntegerDivide]("div"),
--- End diff --
I don't think so.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14036: [SPARK-16323] [SQL] Add IntegerDivide to avoid un...
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/14036#discussion_r69392173 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala --- @@ -285,6 +284,75 @@ case class Divide(left: Expression, right: Expression) } @ExpressionDescription( + usage = "a _FUNC_ b - Divides a by b.", + extended = "> SELECT 3 _FUNC_ 2;\n 1") +case class IntegerDivide(left: Expression, right: Expression) --- End diff -- There are a lot of duplicated code between this class and `Divide`, is there any possibility we can abstract them? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14036: [SPARK-16323] [SQL] Add IntegerDivide to avoid un...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/14036#discussion_r69392061
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala
---
@@ -234,6 +234,7 @@ object FunctionRegistry {
expression[Subtract]("-"),
expression[Multiply]("*"),
expression[Divide]("/"),
+expression[IntegerDivide]("div"),
--- End diff --
does hive support this syntax? i.e. `div(4, 2)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14036: [SPARK-16323] [SQL] Add IntegerDivide to avoid un...
GitHub user techaddict opened a pull request:
https://github.com/apache/spark/pull/14036
[SPARK-16323] [SQL] Add IntegerDivide to avoid unnecessary cast
## What changes were proposed in this pull request?
Add IntegerDivide to avoid unnecessary cast
Before:
```
scala> spark.sql("select 6 div 3").explain(true)
...
== Analyzed Logical Plan ==
CAST((6 / 3) AS BIGINT): bigint
Project [cast((cast(6 as double) / cast(3 as double)) as bigint) AS
CAST((6 / 3) AS BIGINT)#5L]
+- OneRowRelation$
...
```
After:
```
scala> spark.sql("select 6 div 3").explain(true)
...
== Analyzed Logical Plan ==
(6 / 3): int
Project [(6 / 3) AS (6 / 3)#11]
+- OneRowRelation$
...
```
## How was this patch tested?
Existing Tests and added new ones
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/techaddict/spark SPARK-16323
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/14036.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #14036
commit 067b788b05846e659615feef8613f8965573f150
Author: Sandeep Singh
Date: 2016-07-03T09:00:11Z
[SPARK-16323] [SQL] Add IntegerDivide to avoid unnecessary cast
Before:
```
scala> spark.sql("select 6 div 3").explain(true)
...
== Analyzed Logical Plan ==
CAST((6 / 3) AS BIGINT): bigint
Project [cast((cast(6 as double) / cast(3 as double)) as bigint) AS
CAST((6 / 3) AS BIGINT)#5L]
+- OneRowRelation$
...
```
After:
```
scala> spark.sql("select 6 div 3").explain(true)
...
== Analyzed Logical Plan ==
(6 / 3): int
Project [(6 / 3) AS (6 / 3)#11]
+- OneRowRelation$
...
```
commit e4e42c35b0236dff6aedf6468a7d94f80bc6023b
Author: Sandeep Singh
Date: 2016-07-03T09:00:59Z
Merge branch 'master' into SPARK-16323
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
