[GitHub] [spark] LuciferYang commented on a diff in pull request #38635: [SPARK-41118][SQL] `to_number`/`try_to_number` should return `null` when format is `null`
LuciferYang commented on code in PR #38635:
URL: https://github.com/apache/spark/pull/38635#discussion_r1023569619
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/numberFormatExpressions.scala:
##
@@ -26,6 +26,62 @@ import org.apache.spark.sql.catalyst.util.ToNumberParser
import org.apache.spark.sql.types.{AbstractDataType, DataType, Decimal,
DecimalType, StringType}
import org.apache.spark.unsafe.types.UTF8String
+abstract class ToNumberBase(left: Expression, right: Expression, errorOnFail:
Boolean)
+ extends BinaryExpression with Serializable with ImplicitCastInputTypes with
NullIntolerant {
+
+ private lazy val numberFormatter = {
+val value = right.eval()
+if (value != null) {
+ new ToNumberParser(value.toString.toUpperCase(Locale.ROOT), errorOnFail)
+} else {
+ null
+}
+ }
+
+ override def dataType: DataType = if (numberFormatter != null) {
+numberFormatter.parsedDecimalType
+ } else {
+DecimalType.USER_DEFAULT
+ }
+
+ override def inputTypes: Seq[DataType] = Seq(StringType, StringType)
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+val inputTypeCheck = super.checkInputDataTypes()
+if (inputTypeCheck.isSuccess) {
+ if (numberFormatter == null) {
+TypeCheckResult.TypeCheckSuccess
+ } else if (right.foldable) {
+numberFormatter.check()
+ } else {
+TypeCheckResult.TypeCheckFailure(s"Format expression must be foldable,
but got $right")
Review Comment:
https://github.com/apache/spark/pull/38531 is merged
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #38635: [SPARK-41118][SQL] `to_number`/`try_to_number` should return `null` when format is `null`
LuciferYang commented on code in PR #38635:
URL: https://github.com/apache/spark/pull/38635#discussion_r1022266640
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/numberFormatExpressions.scala:
##
@@ -26,6 +26,62 @@ import org.apache.spark.sql.catalyst.util.ToNumberParser
import org.apache.spark.sql.types.{AbstractDataType, DataType, Decimal,
DecimalType, StringType}
import org.apache.spark.unsafe.types.UTF8String
+abstract class ToNumberBase(left: Expression, right: Expression, errorOnFail:
Boolean)
+ extends BinaryExpression with Serializable with ImplicitCastInputTypes with
NullIntolerant {
+
+ private lazy val numberFormatter = {
+val value = right.eval()
+if (value != null) {
+ new ToNumberParser(value.toString.toUpperCase(Locale.ROOT), errorOnFail)
+} else {
+ null
+}
+ }
+
+ override def dataType: DataType = if (numberFormatter != null) {
+numberFormatter.parsedDecimalType
+ } else {
+DecimalType.USER_DEFAULT
+ }
+
+ override def inputTypes: Seq[DataType] = Seq(StringType, StringType)
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+val inputTypeCheck = super.checkInputDataTypes()
+if (inputTypeCheck.isSuccess) {
+ if (numberFormatter == null) {
+TypeCheckResult.TypeCheckSuccess
+ } else if (right.foldable) {
+numberFormatter.check()
+ } else {
+TypeCheckResult.TypeCheckFailure(s"Format expression must be foldable,
but got $right")
Review Comment:
also cc @panbingkun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #38635: [SPARK-41118][SQL] `to_number`/`try_to_number` should return `null` when format is `null`
LuciferYang commented on code in PR #38635:
URL: https://github.com/apache/spark/pull/38635#discussion_r1022266469
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/numberFormatExpressions.scala:
##
@@ -26,6 +26,62 @@ import org.apache.spark.sql.catalyst.util.ToNumberParser
import org.apache.spark.sql.types.{AbstractDataType, DataType, Decimal,
DecimalType, StringType}
import org.apache.spark.unsafe.types.UTF8String
+abstract class ToNumberBase(left: Expression, right: Expression, errorOnFail:
Boolean)
+ extends BinaryExpression with Serializable with ImplicitCastInputTypes with
NullIntolerant {
+
+ private lazy val numberFormatter = {
+val value = right.eval()
+if (value != null) {
+ new ToNumberParser(value.toString.toUpperCase(Locale.ROOT), errorOnFail)
+} else {
+ null
+}
+ }
+
+ override def dataType: DataType = if (numberFormatter != null) {
+numberFormatter.parsedDecimalType
+ } else {
+DecimalType.USER_DEFAULT
+ }
+
+ override def inputTypes: Seq[DataType] = Seq(StringType, StringType)
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+val inputTypeCheck = super.checkInputDataTypes()
+if (inputTypeCheck.isSuccess) {
+ if (numberFormatter == null) {
+TypeCheckResult.TypeCheckSuccess
+ } else if (right.foldable) {
+numberFormatter.check()
+ } else {
+TypeCheckResult.TypeCheckFailure(s"Format expression must be foldable,
but got $right")
Review Comment:
error framework should be used, and seems
https://github.com/apache/spark/pull/38531 is doing similar work
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
