[ https://issues.apache.org/jira/browse/SPARK-34584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17292963#comment-17292963 ]
Gengliang Wang edited comment on SPARK-34584 at 3/1/21, 3:35 PM: ----------------------------------------------------------------- The difference is from https://github.com/apache/spark/pull/27597, which only covers the V1 write path. We should use ANSI cast in `ResolveInsertInto` for the same case as well. cc [~Ngone51][~cloud_fan] was (Author: gengliang.wang): The difference is from https://github.com/apache/spark/pull/27597, which only covers the V1 write path. cc [~Ngone51][~cloud_fan] > When insert into a partition table with a illegal partition value, DSV2 > behavior different as others > ---------------------------------------------------------------------------------------------------- > > Key: SPARK-34584 > URL: https://issues.apache.org/jira/browse/SPARK-34584 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 3.1.1 > Reporter: angerszhu > Priority: Major > > With below UTÂ > {code:java} > test("test insert to partition with wrong value") { > withTable("t") { > val binaryStr = "Spark SQL" > val binaryHexStr = > Hex.hex(UTF8String.fromString(binaryStr).getBytes).toString > sql("CREATE TABLE t(name STRING, part DATE) USING PARQUET PARTITIONED > BY (part)") > sql(s"INSERT INTO t PARTITION(part = X'$binaryHexStr') VALUES('a')") > sql("SELECT * FROM t").show() > } > } > {code} > Result : > {code:java} > [info] DSV2SQLInsertTestSuite: > 21:35:32.369 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load > native-hadoop library for your platform... using builtin-java classes where > applicable > +----+----+ > |name|part| > +----+----+ > | a|null| > +----+----+ > [info] - test insert to partition with wrong value (4 seconds, 41 > milliseconds) > 21:35:37.639 WARN org.apache.spark.sql.DSV2SQLInsertTestSuite:===== POSSIBLE > THREAD LEAK IN SUITE o.a.s.sql.DSV2SQLInsertTestSuite, thread names: > rpc-boss-3-1, shuffle-boss-6-1 ===== > [info] FileSourceSQLInsertTestSuite: > [info] - test insert to partition with wrong value *** FAILED *** (200 > milliseconds) > [info] java.time.DateTimeException: Cannot cast X'537061726B2053514C' to > DateType. > [info] at > org.apache.spark.sql.catalyst.util.DateTimeUtils$.$anonfun$stringToDateAnsi$1(DateTimeUtils.scala:471) > [info] at scala.Option.getOrElse(Option.scala:189) > [info] at > org.apache.spark.sql.catalyst.util.DateTimeUtils$.stringToDateAnsi(DateTimeUtils.scala:471) > [info] at > org.apache.spark.sql.catalyst.expressions.CastBase.$anonfun$castToDate$2(Cast.scala:509) > [info] at > org.apache.spark.sql.catalyst.expressions.CastBase.$anonfun$castToDate$2$adapted(Cast.scala:509) > [info] at > org.apache.spark.sql.catalyst.expressions.CastBase.buildCast(Cast.scala:301) > [info] at > org.apache.spark.sql.catalyst.expressions.CastBase.$anonfun$castToDate$1(Cast.scala:509) > [info] at > org.apache.spark.sql.catalyst.expressions.CastBase.nullSafeEval(Cast.scala:850) > [info] at > org.apache.spark.sql.catalyst.expressions.UnaryExpression.eval(Expression.scala:482) > [info] at > org.apache.spark.sql.catalyst.optimizer.ConstantFolding$$anonfun$apply$1$$anonfun$applyOrElse$1.applyOrElse(expressions.scala:66) > [info] at > org.apache.spark.sql.catalyst.optimizer.ConstantFolding$$anonfun$apply$1$$anonfun$applyOrElse$1.applyOrElse(expressions.scala:54) > [info] at > org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDown$1(TreeNode.scala:316) > [info] at > org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:72) > [info] at > org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:316) > [info] at > org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDown$3(TreeNode.scala:321) > [info] at > org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$mapChildren$1(TreeNode.scala:406) > [info] at > org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:242) > [info] at > org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:404) > [info] at > org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:357) > [info] at > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org