Eugene-Mark commented on code in PR #36499:
URL: https://github.com/apache/spark/pull/36499#discussion_r891588142
##########
sql/core/src/main/scala/org/apache/spark/sql/jdbc/TeradataDialect.scala:
##########
@@ -96,4 +97,29 @@ private case object TeradataDialect extends JdbcDialect {
override def getLimitClause(limit: Integer): String = {
""
}
+
+ override def getCatalystType(
+ sqlType: Int, typeName: String, size: Int, md: MetadataBuilder):
Option[DataType] = {
+ if (sqlType == Types.NUMERIC) {
+ if (md == null) {
+ Option(DecimalType.SYSTEM_DEFAULT)
+ } else {
+ val scale = md.build().getLong("scale")
+ // In Teradata, Number or Number(*) means precision and scale is
flexible.
+ // However, the scale returned from JDBC is 0, which leads to
fractional part loss.
+ // Handle this special case by adding explicit conversion to system
default decimal type.
+ // Note, even if the NUMBER is defined with explicit precision and
scale like NUMBER(20, 0),
+ // Spark will treat it as DecimalType.SYSTEM_DEFAULT, which is
NUMBER(38,18)
+ if (scale == 0) {
+ Option(DecimalType.SYSTEM_DEFAULT)
+ } else {
+ // In Teradata, Number(*, scale) will return size, namely precision,
as 40,
+ // which conflicts to DecimalType.MAX_PRECISION
+ Option(DecimalType(Math.min(size, DecimalType.MAX_PRECISION),
scale.toInt))
Review Comment:
Thanks for this comment! It reminds me of sth more reasonable than my
current practice! Since in Teradata, only Number(*)/Number, Number(*,scale) and
Number(precision,scale) is valid expression, which means when scale is
flexible, the precision returned must be 40. So we don't need to convert all
scale = 0 to default decimal type, but only need to do it when the precision =
40 is detected. Which means we will respect user's explicit scale = 0 settings,
like Number(20,0), will be converted to DecimalType(20,0).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
