[jira] [Commented] (SPARK-30858) IntegralDivide's dataType should not depend on SQLConf.get
[ https://issues.apache.org/jira/browse/SPARK-30858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17039410#comment-17039410 ] Maxim Gekk commented on SPARK-30858: > This is a problem because the configuration can change between different > phases of planning [~hvanhovell] Is the code below right solution for the problem? {code:scala} case class IntegralDivide( left: Expression, right: Expression, integralDivideReturnLong: Boolean) extends DivModLike { def this(left: Expression, right: Expression) = { this(left, right, SQLConf.get.integralDivideReturnLong) } override def dataType: DataType = if (integralDivideReturnLong) { LongType } else { left.dataType } {code} > IntegralDivide's dataType should not depend on SQLConf.get > -- > > Key: SPARK-30858 > URL: https://issues.apache.org/jira/browse/SPARK-30858 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.0.0 >Reporter: Herman van Hövell >Priority: Blocker > > {{IntegralDivide}}'s dataType depends on the value of > {{SQLConf.get.integralDivideReturnLong}}. This is a problem because the > configuration can change between different phases of planning, and this can > silently break a query plan which can lead to crashes or data corruption. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-30858) IntegralDivide's dataType should not depend on SQLConf.get
[ https://issues.apache.org/jira/browse/SPARK-30858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17039414#comment-17039414 ] Herman van Hövell commented on SPARK-30858: --- That would work. I would sort of prefer if we can fix this during analysis, instead of relying on SQLConf.get for which we sometimes are not super sure where it comes from. > IntegralDivide's dataType should not depend on SQLConf.get > -- > > Key: SPARK-30858 > URL: https://issues.apache.org/jira/browse/SPARK-30858 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.0.0 >Reporter: Herman van Hövell >Priority: Blocker > > {{IntegralDivide}}'s dataType depends on the value of > {{SQLConf.get.integralDivideReturnLong}}. This is a problem because the > configuration can change between different phases of planning, and this can > silently break a query plan which can lead to crashes or data corruption. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-30858) IntegralDivide's dataType should not depend on SQLConf.get
[ https://issues.apache.org/jira/browse/SPARK-30858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17039433#comment-17039433 ] Maxim Gekk commented on SPARK-30858: The *div* function binds on this particular expressions [https://github.com/apache/spark/blob/919d551ddbf7575abe7fe47d4bbba62164d6d845/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala#L282] . I am not sure that we can replace it during analysis. > IntegralDivide's dataType should not depend on SQLConf.get > -- > > Key: SPARK-30858 > URL: https://issues.apache.org/jira/browse/SPARK-30858 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.0.0 >Reporter: Herman van Hövell >Priority: Blocker > > {{IntegralDivide}}'s dataType depends on the value of > {{SQLConf.get.integralDivideReturnLong}}. This is a problem because the > configuration can change between different phases of planning, and this can > silently break a query plan which can lead to crashes or data corruption. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org