[
https://issues.apache.org/jira/browse/SPARK-4812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shixiong Zhu updated SPARK-4812:
--------------------------------
Description:
The problem is `codegenEnabled` is `val`, but it uses a `val` `sqlContext`,
which can be override by subclasses. Here is a simple example to show this
issue.
{code}
scala> :paste
// Entering paste mode (ctrl-D to finish)
abstract class Foo {
protected val sqlContext = "Foo"
val codegenEnabled: Boolean = {
println(sqlContext) // it will call subclass's `sqlContext` which has not
yet been initialized.
if (sqlContext != null) {
true
} else {
false
}
}
}
class Bar extends Foo {
override val sqlContext = "Bar"
}
println(new Bar().codegenEnabled)
// Exiting paste mode, now interpreting.
null
false
defined class Foo
defined class Bar
scala>
{code}
To fix it, should override codegenEnabled in `InMemoryColumnarTableScan`.
was:
The problem is `codegenEnabled` is `val`, but it uses a `val` `sqlContext`,
which can be override by subclasses. Here is a simple example to show this
issue.
{code}
scala> :paste
// Entering paste mode (ctrl-D to finish)
abstract class Foo {
protected val sqlContext = "Foo"
val codegenEnabled: Boolean = {
println(sqlContext) // it will call subclass's `sqlContext` which has not
yet been initialized.
if (sqlContext != null) {
true
} else {
false
}
}
}
class Bar extends Foo {
override val sqlContext = "Bar"
}
println(new Bar().codegenEnabled)
// Exiting paste mode, now interpreting.
null
false
defined class Foo
defined class Bar
scala>
{code}
To fix it, we should mark `codegenEnabled` as `lazy`.
> SparkPlan.codegenEnabled may be initialized to a wrong value
> ------------------------------------------------------------
>
> Key: SPARK-4812
> URL: https://issues.apache.org/jira/browse/SPARK-4812
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Reporter: Shixiong Zhu
>
> The problem is `codegenEnabled` is `val`, but it uses a `val` `sqlContext`,
> which can be override by subclasses. Here is a simple example to show this
> issue.
> {code}
> scala> :paste
> // Entering paste mode (ctrl-D to finish)
> abstract class Foo {
> protected val sqlContext = "Foo"
> val codegenEnabled: Boolean = {
> println(sqlContext) // it will call subclass's `sqlContext` which has not
> yet been initialized.
> if (sqlContext != null) {
> true
> } else {
> false
> }
> }
> }
> class Bar extends Foo {
> override val sqlContext = "Bar"
> }
> println(new Bar().codegenEnabled)
> // Exiting paste mode, now interpreting.
> null
> false
> defined class Foo
> defined class Bar
> scala>
> {code}
> To fix it, should override codegenEnabled in `InMemoryColumnarTableScan`.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
