>From branch-2.0, Spark 2.0.0 preview,
I found it interesting, no matter what you do by configuring
spark.sql.warehouse.dir
it will always pull up the default path which is /user/hive/warehouse
In the code, I notice that at LOC45
./sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
object SimpleAnalyzer extends Analyzer(
new SessionCatalog(
new InMemoryCatalog,
EmptyFunctionRegistry,
new SimpleCatalystConf(caseSensitiveAnalysis = true)),
new SimpleCatalystConf(caseSensitiveAnalysis = true))
It will always initialize with the SimpleCatalystConf which is applying the
hardcoded default value
defined in LOC58
./sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystConf.scala
case class SimpleCatalystConf(
caseSensitiveAnalysis: Boolean,
orderByOrdinal: Boolean = true,
groupByOrdinal: Boolean = true,
optimizerMaxIterations: Int = 100,
optimizerInSetConversionThreshold: Int = 10,
maxCaseBranchesForCodegen: Int = 20,
runSQLonFile: Boolean = true,
warehousePath: String = "/user/hive/warehouse")
extends CatalystConf
I couldn't find any other way to get around this.
It looks like this was fixed (in SPARK-15387) after
https://github.com/apache/spark/commit/9c817d027713859cac483b4baaaf8b53c040ad93
[https://avatars0.githubusercontent.com/u/4736016?v=3&s=200]<https://github.com/apache/spark/commit/9c817d027713859cac483b4baaaf8b53c040ad93>
[SPARK-15387][SQL] SessionCatalog in SimpleAnalyzer does not need to ... ยท
apache/spark@9c817d0<https://github.com/apache/spark/commit/9c817d027713859cac483b4baaaf8b53c040ad93>
github.com
...make database directory. ## What changes were proposed in this pull request?
After #12871 is fixed, we are forced to make `/user/hive/warehouse` when
SimpleAnalyzer is used but SimpleAnalyzer ma...
Just want to confirm this was the root cause and the PR that fixed it. Thanks.
