[jira] [Commented] (SPARK-15269) Creating external table in test code leaves empty directory under warehouse directory
[
https://issues.apache.org/jira/browse/SPARK-15269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15281657#comment-15281657
]
Cheng Lian commented on SPARK-15269:
[~xwu0226] Thanks a lot for the detailed investigation! Would you like to fix
this issue?
> Creating external table in test code leaves empty directory under warehouse
> directory
> -
>
> Key: SPARK-15269
> URL: https://issues.apache.org/jira/browse/SPARK-15269
> Project: Spark
> Issue Type: Bug
> Components: SQL, Tests
>Affects Versions: 2.0.0
>Reporter: Cheng Lian
>
> It seems that this issue doesn't affect production code. I couldn't reproduce
> it using Spark shell.
> Adding the following test case in {{HiveDDLSuite}} may reproduce this issue:
> {code}
> test("foo") {
> withTempPath { dir =>
> val path = dir.getCanonicalPath
> spark.range(1).write.json(path)
> withTable("ddl_test1") {
> sql(s"CREATE TABLE ddl_test1 USING json OPTIONS (PATH '$path')")
> sql("DROP TABLE ddl_test1")
> sql(s"CREATE TABLE ddl_test1 USING json AS SELECT 1 AS a")
> }
> }
> }
> {code}
> Note that the first {{CREATE TABLE}} command creates an external table since
> data source tables are always external when {{PATH}} option is specified.
> When executing the second {{CREATE TABLE}} command, which creates a managed
> table with the same name, it fails because there's already an unexpected
> directory with the same name as the table name in the warehouse directory:
> {noformat}
> [info] - foo *** FAILED *** (7 seconds, 649 milliseconds)
> [info] org.apache.spark.sql.AnalysisException: path
> file:/Users/lian/local/src/spark/workspace-b/target/tmp/warehouse-205e25e7-8918-4615-acf1-10e06af7c35c/ddl_test1
> already exists.;
> [info] at
> org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation.run(InsertIntoHadoopFsRelation.scala:88)
> [info] at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:57)
> [info] at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:55)
> [info] at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:69)
> [info] at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)
> [info] at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)
> [info] at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:136)
> [info] at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
> [info] at
> org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:133)
> [info] at
> org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:114)
> [info] at
> org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:85)
> [info] at
> org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:85)
> [info] at
> org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:417)
> [info] at
> org.apache.spark.sql.execution.command.CreateDataSourceTableAsSelectCommand.run(createDataSourceTables.scala:231)
> [info] at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:57)
> [info] at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:55)
> [info] at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:69)
> [info] at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)
> [info] at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)
> [info] at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:136)
> [info] at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
> [info] at
> org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:133)
> [info] at
> org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:114)
> [info] at
> org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:85)
> [info] at
> org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:85)
> [info] at org.apache.spark.sql.Dataset.(Dataset.scala:186)
> [info] at org.apache.spark.sql.Dataset.(Dataset.scala:167)
> [info] at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:62)
> [info] at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:541)
> [info] at
> org.apache.spark.sql.test.SQLTestUtils$$anonfun$sql$1.apply(SQLTestUtils.scala:59)
> [info] at
>
[jira] [Commented] (SPARK-15269) Creating external table in test code leaves empty directory under warehouse directory
[
https://issues.apache.org/jira/browse/SPARK-15269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15281621#comment-15281621
]
Xin Wu commented on SPARK-15269:
For the case where we can not recreate this issue, it is because the default
database path we got at {code}if (!new
CaseInsensitiveMap(options).contains("path")) {
isExternal = false
options + ("path" -> sessionState.catalog.defaultTablePath(tableIdent))
} else {
options
}{code} is different from hive metastore's default warehouse dir. They
are "/user/hive/warehouse" and "/spark-warehouse", respectively.
When creating the first table, hive metastore's default warehouse dir is
"/spark-warehouse", while when creating the second table without
PATH option, the sessionState.catalog.defaultTablePath returns
"/user/hive/warehouse". Therefore, the 2nd table creation will not hit the
issue. But the first table still leave the empty table directory behind after
being dropped.
Two questions:
1. Should we keep these 2 default database path consistent?
2. If they are consistent, we will hit the issue reported in this JIRA.. Then,
can we also assign the provided path to the CatalogTable.storage.locationURI,
even though newSparkSQLSpecificMetastoreTable is called in
createDataSourceTables for a non-hive compatible metastore table?
> Creating external table in test code leaves empty directory under warehouse
> directory
> -
>
> Key: SPARK-15269
> URL: https://issues.apache.org/jira/browse/SPARK-15269
> Project: Spark
> Issue Type: Bug
> Components: SQL, Tests
>Affects Versions: 2.0.0
>Reporter: Cheng Lian
>
> It seems that this issue doesn't affect production code. I couldn't reproduce
> it using Spark shell.
> Adding the following test case in {{HiveDDLSuite}} may reproduce this issue:
> {code}
> test("foo") {
> withTempPath { dir =>
> val path = dir.getCanonicalPath
> spark.range(1).write.json(path)
> withTable("ddl_test1") {
> sql(s"CREATE TABLE ddl_test1 USING json OPTIONS (PATH '$path')")
> sql("DROP TABLE ddl_test1")
> sql(s"CREATE TABLE ddl_test1 USING json AS SELECT 1 AS a")
> }
> }
> }
> {code}
> Note that the first {{CREATE TABLE}} command creates an external table since
> data source tables are always external when {{PATH}} option is specified.
> When executing the second {{CREATE TABLE}} command, which creates a managed
> table with the same name, it fails because there's already an unexpected
> directory with the same name as the table name in the warehouse directory:
> {noformat}
> [info] - foo *** FAILED *** (7 seconds, 649 milliseconds)
> [info] org.apache.spark.sql.AnalysisException: path
> file:/Users/lian/local/src/spark/workspace-b/target/tmp/warehouse-205e25e7-8918-4615-acf1-10e06af7c35c/ddl_test1
> already exists.;
> [info] at
> org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation.run(InsertIntoHadoopFsRelation.scala:88)
> [info] at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:57)
> [info] at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:55)
> [info] at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:69)
> [info] at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)
> [info] at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)
> [info] at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:136)
> [info] at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
> [info] at
> org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:133)
> [info] at
> org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:114)
> [info] at
> org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:85)
> [info] at
> org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:85)
> [info] at
> org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:417)
> [info] at
> org.apache.spark.sql.execution.command.CreateDataSourceTableAsSelectCommand.run(createDataSourceTables.scala:231)
> [info] at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:57)
> [info] at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:55)
> [info] at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:69)
> [info] at
>
[jira] [Commented] (SPARK-15269) Creating external table in test code leaves empty directory under warehouse directory
[
https://issues.apache.org/jira/browse/SPARK-15269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15280888#comment-15280888
]
Xin Wu commented on SPARK-15269:
In spark-shell, I can recreate it as following:
{code}
scala> spark.range(1).write.json("/home/xwu0226/spark-test/data/spark-15269")
Datasource.write -> Path: file:/home/xwu0226/spark-test/data/spark-15269
scala> spark.sql("create table spark_15269 using json options(PATH
'/home/xwu0226/spark-test/data/spark-15269')")
16/05/11 14:51:00 WARN CreateDataSourceTableUtils: Couldn't find corresponding
Hive SerDe for data source provider json. Persisting data source relation
`spark_15269` into Hive metastore in Spark SQL specific format, which is NOT
compatible with Hive.
going through newSparkSQLSpecificMetastoreTable()
res1: org.apache.spark.sql.DataFrame = []
scala> spark.sql("drop table spark_15269")
res2: org.apache.spark.sql.DataFrame = []
scala> spark.sql("create table spark_15269 using json as select 1 as a")
org.apache.spark.sql.AnalysisException: path
file:/user/hive/warehouse/spark_15269 already exists.;
at
org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation.run(InsertIntoHadoopFsRelation.scala:88)
at
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:62)
at
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:60)
at
org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:74)
at
org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)
at
org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)
at
org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:136)
at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:133)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:114)
at
org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:85)
at
org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:85)
at
org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:418)
at
org.apache.spark.sql.execution.command.CreateDataSourceTableAsSelectCommand.run(createDataSourceTables.scala:229)
at
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:62)
at
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:60)
at
org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:74)
at
org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)
at
org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)
at
org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:136)
at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:133)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:114)
at
org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:85)
at
org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:85)
at org.apache.spark.sql.Dataset.(Dataset.scala:186)
at org.apache.spark.sql.Dataset.(Dataset.scala:167)
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:62)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:541)
... 48 elided
{code}
> Creating external table in test code leaves empty directory under warehouse
> directory
> -
>
> Key: SPARK-15269
> URL: https://issues.apache.org/jira/browse/SPARK-15269
> Project: Spark
> Issue Type: Bug
> Components: SQL, Tests
>Affects Versions: 2.0.0
>Reporter: Cheng Lian
>
> It seems that this issue doesn't affect production code. I couldn't reproduce
> it using Spark shell.
> Adding the following test case in {{HiveDDLSuite}} may reproduce this issue:
> {code}
> test("foo") {
> withTempPath { dir =>
> val path = dir.getCanonicalPath
> spark.range(1).write.json(path)
> withTable("ddl_test1") {
> sql(s"CREATE TABLE ddl_test1 USING json OPTIONS (PATH '$path')")
> sql("DROP TABLE ddl_test1")
> sql(s"CREATE TABLE ddl_test1 USING json AS SELECT 1 AS a")
> }
> }
> }
> {code}
> Note that the first {{CREATE TABLE}} command creates an external table since
> data source tables are always external when {{PATH}} option is
[jira] [Commented] (SPARK-15269) Creating external table in test code leaves empty directory under warehouse directory
[
https://issues.apache.org/jira/browse/SPARK-15269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15280877#comment-15280877
]
Xin Wu commented on SPARK-15269:
The root cause maybe the following?
When the first table is created as external table with the data source path,
but as `json`, `createDataSourceTables` considers it as non-hive compatible
table because `json` is not a Hive SerDe. Then,
`newSparkSQLSpecificMetastoreTable` is invoked to create the `CatalogTable`
before asking `HiveClient` to create the metastore table. In this call,
`locationURI` is not set. So when we convert CatalogTable to HiveTable before
passing to Hive Metastore, hive table's data location is not set. Then, Hive
metastore implicitly creates a data location as `/tableName`,
which
`/Users/lian/local/src/spark/workspace-b/target/tmp/warehouse-205e25e7-8918-4615-acf1-10e06af7c35c/ddl_test1`
in this JIRA. I also verified that creating an external directly in Hive shell
without a path will result in a default table directory created by hive.
Then, even after dropping table, hive will not delete this stealth directory
because the table is external.
> Creating external table in test code leaves empty directory under warehouse
> directory
> -
>
> Key: SPARK-15269
> URL: https://issues.apache.org/jira/browse/SPARK-15269
> Project: Spark
> Issue Type: Bug
> Components: SQL, Tests
>Affects Versions: 2.0.0
>Reporter: Cheng Lian
>
> It seems that this issue doesn't affect production code. I couldn't reproduce
> it using Spark shell.
> Adding the following test case in {{HiveDDLSuite}} may reproduce this issue:
> {code}
> test("foo") {
> withTempPath { dir =>
> val path = dir.getCanonicalPath
> spark.range(1).write.json(path)
> withTable("ddl_test1") {
> sql(s"CREATE TABLE ddl_test1 USING json OPTIONS (PATH '$path')")
> sql("DROP TABLE ddl_test1")
> sql(s"CREATE TABLE ddl_test1 USING json AS SELECT 1 AS a")
> }
> }
> }
> {code}
> Note that the first {{CREATE TABLE}} command creates an external table since
> data source tables are always external when {{PATH}} option is specified.
> When executing the second {{CREATE TABLE}} command, which creates a managed
> table with the same name, it fails because there's already an unexpected
> directory with the same name as the table name in the warehouse directory:
> {noformat}
> [info] - foo *** FAILED *** (7 seconds, 649 milliseconds)
> [info] org.apache.spark.sql.AnalysisException: path
> file:/Users/lian/local/src/spark/workspace-b/target/tmp/warehouse-205e25e7-8918-4615-acf1-10e06af7c35c/ddl_test1
> already exists.;
> [info] at
> org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation.run(InsertIntoHadoopFsRelation.scala:88)
> [info] at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:57)
> [info] at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:55)
> [info] at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:69)
> [info] at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)
> [info] at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)
> [info] at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:136)
> [info] at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
> [info] at
> org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:133)
> [info] at
> org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:114)
> [info] at
> org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:85)
> [info] at
> org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:85)
> [info] at
> org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:417)
> [info] at
> org.apache.spark.sql.execution.command.CreateDataSourceTableAsSelectCommand.run(createDataSourceTables.scala:231)
> [info] at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:57)
> [info] at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:55)
> [info] at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:69)
> [info] at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)
> [info] at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)
> [info] at
>
[jira] [Commented] (SPARK-15269) Creating external table in test code leaves empty directory under warehouse directory
[
https://issues.apache.org/jira/browse/SPARK-15269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15279992#comment-15279992
]
Cheng Lian commented on SPARK-15269:
Investigated this issue for a while, and observed the following facts:
When executing the first DDL command, we may have the following call chain:
{noformat}
CreateDataSourceTableUtils.createDataSourceTable
...
HiveClientImpl.createTable
withHiveState(1)
shim.setCurrentSessionState(state) (2)
{noformat}
The unexpected empty table directory doesn't exist at (1), but showed up almost
immediately after invoking (2) although (2) itself doesn't create any
directories.
It seems like some background thread is scanning something...
> Creating external table in test code leaves empty directory under warehouse
> directory
> -
>
> Key: SPARK-15269
> URL: https://issues.apache.org/jira/browse/SPARK-15269
> Project: Spark
> Issue Type: Bug
> Components: SQL, Tests
>Affects Versions: 2.0.0
>Reporter: Cheng Lian
>
> It seems that this issue doesn't affect production code. I couldn't reproduce
> it using Spark shell.
> Adding the following test case in {{HiveDDLSuite}} may reproduce this issue:
> {code}
> test("foo") {
> withTempPath { dir =>
> val path = dir.getCanonicalPath
> spark.range(1).write.json(path)
> withTable("ddl_test1") {
> sql(s"CREATE TABLE ddl_test1 USING json OPTIONS (PATH '$path')")
> sql("DROP TABLE ddl_test1")
> sql(s"CREATE TABLE ddl_test1 USING json AS SELECT 1 AS a")
> }
> }
> }
> {code}
> Note that the first {{CREATE TABLE}} command creates an external table since
> data source tables are always external when {{PATH}} option is specified.
> When executing the second {{CREATE TABLE}} command, which creates a managed
> table with the same name, it fails because there's already an unexpected
> directory with the same name as the table name in the warehouse directory:
> {noformat}
> [info] - foo *** FAILED *** (7 seconds, 649 milliseconds)
> [info] org.apache.spark.sql.AnalysisException: path
> file:/Users/lian/local/src/spark/workspace-b/target/tmp/warehouse-205e25e7-8918-4615-acf1-10e06af7c35c/ddl_test1
> already exists.;
> [info] at
> org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation.run(InsertIntoHadoopFsRelation.scala:88)
> [info] at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:57)
> [info] at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:55)
> [info] at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:69)
> [info] at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)
> [info] at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)
> [info] at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:136)
> [info] at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
> [info] at
> org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:133)
> [info] at
> org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:114)
> [info] at
> org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:85)
> [info] at
> org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:85)
> [info] at
> org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:417)
> [info] at
> org.apache.spark.sql.execution.command.CreateDataSourceTableAsSelectCommand.run(createDataSourceTables.scala:231)
> [info] at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:57)
> [info] at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:55)
> [info] at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:69)
> [info] at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)
> [info] at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)
> [info] at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:136)
> [info] at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
> [info] at
> org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:133)
> [info] at
> org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:114)
> [info] at
>
