[
https://issues.apache.org/jira/browse/SPARK-24009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16739952#comment-16739952
]
ant_nebula commented on SPARK-24009:
------------------------------------
Because on yarn client mode, "INSERT OVERWRITE LOCAL DIRECTORY" does not write
back to dirver node,
but on the yarn nodemanager which the last task run, and you does not has the
pemission to create directory '/home/spark/aaaaab'.
Of cause, It is not reasonable. It should write back to driver.
> spark2.3.0 INSERT OVERWRITE LOCAL DIRECTORY '/home/spark/aaaaab'
> -----------------------------------------------------------------
>
> Key: SPARK-24009
> URL: https://issues.apache.org/jira/browse/SPARK-24009
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.3.0
> Reporter: chris_j
> Priority: Major
>
> local mode spark execute "INSERT OVERWRITE LOCAL DIRECTORY " successfully.
> on yarn spark execute "INSERT OVERWRITE LOCAL DIRECTORY " failed, not
> permission problem also
>
> 1.spark-sql -e "INSERT OVERWRITE LOCAL DIRECTORY '/home/spark/aaaaab'row
> format delimited FIELDS TERMINATED BY '\t' STORED AS TEXTFILE select * from
> default.dim_date" write local directory successful
> 2.spark-sql --master yarn -e "INSERT OVERWRITE DIRECTORY 'aaaaab'row format
> delimited FIELDS TERMINATED BY '\t' STORED AS TEXTFILE select * from
> default.dim_date" write hdfs successful
> 3.spark-sql --master yarn -e "INSERT OVERWRITE LOCAL DIRECTORY
> '/home/spark/aaaaab'row format delimited FIELDS TERMINATED BY '\t' STORED AS
> TEXTFILE select * from default.dim_date" on yarn write local directory failed
>
>
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.io.IOException: Mkdirs failed to create
> [file:/home/spark/aaaaab/.hive-staging_hive_2018-04-18_14-14-37_208_1244164279218288723-1/-ext-10000/_temporary/0/_temporary/attempt_20180418141439_0000_m_000000_0|file://home/spark/aaaaab/.hive-staging_hive_2018-04-18_14-14-37_208_1244164279218288723-1/-ext-10000/_temporary/0/_temporary/attempt_20180418141439_0000_m_000000_0]
> (exists=false,
> cwd=[file:/data/hadoop/tmp/nm-local-dir/usercache/spark/appcache/application_1523246226712_0403/container_1523246226712_0403_01_000002|file://data/hadoop/tmp/nm-local-dir/usercache/spark/appcache/application_1523246226712_0403/container_1523246226712_0403_01_000002])
> at
> org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:249)
> at
> org.apache.spark.sql.hive.execution.HiveOutputWriter.<init>(HiveFileFormat.scala:123)
> at
> org.apache.spark.sql.hive.execution.HiveFileFormat$$anon$1.newInstance(HiveFileFormat.scala:103)
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$SingleDirectoryWriteTask.newOutputWriter(FileFormatWriter.scala:367)
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$SingleDirectoryWriteTask.execute(FileFormatWriter.scala:378)
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:269)
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:267)
> at
> org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1411)
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:272)
> ... 8 more
> Caused by: java.io.IOException: Mkdirs failed to create
> [file:/home/spark/aaaaab/.hive-staging_hive_2018-04-18_14-14-37_208_1244164279218288723-1/-ext-10000/_temporary/0/_temporary/attempt_20180418141439_0000_m_000000_0|file://home/spark/aaaaab/.hive-staging_hive_2018-04-18_14-14-37_208_1244164279218288723-1/-ext-10000/_temporary/0/_temporary/attempt_20180418141439_0000_m_000000_0]
> (exists=false,
> cwd=[file:/data/hadoop/tmp/nm-local-dir/usercache/spark/appcache/application_1523246226712_0403/container_1523246226712_0403_01_000002|file://data/hadoop/tmp/nm-local-dir/usercache/spark/appcache/application_1523246226712_0403/container_1523246226712_0403_01_000002])
> at
> org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:447)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:433)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:908)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:801)
> at
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat.getHiveRecordWriter(HiveIgnoreKeyTextOutputFormat.java:80)
> at
> org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getRecordWriter(HiveFileFormatUtils.java:261)
> at
> org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:246)
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
