[
https://issues.apache.org/jira/browse/SPARK-30739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ohad Raviv resolved SPARK-30739.
--------------------------------
Resolution: Workaround
> unable to turn off Hadoop's trash feature
> -----------------------------------------
>
> Key: SPARK-30739
> URL: https://issues.apache.org/jira/browse/SPARK-30739
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.3.0
> Reporter: Ohad Raviv
> Priority: Minor
>
> We're trying to turn off the `TrashPolicyDefault` in one of our Spark
> applications by setting `spark.hadoop.fs.trash.interval=0`, but it just stays
> `360` as configured in our cluster's `core-site.xml`.
> Trying to debug it we managed to set
> `spark.hadoop.fs.trash.classname=OtherTrashPolicy` and it worked. the main
> difference seems to be that `spark.hadoop.fs.trash.classname` does not appear
> in any of the `*-site.xml` files.
> when we print the conf that get initialized in `TrashPolicyDefault` we get:
> ```
> Configuration: core-default.xml, core-site.xml, yarn-default.xml,
> yarn-site.xml, mapred-default.xml, mapred-site.xml, hdfs-default.xml,
> hdfs-site.xml,
> org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@561f0431,
> file:/hadoop03/yarn/local/usercache/.../hive-site.xml
> ```
> and:
> `fs.trash.interval=360 [programatically]`
> `fs.trash.classname=OtherTrashPolicy [programatically]`
>
> any idea why `fs.trash.classname` works but `fs.trash.interval` doesn't?
> this seems maybe related to: -SPARK-9825.-
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
