Ohad Raviv created SPARK-30739:
----------------------------------
Summary: unable to turn off Hadoop's trash feature
Key: SPARK-30739
URL: https://issues.apache.org/jira/browse/SPARK-30739
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 2.3.0
Reporter: Ohad Raviv
We're trying to turn off the `TrashPolicyDefault` in one of our Spark
applications by setting `spark.hadoop.fs.trash.interval=0`, but it just stays
`360` as configured in our cluster's `core-site.xml`.
Trying to debug it we managed to set
`spark.hadoop.fs.trash.classname=OtherTrashPolicy` and it worked. the main
difference seems to be that `spark.hadoop.fs.trash.classname` does not appear
in any of the `*-site.xml` files.
when we print the conf that get initialized in `TrashPolicyDefault` we get:
```
Configuration: core-default.xml, core-site.xml, yarn-default.xml,
yarn-site.xml, mapred-default.xml, mapred-site.xml, hdfs-default.xml,
hdfs-site.xml,
org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@561f0431,
file:/hadoop03/yarn/local/usercache/.../hive-site.xml
```
and:
`fs.trash.interval=360 [programatically]`
`fs.trash.classname=OtherTrashPolicy [programatically]`
any idea why `fs.trash.classname` works but `fs.trash.interval` doesn't?
this seems maybe related to: -SPARK-9825.-
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
