[
https://issues.apache.org/jira/browse/SPARK-30739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17031437#comment-17031437
]
Ohad Raviv commented on SPARK-30739:
------------------------------------
Closing as I realized this is actually the documented behaviour
[here|https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/core-default.xml].
_fs.trash.interval_
_Number of minutes between trash checkpoints. Should be smaller or equal to
fs.trash.interval. If zero, the value is set to the value of fs.trash.interval.
Every time the checkpointer runs it creates a new checkpoint out of current and
removes checkpoints created more than fs.trash.interval minutes ago._
so decided to use the _fs.trash.classname_ approach.
> unable to turn off Hadoop's trash feature
> -----------------------------------------
>
> Key: SPARK-30739
> URL: https://issues.apache.org/jira/browse/SPARK-30739
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.3.0
> Reporter: Ohad Raviv
> Priority: Minor
>
> We're trying to turn off the `TrashPolicyDefault` in one of our Spark
> applications by setting `spark.hadoop.fs.trash.interval=0`, but it just stays
> `360` as configured in our cluster's `core-site.xml`.
> Trying to debug it we managed to set
> `spark.hadoop.fs.trash.classname=OtherTrashPolicy` and it worked. the main
> difference seems to be that `spark.hadoop.fs.trash.classname` does not appear
> in any of the `*-site.xml` files.
> when we print the conf that get initialized in `TrashPolicyDefault` we get:
> ```
> Configuration: core-default.xml, core-site.xml, yarn-default.xml,
> yarn-site.xml, mapred-default.xml, mapred-site.xml, hdfs-default.xml,
> hdfs-site.xml,
> org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@561f0431,
> file:/hadoop03/yarn/local/usercache/.../hive-site.xml
> ```
> and:
> `fs.trash.interval=360 [programatically]`
> `fs.trash.classname=OtherTrashPolicy [programatically]`
>
> any idea why `fs.trash.classname` works but `fs.trash.interval` doesn't?
> this seems maybe related to: -SPARK-9825.-
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
