[
https://issues.apache.org/jira/browse/SPARK-15619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15304911#comment-15304911
]
shane knapp edited comment on SPARK-15619 at 5/27/16 10:33 PM:
---------------------------------------------------------------
next time we have a maintenance, i will wipe /tmp completely so that we can at
least try and see what's creating what... right now it's such a mess that it's
hard to attribute anything to anything.
i did watch as a spark build (spark-master-test-maven-hadoop-2.7, IIRC) dump a
bunch of the liblz4-java3891256912513794605.so files in /tmp this morning on
worker-08 (the number string changes for each file).
and, so far today, on just one worker, we've had 2628 of these files left in
/tmp:
[root@amp-jenkins-worker-08 tmp]# ls -lt | grep liblz4 |grep "May 27" | wc -l
2628
i'm not worried about us running out of disk, and this is something i can
manage on the system-level, but it'd still be nice to have well behaved tests.
:)
was (Author: shaneknapp):
next time we have a maintenance, i will wipe /tmp completely so that we can at
least try and see what's creating what... right now it's such a mess that it's
hard to attribute anything to anything.
i did watch as a spark build (spark-master-test-maven-hadoop-2.7, IIRC) dump a
bunch of the liblz4-java3891256912513794605.so files in /tmp this morning on
worker-08 (the number string changes for each file).
and, so far today, we've had 2628 of these files left in /tmp:
[root@amp-jenkins-worker-08 tmp]# ls -lt | grep liblz4 |grep "May 27" | wc -l
2628
i'm not worried about us running out of disk, and this is something i can
manage on the system-level, but it'd still be nice to have well behaved tests.
:)
> spark builds filling up /tmp
> ----------------------------
>
> Key: SPARK-15619
> URL: https://issues.apache.org/jira/browse/SPARK-15619
> Project: Spark
> Issue Type: Bug
> Components: Build
> Reporter: shane knapp
> Priority: Minor
>
> spark builds aren't cleaning up /tmp after they run... it's hard to pinpoint
> EXACTLY what is left there by the spark builds (as other builds are also
> guilty of doing this), but a quick perusal of the /tmp directory during some
> spark builds show that there are myriad empty directories being created and a
> massive pile of shared object libraries being dumped there.
> $ for x in $(cat jenkins_workers.txt ); do echo $x; ssh $x "ls -l /tmp/*.so |
> wc -l"; done
> amp-jenkins-worker-01
> 0
> ls: cannot access /tmp/*.so: No such file or directory
> amp-jenkins-worker-02
> 22312
> amp-jenkins-worker-03
> 39673
> amp-jenkins-worker-04
> 39548
> amp-jenkins-worker-05
> 39577
> amp-jenkins-worker-06
> 39299
> amp-jenkins-worker-07
> 39315
> amp-jenkins-worker-08
> 38529
> to help combat this, i set up a cron job on each worker that runs tmpwatch
> during system downtime on sundays to clean up files older than a week.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
