[ https://issues.apache.org/jira/browse/SPARK-1154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Wendell updated SPARK-1154: ----------------------------------- Priority: Critical (was: Major) > Spark fills up disk with app-* folders > -------------------------------------- > > Key: SPARK-1154 > URL: https://issues.apache.org/jira/browse/SPARK-1154 > Project: Spark > Issue Type: Improvement > Components: Deploy > Reporter: Evan Chan > Assignee: Piyush Kansal > Priority: Critical > Labels: starter > Fix For: 1.0.0 > > > Current version of Spark fills up the disk with many app-* folders: > $ ls /var/lib/spark > app-20140210022347-0597 app-20140212173327-0627 app-20140218154110-0657 > app-20140225232537-0017 app-20140225233548-0047 > app-20140210022407-0598 app-20140212173347-0628 app-20140218154130-0658 > app-20140225232551-0018 app-20140225233556-0048 > app-20140210022427-0599 app-20140212173754-0629 app-20140218164232-0659 > app-20140225232611-0019 app-20140225233603-0049 > app-20140210022447-0600 app-20140212182235-0630 app-20140218165133-0660 > app-20140225232802-0020 app-20140225233610-0050 > app-20140210022508-0601 app-20140212182256-0631 app-20140218165148-0661 > app-20140225232822-0021 app-20140225233617-0051 > app-20140210022528-0602 app-20140213000014-0632 app-20140218165225-0662 > app-20140225232940-0022 app-20140225233624-0052 > app-20140211024356-0603 app-20140213002026-0633 app-20140218165249-0663 > app-20140225233002-0023 app-20140225233631-0053 > app-20140211024417-0604 app-20140213154948-0634 app-20140218172030-0664 > app-20140225233056-0024 app-20140225233725-0054 > app-20140211024437-0605 app-20140213171810-0635 app-20140218193853-0665 > app-20140225233108-0025 app-20140225233731-0055 > app-20140211024457-0606 app-20140213193637-0636 app-20140218194442-0666 > app-20140225233124-0026 app-20140225233733-0056 > app-20140211024517-0607 app-20140214011513-0637 app-20140218194746-0667 > app-20140225233133-0027 app-20140225233734-0057 > app-20140211024538-0608 app-20140214012151-0638 app-20140218194822-0668 > app-20140225233147-0028 app-20140225233749-0058 > app-20140211193443-0609 app-20140214013134-0639 app-20140218212317-0669 > app-20140225233208-0029 app-20140225233759-0059 > app-20140211195210-0610 app-20140214013332-0640 app-20140225180142-0000 > app-20140225233215-0030 app-20140225233809-0060 > app-20140211213935-0611 app-20140214013642-0641 app-20140225180411-0001 > app-20140225233224-0031 app-20140225233828-0061 > app-20140211214227-0612 app-20140214014246-0642 app-20140225180431-0002 > app-20140225233232-0032 app-20140225234719-0062 > app-20140211215317-0613 app-20140214014607-0643 app-20140225180452-0003 > app-20140225233239-0033 app-20140226032845-0063 > app-20140211224601-0614 app-20140214184943-0644 app-20140225180512-0004 > app-20140225233320-0034 app-20140226033004-0064 > app-20140212022206-0615 app-20140214185118-0645 app-20140225180533-0005 > app-20140225233328-0035 app-20140226033119-0065 > app-20140212022226-0616 app-20140214185851-0646 app-20140225180553-0006 > app-20140225233354-0036 app-20140226033334-0066 > app-20140212022246-0617 app-20140214222856-0647 app-20140225181115-0007 > app-20140225233402-0037 app-20140226033354-0067 > app-20140212043704-0618 app-20140214231312-0648 app-20140225181244-0008 > app-20140225233409-0038 app-20140226033538-0068 > app-20140212043724-0619 app-20140214231434-0649 app-20140225182051-0009 > app-20140225233416-0039 app-20140226033826-0069 > app-20140212043745-0620 app-20140214231542-0650 app-20140225183009-0010 > app-20140225233426-0040 app-20140226034002-0070 > app-20140212044016-0621 app-20140214231616-0651 app-20140225184133-0011 > app-20140225233432-0041 app-20140226034053-0071 > app-20140212044203-0622 app-20140214233016-0652 app-20140225184318-0012 > app-20140225233439-0042 app-20140226034234-0072 > app-20140212044224-0623 app-20140214233037-0653 app-20140225184709-0013 > app-20140225233447-0043 app-20140226034426-0073 > app-20140212045034-0624 app-20140218153242-0654 app-20140225184844-0014 > app-20140225233526-0044 app-20140226034447-0074 > app-20140212045119-0625 app-20140218153341-0655 app-20140225190051-0015 > app-20140225233534-0045 > app-20140212173310-0626 app-20140218153442-0656 app-20140225232516-0016 > app-20140225233540-0046 > This problem is particularly bad if you have a whole bunch of fast jobs. > Also what makes the problem worse is that any jars for jobs is downloaded > into the app-* folder, so that fills up the disk particularly fast. > I would like to propose two things: > 1) Spark should have a cleanup thread (or actor) which periodically removes > old app-* folders; This should not be the responsibility of people > deploying Spark. > 2) The download of jars should not go to each app-* folder. This wastes a > huge amount of space because most jobs use the same jars. Maybe I can open a > separate ticket for this. -- This message was sent by Atlassian JIRA (v6.2#6252)