Github user uncleGen commented on the issue:
https://github.com/apache/spark/pull/16142
@vanzin I will close this PR, but can't take your point completely.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/16142
> #16165 has supported deleting too old in-progress job logs. So I think it
is OK in this case.
It's *not* OK. That change uses an heuristic to decide when to delete
"inprogress" logs: if
Github user uncleGen commented on the issue:
https://github.com/apache/spark/pull/16142
@srowen It has real demand. I have received some reports from our users:
they care about space cost from job event logs, but not care about logs age, no
matter how old it is.
If you guys
Github user uncleGen commented on the issue:
https://github.com/apache/spark/pull/16142
> The current scan code does not make one request to the NameNode per log
file in the directory. Your code does. That should be avoided.
Make sense, current implementation can be optimized
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/16142
I personally don't see much value in deleting by 'size' because there's no
particular reason to believe that files that happen to be bigger contain less
important info. If you want to delete more
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/16142
> I just add a new clean-up mode, but not add the cleaner itself.
But that's kinda the point. How many different ways of cleaning need to be
added? Will this one be enough? Will people ask
Github user uncleGen commented on the issue:
https://github.com/apache/spark/pull/16142
Besides, the unit test has proved that the older file will be cleaned first.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user uncleGen commented on the issue:
https://github.com/apache/spark/pull/16142
@vanzin
> For the feature, it feels like it's trying to make the SHS more like a
"log management system" than a history server.
Sorry, I do not get it. I just add a new clean-up
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/16142
I'm not such a big fan of this feature, but mostly I'm not a big fan of the
current implementation.
For the feature, it feels like it's trying to make the SHS more like a "log
management
Github user uncleGen commented on the issue:
https://github.com/apache/spark/pull/16142
cc @vanzin
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if
Github user uncleGen commented on the issue:
https://github.com/apache/spark/pull/16142
@AmplabJenkins retest please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user uncleGen commented on the issue:
https://github.com/apache/spark/pull/16142
retest it please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so,
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16142
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70066/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16142
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16142
**[Test build #70066 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70066/consoleFull)**
for PR 16142 at commit
Github user uncleGen commented on the issue:
https://github.com/apache/spark/pull/16142
retest please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16142
**[Test build #70059 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70059/consoleFull)**
for PR 16142 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16142
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70059/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16142
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16142
**[Test build #70059 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70059/consoleFull)**
for PR 16142 at commit
Github user uncleGen commented on the issue:
https://github.com/apache/spark/pull/16142
@vanzin I have removed related changes in `EventLoggingListener`, and
provide a new clean mode, i.e. `space` based mode. Please take a review.
---
If your project is set up for it, you can reply
Github user uncleGen commented on the issue:
https://github.com/apache/spark/pull/16142
@vanzin I found you provided this base work, could you please take a look?
Thank you.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user uncleGen commented on the issue:
https://github.com/apache/spark/pull/16142
@srowen If I have understand what you mean correctly, the **"log
rotation"** is different with **"job event log clean up"**. The "job event
log" is used to reply to build spark history ui.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/16142
Yes, but the alternative is reimplementing an ad-hoc log rotation system
here, which isn't great either. Are you saying the history server already
manages logs? pardon, I don't know it at all.
---
Github user uncleGen commented on the issue:
https://github.com/apache/spark/pull/16142
@srowen Spark History Server may do the clean-up work. The precondition is
we start it and it keeps running. Besides, if there are abundant applications
constantly, the event log may still take up
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/16142
Hm, does Spark generally manage log rotation? I confess ignorance. It's
something you often leave to a cron job or something to archive and clean up.
---
If your project is set up for it, you can
Github user uncleGen commented on the issue:
https://github.com/apache/spark/pull/16142
cc @srowen
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16142
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69668/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16142
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16142
**[Test build #69668 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69668/consoleFull)**
for PR 16142 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16142
**[Test build #69668 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69668/consoleFull)**
for PR 16142 at commit
Github user uncleGen commented on the issue:
https://github.com/apache/spark/pull/16142
@SparkQA Test it agagin
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user uncleGen commented on the issue:
https://github.com/apache/spark/pull/16142
retest it please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so,
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16142
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16142
**[Test build #69666 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69666/consoleFull)**
for PR 16142 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16142
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69666/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16142
**[Test build #69666 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69666/consoleFull)**
for PR 16142 at commit
37 matches
Mail list logo
