Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/16924
My only concern is that this will cause the SHS to parse these logs more
often, but that's really a problem in other parts of the code that need to be
made smarter.
Merging to master.
---
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/16924
ok then this LGTM, but I'm not very confident about this part of code, cc
@vanzin
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/16924
Is there any concern regarding checking file length? I guess we have no
much choices here to detect file changes?
There are comments for why using file length to detect file changes:
https:/
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/16924
> Hadoop doesn't update file length until close and therefor Spark's
history server can't detect any changes
So the history server detect file changes by checking file length? Is there
a
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/16924
In `FsHistoryProvider`, since there is a check for file size, I think it is
designed to find updated logs from running applications?
https://github.com/apache/spark/blob/e26dac5feb02033f980b
Github user ajbozarth commented on the issue:
https://github.com/apache/spark/pull/16924
@cloud-fan The History Server has an incomplete applications page as part
of its functionality that reads through files still marked .inprogress. Though
I'm not a big fan of this use case for the
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/16924
I'm confused, how can your customer see a still running application in
history server?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as we
Github user dosoft commented on the issue:
https://github.com/apache/spark/pull/16924
This issue has been reported by our customer as following:
"Subsequently connecting to spark thriftserver via beeline and running any
MR job, it doesnt get reflected in spark history server U
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/16924
no, history server is not a good use case, for long running jobs/apps,
history server should ONLY detect the change and show the app after it's
finished.
Can you explain more about your u
Github user dosoft commented on the issue:
https://github.com/apache/spark/pull/16924
Just an example of the use case. This issue affects all long-life apps.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your projec
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/16924
so this PR is not for history server, but for thrift server?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does no
Github user dosoft commented on the issue:
https://github.com/apache/spark/pull/16924
@cloud-fan spark thriftserver as long-running application?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not hav
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/16924
what's the use case of this patch? I think it makes sense to show the
history entry after the app is finished, I'm not sure why we want the history
server to detect it earlier.
---
If your proje
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16924
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78487/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16924
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16924
**[Test build #78487 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78487/testReport)**
for PR 16924 at commit
[`b54b18d`](https://github.com/apache/spark/commit/b
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16924
**[Test build #78487 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78487/testReport)**
for PR 16924 at commit
[`b54b18d`](https://github.com/apache/spark/commit/b5
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/16924
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes s
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16924
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78461/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16924
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16924
**[Test build #78461 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78461/testReport)**
for PR 16924 at commit
[`b54b18d`](https://github.com/apache/spark/commit/b
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16924
**[Test build #78461 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78461/testReport)**
for PR 16924 at commit
[`b54b18d`](https://github.com/apache/spark/commit/b5
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/16924
ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/16924
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled an
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/16924
test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so,
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/16924
add to whitelist
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so,
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/16924
test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so,
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/16924
Could you bring this up to date? @dosoft
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this featu
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16924
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feat
29 matches
Mail list logo
