Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/18334
thanks, merging to master!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18334
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78972/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18334
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18334
**[Test build #78972 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78972/testReport)**
for PR 18334 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18334
**[Test build #78972 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78972/testReport)**
for PR 18334 at commit
Github user wzhfy commented on the issue:
https://github.com/apache/spark/pull/18334
@rxin Currently we are re-calculating the stats. If we want to support
incremental stats update, we may need to maintain some specific data
structures. e.g. for ndv in column stats, store data
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/18334
Can the stats be updated incrementally?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18334
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78918/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18334
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18334
**[Test build #78918 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78918/testReport)**
for PR 18334 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18334
**[Test build #78918 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78918/testReport)**
for PR 18334 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18334
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18334
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78903/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18334
**[Test build #78903 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78903/testReport)**
for PR 18334 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18334
**[Test build #78903 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78903/testReport)**
for PR 18334 at commit
Github user wzhfy commented on the issue:
https://github.com/apache/spark/pull/18334
PR for invalidating stats is submitted:
[#18449](https://github.com/apache/spark/pull/18449)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user wzhfy commented on the issue:
https://github.com/apache/spark/pull/18334
@cloud-fan OK. I'll create another ticket for invalidating stats.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18334
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78739/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18334
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18334
**[Test build #78739 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78739/testReport)**
for PR 18334 at commit
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/18334
@wzhfy Let's create a new JIRA ticket and link it in this PR, as this PR
does 2 things:
1. invalidate stats after data changing
2. auto update stats if the config is on
---
If your
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18334
**[Test build #78739 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78739/testReport)**
for PR 18334 at commit
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/18334
Will do a review tonight. Sorry for the delay
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/18334
LGTM except one minor comment
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18334
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78629/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18334
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18334
**[Test build #78629 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78629/testReport)**
for PR 18334 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18334
**[Test build #78629 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78629/testReport)**
for PR 18334 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18334
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78602/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18334
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18334
**[Test build #78602 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78602/testReport)**
for PR 18334 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18334
**[Test build #78602 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78602/testReport)**
for PR 18334 at commit
Github user wzhfy commented on the issue:
https://github.com/apache/spark/pull/18334
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18334
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18334
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78592/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18334
**[Test build #78592 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78592/testReport)**
for PR 18334 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18334
**[Test build #78592 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78592/testReport)**
for PR 18334 at commit
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/18334
LGTM, let's resolve the conflict
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18334
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18334
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78389/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18334
**[Test build #78389 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78389/testReport)**
for PR 18334 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18334
**[Test build #78389 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78389/testReport)**
for PR 18334 at commit
Github user wzhfy commented on the issue:
https://github.com/apache/spark/pull/18334
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so,
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18334
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78370/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18334
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18334
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78369/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18334
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18334
**[Test build #78370 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78370/testReport)**
for PR 18334 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18334
**[Test build #78369 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78369/testReport)**
for PR 18334 at commit
Github user wzhfy commented on the issue:
https://github.com/apache/spark/pull/18334
@cloud-fan @gatorsmile I made the following changes:
- add a config to trigger stats update and set it false by default.
- update stats after add partition command, by adding the total size of
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/18334
Listing the files of a partitioned table in the cloud is expensive when the
number of files is large.
---
If your project is set up for it, you can reply to this email and have your
reply
Github user wzhfy commented on the issue:
https://github.com/apache/spark/pull/18334
OK since we'll remove rows and column stats in these commands, maybe it's
better to set it false.
---
If your project is set up for it, you can reply to this email and have your
reply appear on
Github user wzhfy commented on the issue:
https://github.com/apache/spark/pull/18334
Yea I'll add a config for this. But how about set it true by default? I
think usually the overhead of getting the file sizes is negligible.
---
If your project is set up for it, you can reply to
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/18334
+1 to provide a flag to automatically trigger the stats updates. We cat set
it false by default to not surprise users
---
If your project is set up for it, you can reply to this email and have
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/18334
These commands will automatically trigger the stats updates, which could be
expensive. Another way is to simply set it to zero or mark it unreliable? Can
we provide a SQLConf conf for this?
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/18334
How about add a partition?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18334
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18334
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78214/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18334
**[Test build #78214 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78214/testReport)**
for PR 18334 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18334
**[Test build #78214 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78214/testReport)**
for PR 18334 at commit
Github user wzhfy commented on the issue:
https://github.com/apache/spark/pull/18334
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18334
Build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18334
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78202/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18334
**[Test build #78202 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78202/testReport)**
for PR 18334 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18334
**[Test build #78202 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78202/testReport)**
for PR 18334 at commit
Github user wzhfy commented on the issue:
https://github.com/apache/spark/pull/18334
cc @cloud-fan @gatorsmile
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
66 matches
Mail list logo
