Github user viirya commented on the issue:
https://github.com/apache/spark/pull/15667
@rxin @ericl Thanks!
@snodawn I will address dynamic partition in a follow-up pr.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/15667
Merging in master. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user snodawn commented on the issue:
https://github.com/apache/spark/pull/15667
@viirya @ericl Thanks a lot. The new patch is also passed in my test. And I
sincerely hope dynamic partition could be completed in the future, thanks.
---
If your project is set up for it, you
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15667
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15667
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67853/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15667
**[Test build #67853 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67853/consoleFull)**
for PR 15667 at commit
Github user ericl commented on the issue:
https://github.com/apache/spark/pull/15667
Let's do it in a follow-up.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/15667
@ericl Dynamic partition would be more complicated. Should we do it in this
or in follow-up?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
Github user ericl commented on the issue:
https://github.com/apache/spark/pull/15667
lgtm if tests pass
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15667
**[Test build #67853 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67853/consoleFull)**
for PR 15667 at commit
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/15667
@snodawn Thanks! I will address dynamic partition in next commit.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user snodawn commented on the issue:
https://github.com/apache/spark/pull/15667
@viirya I have tested the newest patch. It performs good in running the
same sql as I ran before.
---
If your project is set up for it, you can reply to this email and have your
reply appear on
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15667
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15667
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67799/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15667
**[Test build #67799 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67799/consoleFull)**
for PR 15667 at commit
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/15667
@snodawn Thanks. I expect it should perform as good as you tested. But
there are few tests failed I fix in newer patch.
---
If your project is set up for it, you can reply to this email and have
Github user snodawn commented on the issue:
https://github.com/apache/spark/pull/15667
Ok, I see. I haven't tested the newest code, I would try it later.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/15667
@snodawn You just said the new patch runs better. Do you use the latest
patch I just updated in about 1 hr ago? Thanks.
---
If your project is set up for it, you can reply to this email and have
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/15667
@snodawn Current fixing does not do anything for dynamic partition. So we
can expect this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user snodawn commented on the issue:
https://github.com/apache/spark/pull/15667
The execution logs in spark show me that, it does the same thing as what it
does before I add the patch, which may be the reason why it runs so slow when
running dynamic insert overwrite statement.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/15667
@snodawn I just updated this with a new commit. Do you use this new patch
to test?
Yeah, current fixing doesn't consider dynamic partition. I would like to
see if we can improve static
Github user snodawn commented on the issue:
https://github.com/apache/spark/pull/15667
@viirya I have tested the new patch, which performs better than expected.
Before patching, it costs about 500~600 seconds, but now it just cost me about
16 seconds to run the same statement. But it
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15667
**[Test build #67799 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67799/consoleFull)**
for PR 15667 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15667
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15667
**[Test build #67785 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67785/consoleFull)**
for PR 15667 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15667
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67785/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15667
**[Test build #67785 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67785/consoleFull)**
for PR 15667 at commit
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/15667
@snodawn Can you try it again? I've updated this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/15667
@snodawn Interesting...I will try to find out it too.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user snodawn commented on the issue:
https://github.com/apache/spark/pull/15667
Sorry, I'm later, for preparing the environment for testing cost me a lot
of time. I have tested the performance before and after the patch. But it seems
to improve a few after patching, where it
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/15667
@ericl I don't have the Hive environment to compare. We need to wait for
issue reporter to verify that.
---
If your project is set up for it, you can reply to this email and have your
reply appear
Github user ericl commented on the issue:
https://github.com/apache/spark/pull/15667
How does the performance look like before / after this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/15667
I checked the failed test. Special characters in the partition path cause
the failure, e..g.,
alter table ppr_test add partition (ds = '12:4');
alter table ppr_test add partition
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/15667
cc @ericl
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15667
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67686/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15667
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15667
**[Test build #67686 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67686/consoleFull)**
for PR 15667 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15667
**[Test build #67686 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67686/consoleFull)**
for PR 15667 at commit
38 matches
Mail list logo
