Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/23207
thanks, merging to master!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23207
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99825/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23207
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23207
**[Test build #99825 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99825/testReport)**
for PR 23207 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23207
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23207
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99817/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23207
**[Test build #99817 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99817/testReport)**
for PR 23207 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23207
**[Test build #99825 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99825/testReport)**
for PR 23207 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23207
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23207
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23207
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23207
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23207
**[Test build #99817 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99817/testReport)**
for PR 23207 at commit
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/23207
retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23207
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99805/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23207
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23207
**[Test build #99805 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99805/testReport)**
for PR 23207 at commit
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/23207
```
the code looks much cleaner now!
```
Sorry for the original rush and code, I should and will pay more attention
on coding clean and more discussion on optional implementation.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23207
**[Test build #99805 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99805/testReport)**
for PR 23207 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23207
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23207
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/23207
the code looks much cleaner now!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands,
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23207
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99782/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23207
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23207
**[Test build #99782 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99782/testReport)**
for PR 23207 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23207
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23207
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/23207
```
Can we put the above in a closure and pass it into shuffle dependency? Then
in SQL we just put the above in SQL using custom metrics.
```
Yea, the commit of a780b70 achieve this
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23207
**[Test build #99782 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99782/testReport)**
for PR 23207 at commit
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/23207
```var writer: ShuffleWriter[Any, Any] = null
try {
val manager = SparkEnv.get.shuffleManager
writer = manager.getWriter[Any, Any](
dep.shuffleHandle,
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23207
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99752/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23207
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23207
**[Test build #99752 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99752/testReport)**
for PR 23207 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23207
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23207
**[Test build #99752 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99752/testReport)**
for PR 23207 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23207
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/23207
```
can you separate the prs to rename read side metric and the write side
change?
```
No problem, next commit will revert the changes of rename read side.
---
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/23207
@xuanyuanking can you separate the prs to rename read side metric and the
write side change?
---
-
To unsubscribe, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23207
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23207
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99736/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23207
**[Test build #99736 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99736/testReport)**
for PR 23207 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23207
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99722/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23207
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23207
**[Test build #99722 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99722/testReport)**
for PR 23207 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23207
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23207
**[Test build #99736 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99736/testReport)**
for PR 23207 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23207
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23207
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23207
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23207
**[Test build #99722 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99722/testReport)**
for PR 23207 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23207
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23207
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99676/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23207
**[Test build #99676 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99676/testReport)**
for PR 23207 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23207
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23207
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99677/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23207
**[Test build #99677 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99677/testReport)**
for PR 23207 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23207
**[Test build #99677 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99677/testReport)**
for PR 23207 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23207
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23207
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23207
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23207
**[Test build #99676 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99676/testReport)**
for PR 23207 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23207
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/23207
Thanks for your reply Wenchen, there's a sketch doc assigned in
JIRA:https://docs.google.com/document/d/1DX0gLkpk_NCE5MwI1_m4gnA2rLdjDkynZ02u2VWDR-8/edit
```
IMO shuffle write
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/23207
Can you share some ideas about it? IMO shuffle write metrics is hard, as an
RDD can have shuffle dependencies with multiple upstream RDDs. That said, in
general the shuffle write metrics should
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/23207
cc @cloud-fan @gatorsmile @rxin
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23207
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23207
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99643/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23207
**[Test build #99643 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99643/testReport)**
for PR 23207 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23207
**[Test build #99643 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99643/testReport)**
for PR 23207 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23207
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23207
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/23207
retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23207
**[Test build #99617 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99617/testReport)**
for PR 23207 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23207
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23207
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99617/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23207
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23207
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99616/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23207
**[Test build #99616 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99616/testReport)**
for PR 23207 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23207
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23207
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23207
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23207
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23207
**[Test build #99616 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99616/testReport)**
for PR 23207 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23207
**[Test build #99617 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99617/testReport)**
for PR 23207 at commit
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/23207
@SparkQA test this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands,
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/23207
test this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/23207
@SparkQA retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands,
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/23207
@AmplabJenkins retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/23207
@AmplabJenkins test this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/23207
@AmplabJenkins
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/23207
retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/23207
test this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
92 matches
Mail list logo
