Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21919
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99658/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21919
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional comma
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21919
**[Test build #99658 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99658/testReport)**
for PR 21919 at commit
[`43fae6a`](https://github.com/apache/spark/commit/4
Github user vackosar commented on the issue:
https://github.com/apache/spark/pull/21919
@tdas, @gatorsmile and @cloud-fan, just resolved conflicts. Are you happy
to merge or any suggestions?
Please respond such that I can either merge or close this PR.
---
--
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21919
**[Test build #99658 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99658/testReport)**
for PR 21919 at commit
[`43fae6a`](https://github.com/apache/spark/commit/43
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21919
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional comma
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21919
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98468/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21919
**[Test build #98468 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98468/testReport)**
for PR 21919 at commit
[`3dc69bf`](https://github.com/apache/spark/commit/3
Github user vackosar commented on the issue:
https://github.com/apache/spark/pull/21919
@tdas, @gatorsmile and @cloud-fan, just resolved conflicts. Are you happy
to merge or any suggestions?
---
-
To unsubscribe, e-
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21919
**[Test build #98468 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98468/testReport)**
for PR 21919 at commit
[`3dc69bf`](https://github.com/apache/spark/commit/3d
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21919
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional comma
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21919
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98398/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21919
**[Test build #98398 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98398/testReport)**
for PR 21919 at commit
[`cd07a53`](https://github.com/apache/spark/commit/c
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21919
**[Test build #98398 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98398/testReport)**
for PR 21919 at commit
[`cd07a53`](https://github.com/apache/spark/commit/cd
Github user vackosar commented on the issue:
https://github.com/apache/spark/pull/21919
@cloud-fan apart from conflicts are you happy to merge?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user arunmahadevan commented on the issue:
https://github.com/apache/spark/pull/21919
LGTM overall except one minor comment.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional co
Github user vackosar commented on the issue:
https://github.com/apache/spark/pull/21919
@cloud-fan happy to merge?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: r
Github user jose-torres commented on the issue:
https://github.com/apache/spark/pull/21919
Sure, but I'm not a committer so I can't make that happen. @cloud-fan
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spar
Github user vackosar commented on the issue:
https://github.com/apache/spark/pull/21919
@jose-torres are you happy to merge?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands,
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21919
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional comma
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21919
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94751/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21919
**[Test build #94751 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94751/testReport)**
for PR 21919 at commit
[`656b503`](https://github.com/apache/spark/commit/6
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21919
**[Test build #94751 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94751/testReport)**
for PR 21919 at commit
[`656b503`](https://github.com/apache/spark/commit/65
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21919
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional comma
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21919
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94745/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21919
**[Test build #94745 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94745/testReport)**
for PR 21919 at commit
[`1a7095e`](https://github.com/apache/spark/commit/1
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21919
**[Test build #94745 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94745/testReport)**
for PR 21919 at commit
[`1a7095e`](https://github.com/apache/spark/commit/1a
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21919
**[Test build #94744 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94744/testReport)**
for PR 21919 at commit
[`507a422`](https://github.com/apache/spark/commit/5
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21919
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94744/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21919
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional comma
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21919
**[Test build #94744 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94744/testReport)**
for PR 21919 at commit
[`507a422`](https://github.com/apache/spark/commit/50
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21919
**[Test build #94743 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94743/testReport)**
for PR 21919 at commit
[`6e85739`](https://github.com/apache/spark/commit/6
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21919
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional comma
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21919
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94743/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21919
**[Test build #94743 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94743/testReport)**
for PR 21919 at commit
[`6e85739`](https://github.com/apache/spark/commit/6e
Github user jose-torres commented on the issue:
https://github.com/apache/spark/pull/21919
No more suggestions, the PR looks fine to me.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For addition
Github user vackosar commented on the issue:
https://github.com/apache/spark/pull/21919
@jose-torres @cloud-fan do you have any other structure and functionality
suggestions for the PR now? Or can I focus on finalizing the work and getting
it merged?
---
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21919
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94311/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21919
Build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21919
**[Test build #94311 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94311/testReport)**
for PR 21919 at commit
[`80d698d`](https://github.com/apache/spark/commit/8
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21919
**[Test build #94311 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94311/testReport)**
for PR 21919 at commit
[`80d698d`](https://github.com/apache/spark/commit/80
Github user vackosar commented on the issue:
https://github.com/apache/spark/pull/21919
@jose-torres @zsxwing I will exclude SinkProgress constructor from binary
compatibility check as this object is constructed internally by Spark. That
will remove current MiMa test failure.
---
-
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21919
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94217/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21919
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional comma
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21919
**[Test build #94217 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94217/testReport)**
for PR 21919 at commit
[`fde6053`](https://github.com/apache/spark/commit/f
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21919
**[Test build #94217 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94217/testReport)**
for PR 21919 at commit
[`fde6053`](https://github.com/apache/spark/commit/fd
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21919
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional comma
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21919
**[Test build #94168 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94168/testReport)**
for PR 21919 at commit
[`d3a00d4`](https://github.com/apache/spark/commit/d
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21919
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94168/
Test FAILed.
---
Github user vackosar commented on the issue:
https://github.com/apache/spark/pull/21919
@jose-torres I removed use of commit to report the row count. Would you
have a look?
---
-
To unsubscribe, e-mail: reviews-unsu
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21919
**[Test build #94168 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94168/testReport)**
for PR 21919 at commit
[`d3a00d4`](https://github.com/apache/spark/commit/d3
Github user arunmahadevan commented on the issue:
https://github.com/apache/spark/pull/21919
`numOutputRows` makes sense for all sinks, but I agree the counting should
be done at the framework and not by individual sinks. For metrics that does not
apply to all sinks, they could report
Github user jose-torres commented on the issue:
https://github.com/apache/spark/pull/21919
If the individual connectors aren't doing the counting, I don't see a good
reason to put the data inside WriterCommitMessage instead of just leaving
StreamWriterCommitProgress as its own separat
Github user vackosar commented on the issue:
https://github.com/apache/spark/pull/21919
@jose-torres I haven't thought about this. Let me investigate bit more.
Shall we return to this PR? Do you agree with extending WriterCommitMessage
and using in DataWritingSparkTask#run to
Github user jose-torres commented on the issue:
https://github.com/apache/spark/pull/21919
I don't think so. The offsets for the file source need to be consumer
owned, because they need to work with files that were generated outside of
Spark.
---
---
Github user vackosar commented on the issue:
https://github.com/apache/spark/pull/21919
Yes, I was hoping to improve that eg using filename as offset or other non
consumer-owned approach, but that would be rather long term. Do you think it is
solvable?
---
-
Github user jose-torres commented on the issue:
https://github.com/apache/spark/pull/21919
For file streams, the offsets are just indices into a log the source keeps
of which files it's seen. So a file sink doesn't have any access to those
offsets.
---
-
Github user vackosar commented on the issue:
https://github.com/apache/spark/pull/21919
@jose-torres why it wouldnt make sense? According to the documentation all
SS sources have offsets, but not all sinks can also be SS sources e.g. ForEach
doesnt have offsets in general. So usually
Github user jose-torres commented on the issue:
https://github.com/apache/spark/pull/21919
Minimum and maximum offset in the sink wouldn't make sense for most
sources. There aren't any meaningful values to report for e.g. writing out
Parquet files. It'd make sense to put them inside j
Github user vackosar commented on the issue:
https://github.com/apache/spark/pull/21919
@jose-torres thx for good point. The reason for placing this into
WriterCommitMessage is to set a standard information that should passed at the
commit time.
But I agree that row counting
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21919
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93960/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21919
**[Test build #93960 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93960/testReport)**
for PR 21919 at commit
[`399562e`](https://github.com/apache/spark/commit/3
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21919
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional comma
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21919
**[Test build #93960 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93960/testReport)**
for PR 21919 at commit
[`399562e`](https://github.com/apache/spark/commit/39
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21919
ok to test
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@s
Github user jose-torres commented on the issue:
https://github.com/apache/spark/pull/21919
Sure.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21919
@jose-torres, is it okay to trigger the test?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For addition
Github user jose-torres commented on the issue:
https://github.com/apache/spark/pull/21919
I like the idea of doing this, but I don't think it really belongs as part
of the WriterCommitMessage interface. Every connector shouldn't have to
independently count its rows; the execution fra
Github user vackosar commented on the issue:
https://github.com/apache/spark/pull/21919
@tdas @zsxwing @jose-torres @jerryshao @arunmahadevan @HyukjinKwon, please
help with the review and merge.
---
-
To unsubscribe
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21919
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21919
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user holdensmagicalunicorn commented on the issue:
https://github.com/apache/spark/pull/21919
@vackosar, thanks! I am a bot who has found some folks who might be able to
help with the review:@tdas, @gatorsmile and @cloud-fan
---
---
72 matches
Mail list logo
