Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/21721
FYI, I submitted #22334 to revert #21819 and #21721.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/21721
BTW I think this is probably SPIP-worthy. At the very least we should write
a design doc on this, similar to the other docs for dsv2 sub-components. We
should really think about whether it'd be
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/21721
Given the uncertainty about how this works across batch, streaming, and CP,
and given we are still flushing out the main APIs, I think we should revert
this, and revisit when the main APIs are done.
Github user HeartSaVioR commented on the issue:
https://github.com/apache/spark/pull/21721
I spent more hours to take a look at how SQL UI can update the metrics
information before task ends, and now I guess I may understand what was the
concern from @cloud-fan here.
This is
Github user HeartSaVioR commented on the issue:
https://github.com/apache/spark/pull/21721
If batch query also leverages AccumulatorV2 for metrics, IMHO it might not
need to redesign metrics API start from scratch. For batch and micro-batch the
metrics API work without any concerns
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21721
Thank you @rxin for your time and efforts.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/21721
I will take a look at this tomorrow, since Iâm already looking at data
source apis myself. Can provide opinion after another look on whether we
should keep it unstable or revert.
On
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21721
So .. @cloud-fan, and @rxin, how about this:
1. Mark this as `Unstable` for now - which means we likely happen to change
this if we face a design issue.
2. Write a design doc to
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/21721
Can someone write a design doc for the metrics support? I think this is an
important feature for data source v2 and we need to be careful here. The design
doc should explain how custom metrics
Github user HeartSaVioR commented on the issue:
https://github.com/apache/spark/pull/21721
I skimmed about how AccumulatorV2 works, and looks like the values in a
task are reported along with CompletionEvent which is triggered when a task
ends. Then in continuous mode driver even
Github user arunmahadevan commented on the issue:
https://github.com/apache/spark/pull/21721
>It seems like its life cycle should be bound to an epoch, but
unfortunately we don't have such an interface in continuous streaming to
represent an epoch. Is it possible that we may end up
Github user HeartSaVioR commented on the issue:
https://github.com/apache/spark/pull/21721
My 2 cents, the root reason is the lifecycle of reporting query progress is
tied to `finishTrigger` and we read updated metrics from executed plan which
continuous mode doesn't have both
Github user arunmahadevan commented on the issue:
https://github.com/apache/spark/pull/21721
I created a follow up PR to move CustomMetrics (and a few other streaming
specific interfaces in that package) to 'streaming' and mark the interfaces as
Unstable here -
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21721
@arunmahadevan, feel free to pick up the commits in my PR in your followup
if they have to be changed. I will close mine.
---
Github user arunmahadevan commented on the issue:
https://github.com/apache/spark/pull/21721
@rxin its for streaming sources and sinks as explained in the [doc](
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/21721
I'm confused by this api. Is this for streaming only? If yes, why are they
not in the stream package? If not, I only found streaming implementation. Maybe
I missed it.
---
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/21721
Stuff like this merits api discussions. Not just implementation changes ...
---
-
To unsubscribe, e-mail:
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21721
I actually thought those all of them are part of DataSource V2. Why are we
fine with changing those interfaces but not okay with this one and we consider
reverting it?
Other things
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/21721
Note that, data source v2 API is not stable yet and we may even change the
abstraction of the APIs. The design of custom metrics may affect the design of
the streaming source APIs.
I had
Github user arunmahadevan commented on the issue:
https://github.com/apache/spark/pull/21721
@HyukjinKwon yes we can mark it unstable. Like I mentioned multiple times
in previous comments the traits added here like CustomMetrics,
SupportsCustomReaderMetrics etc have nothing specific
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21721
Can we remove and replace this to another one when we are clear on how to
deal with continuous mode?
I merged this after having sufficient talks with @HeartSaVioR. The JIRA
Github user arunmahadevan commented on the issue:
https://github.com/apache/spark/pull/21721
There are many unknowns to be figured out for continuous mode. Though the
way to capture the metrics would be different for continuous execution, the
interface of whats reported is not
Github user HeartSaVioR commented on the issue:
https://github.com/apache/spark/pull/21721
Unfortunate thing is that continuous mode allows different epochs between
partitions, hence query progress just doesn't fit so hard to address
SPARK-23887 for now. My 2 cents is continuous mode
Github user HeartSaVioR commented on the issue:
https://github.com/apache/spark/pull/21721
@zsxwing @gatorsmile @cloud-fan
As you all know, IMHO, the thing is that continuous mode doesn't support
StreamingQueryProgress for now. We allowed these incomplete state because
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/21721
@arunmahadevan yeah, it's better to figure out the solution for continuous
mode as well. As you mentioned, the current SQL metrics are not updated unless
the task completes, so we may need to add
Github user arunmahadevan commented on the issue:
https://github.com/apache/spark/pull/21721
The CustomMetrics are traits which can be mixed in if necessary. (see
https://github.com/apache/spark/pull/21721#issuecomment-403878383) and does not
affect any other API as such. When query
Github user arunmahadevan commented on the issue:
https://github.com/apache/spark/pull/21721
@zsxwing @gatorsmile , this PR does not add new APIs as such. It builds on
the existing StreamingQueryProgress and adds custom metrics to it.
StreamingQueryProgress as such is not reported
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21721
I also agree with @zsxwing . We should revert these two PRs.
---
-
To unsubscribe, e-mail:
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/21721
Itâs better to not release such APIs without thinking about how to
support continuous queries, since it may need to change APIs, which should be
avoided if possible. I propose to revert this PR.
Github user arunmahadevan commented on the issue:
https://github.com/apache/spark/pull/21721
For continuous queries, the progress could still be reported by posting
QueryProgressEvent to the listener for each epoch (instead of micro-batch). The
`StreamingQueryProgress` also could
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21721
Also ping @rxin @marmbrus . Another related PR is
https://github.com/apache/spark/pull/21819.
---
-
To unsubscribe,
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/21721
Since I'm continuously working on data source v2 API, this gets my
attention. Do we have a story for the metrics in data source v2 streaming API?
It's weird to me that we add public APIs that
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21721
Merged to master.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21721
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21721
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94320/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21721
**[Test build #94320 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94320/testReport)**
for PR 21721 at commit
Github user arunmahadevan commented on the issue:
https://github.com/apache/spark/pull/21721
@HyukjinKwon , the master code changed and I had to rebase and fix issues.
Can you take it forward ? There seems to be unrelated test failures in Kafka
0.10 integration suite.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21721
**[Test build #94320 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94320/testReport)**
for PR 21721 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21721
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94200/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21721
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21721
**[Test build #94200 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94200/testReport)**
for PR 21721 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21721
**[Test build #94200 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94200/testReport)**
for PR 21721 at commit
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21721
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21721
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21721
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94147/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21721
**[Test build #94147 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94147/testReport)**
for PR 21721 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21721
**[Test build #94147 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94147/testReport)**
for PR 21721 at commit
Github user arunmahadevan commented on the issue:
https://github.com/apache/spark/pull/21721
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21721
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94121/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21721
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21721
**[Test build #94121 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94121/testReport)**
for PR 21721 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21721
**[Test build #94121 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94121/testReport)**
for PR 21721 at commit
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21721
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21721
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21721
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94097/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21721
**[Test build #94097 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94097/testReport)**
for PR 21721 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21721
**[Test build #94097 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94097/testReport)**
for PR 21721 at commit
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21721
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21721
Yup, looks the resent Kafka upgrade has an issue.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21721
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21721
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94038/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21721
**[Test build #94038 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94038/testReport)**
for PR 21721 at commit
Github user arunmahadevan commented on the issue:
https://github.com/apache/spark/pull/21721
The tests keeps failing and looks unrelated. @HyukjinKwon Let me know if
you think theres something I should look into.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21721
**[Test build #94038 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94038/testReport)**
for PR 21721 at commit
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21721
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21721
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94008/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21721
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21721
**[Test build #94008 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94008/testReport)**
for PR 21721 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21721
**[Test build #94008 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94008/testReport)**
for PR 21721 at commit
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21721
Looks fine otherwise to me too if the test passes
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21721
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93907/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21721
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21721
**[Test build #93907 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93907/testReport)**
for PR 21721 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21721
**[Test build #93907 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93907/testReport)**
for PR 21721 at commit
Github user arunmahadevan commented on the issue:
https://github.com/apache/spark/pull/21721
@HyukjinKwon , have addressed the comments and modified SourceProgress and
SinkProgress to take String instead of JValue so that this can be easily used
from Java. Regarding the default value
Github user HeartSaVioR commented on the issue:
https://github.com/apache/spark/pull/21721
Looks like we would we also need to add SourceProgress and SinkProgress
into mima exclude list.
---
-
To unsubscribe,
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21721
**[Test build #93813 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93813/testReport)**
for PR 21721 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21721
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21721
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93813/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21721
**[Test build #93813 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93813/testReport)**
for PR 21721 at commit
Github user arunmahadevan commented on the issue:
https://github.com/apache/spark/pull/21721
@HeartSaVioR thanks for taking time to review. Addressed the comments, can
you take a look again?
Regarding the mixin interface, would like to take feedback from others.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21721
**[Test build #93793 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93793/testReport)**
for PR 21721 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21721
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93793/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21721
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21721
**[Test build #93793 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93793/testReport)**
for PR 21721 at commit
Github user arunmahadevan commented on the issue:
https://github.com/apache/spark/pull/21721
@jose-torres, addressed initial comments.
@tdas, can you also take a look when possible ?
---
-
To unsubscribe,
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21721
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92876/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21721
**[Test build #92876 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92876/testReport)**
for PR 21721 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21721
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21721
**[Test build #92876 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92876/testReport)**
for PR 21721 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21721
**[Test build #92874 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92874/testReport)**
for PR 21721 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21721
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21721
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92874/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21721
**[Test build #92874 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92874/testReport)**
for PR 21721 at commit
Github user arunmahadevan commented on the issue:
https://github.com/apache/spark/pull/21721
@jose-torres I have removed the Kafka lag metrics out of this PR and added
writer metrics and the number of rows in the memory sink as an example.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21721
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92870/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21721
**[Test build #92870 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92870/testReport)**
for PR 21721 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21721
Build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands,
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21721
**[Test build #92870 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92870/testReport)**
for PR 21721 at commit
Github user jose-torres commented on the issue:
https://github.com/apache/spark/pull/21721
Looks fine to me with a MemorySink example. I don't think a formal
discussion is super necessary - the major advantage of the mixin model is to
let us add things like this without impacting the
1 - 100 of 115 matches
Mail list logo
