[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[zsxwing]

Github user zsxwing commented on the issue:

https://github.com/apache/spark/pull/21721
  
FYI, I submitted #22334 to revert #21819 and #21721.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[rxin]

Github user rxin commented on the issue:

https://github.com/apache/spark/pull/21721
  
BTW I think this is probably SPIP-worthy. At the very least we should write 
a design doc on this, similar to the other docs for dsv2 sub-components. We 
should really think about whether it'd be possible to unify the three modes 
(batch, microbatch streaming, CP).



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[rxin]

Github user rxin commented on the issue:

https://github.com/apache/spark/pull/21721
  
Given the uncertainty about how this works across batch, streaming, and CP, 
and given we are still flushing out the main APIs, I think we should revert 
this, and revisit when the main APIs are done.

In general for API design, it is best to flush out the big skeletons first, 
and then work on filling the gaps. Think about building a house. You build the 
frame, and put the stud in, the walls, and then do the final finish. You don't 
start by putting plumbing fixtures in one room when you are still moving the 
main plumbing lines.



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[HeartSaVioR]

Github user HeartSaVioR commented on the issue:

https://github.com/apache/spark/pull/21721
  
I spent more hours to take a look at how SQL UI can update the metrics 
information before task ends, and now I guess I may understand what was the 
concern from @cloud-fan here.

This is different from how we allow custom metrics in StateStore. Every SQL 
metrics even custom metrics in StateStore are accumulators, which are taken 
care of executor heartbeat (Honestly I didn't notice it. My bad) and UI updates 
these information. Custom metrics in StateStore is only updated when state 
operation is going to be finished for each partition, but they are exposed to 
SQL UI anyway and gets updated dynamically in the UI (I meant the values can be 
updated even for running batch).

With StreamingQueryProgress, we are also exposing information which are 
only calculated when they're needed, and now it is when finishTrigger is 
called, so mostly batch ends. Custom metrics in this patch placed here: they're 
additional information for StreamingQueryProgress, hence intentional to be 
updated per batch. They're not actually SQL metrics, but its name would lead 
someone thinking why it doesn't follow SQL metrics. Maybe the name matters?

So there're two desires to add custom information:

1. metrics to be updated in every heartbeat: they will be exposed to SQL 
UI, and also can be collected and added to StreamingQueryProgress like custom 
metrics in StateStore.
2. information to be updated in each batch: they will be exposed to only 
StreamingQueryProgress.

And the target of the patch is latter.

But we know 2 is only applied to micro-batch, and current 
StreamingQueryProgress is not suitable for continuous mode because of these 
reasons: 1. Unless we stop processing or snapshot metrics once epoch ends, 
metrics can't be correct for specific epoch. 2. Showing the information for 
latest epoch (which all partitions finished) no longer represents the most 
recent. 3. Some metrics are expected to be reset per batch, whereas it doesn't 
happen in continuous mode. If we reset metrics per epoch, metrics in SQL tab in 
UI will be really looking odd (because it just shows current state of metrics, 
not bound to epoch).

So IMHO it's likely that StreamingQueryProgress will not be available for 
continuous mode even afterwards (not only for custom metrics), and we may want 
to rely on running SQL metrics. That's actually what other streaming frameworks 
are providing metrics as of now, but they are also showing these metrics as 
aggregated values in time window or even time-series. Spark doesn't need to 
have such feature for batch and micro-batch, but in continuous mode, without 
that feature these SQL metrics will be really hard to see after long run (say 1 
month). That's the hard thing when we want to make modes being transparent: the 
requirements of metrics for batch/micro-batch and continuous mode are just 
different, and metrics may not be only issue.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[HeartSaVioR]

Github user HeartSaVioR commented on the issue:

https://github.com/apache/spark/pull/21721
  
If batch query also leverages AccumulatorV2 for metrics, IMHO it might not 
need to redesign metrics API start from scratch. For batch and micro-batch the 
metrics API work without any concerns (it is getting requests for improvement 
though), and for continuous mode the metrics just don't work because task never 
finishes.

The change in metrics affects both query status as well as SQL tab in UI. I 
haven't concerned too deeply with metrics on continuous mode so not sure about 
current state of UI and the ideal shape of UI, so will spend time to play with. 
My 2 cents, once we have existing metrics work well, we could find out some 
ways to let current metrics work well with continuous mode, to not break other 
things as well.

One thing I would like to ask to ourselves is, would we treat epoch id as 
batch id? For checkpointing we already did it, and in some streaming framework 
they represent `stream between epochs` as `logical batch` which makes sense to 
me. If we deal with watermark we are likely to update watermark per epoch, as 
well as dealing with state, and if my understanding is correct epoch id looks 
like just an alias of batch id.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/21721
  
Thank you @rxin for your time and efforts.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[rxin]

Github user rxin commented on the issue:

https://github.com/apache/spark/pull/21721
  
I will take a look at this tomorrow, since I’m already looking at data
source apis myself. Can provide opinion after another look on whether we
should keep it unstable or revert.

On Fri, Aug 31, 2018 at 12:07 AM Hyukjin Kwon 
wrote:

> So .. @cloud-fan , and @rxin
> , how about this:
>
>1. Mark this as Unstable for now - which means we likely happen to
>change this if we face a design issue.
>2. Write a design doc to cover up continuous and batch and find out
>better way (or check if the current way works).
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> , or 
mute
> the thread
> 

> .
>



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/21721
  
So .. @cloud-fan, and @rxin, how about this:

1. Mark this as `Unstable` for now - which means we likely happen to change 
this if we face a design issue.
2. Write a design doc to cover up continuous and batch and find out better 
way (or check if the current way works).




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/21721
  
Can someone write a design doc for the metrics support? I think this is an 
important feature for data source v2 and we need to be careful here. The design 
doc should explain how custom metrics fit in the abstraction of data source v2 
API, how the metrics API would look like for batch, micro-batch and continuous 
(I feel metrics is also important for batch sources), and how the sources 
report metrics physically (via task complete event? via heartbeat? via RPC?).

@rxin just sent an email to the dev list about the data source v2 API 
abstraction, it would be great if you guys can kick it and talk about the 
metrics support.

It's very likely that the custom metrics API would be replaced by something 
totally different after we finish the design. I don't think we should rush into 
something that works but not well designed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[HeartSaVioR]

Github user HeartSaVioR commented on the issue:

https://github.com/apache/spark/pull/21721
  
I skimmed about how AccumulatorV2 works, and looks like the values in a 
task are reported along with CompletionEvent which is triggered when a task 
ends. Then in continuous mode driver even doesn't have updated metrics. It 
should not couple with lifecycle of task.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[arunmahadevan]

Github user arunmahadevan commented on the issue:

https://github.com/apache/spark/pull/21721
  
>It seems like its life cycle should be bound to an epoch, but 
unfortunately we don't have such an interface in continuous streaming to 
represent an epoch. Is it possible that we may end up with 2 sets of custom 
metrics APIs for micro-batch and continuous?

@cloud-fan we could still report progress at the end of each epoch (e.g. 
[here](https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousExecution.scala#L231)
 and via the EpochCordinator). There need not be separate interfaces for the 
progress or the custom metrics, just the mechanisms could be different.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[HeartSaVioR]

Github user HeartSaVioR commented on the issue:

https://github.com/apache/spark/pull/21721
  
My 2 cents, the root reason is the lifecycle of reporting query progress is 
tied to `finishTrigger` and we read updated metrics from executed plan which 
continuous mode doesn't have both `finishTrigger` as well as finished plan to 
be executed.

I'm not aware of how/when updated information of nodes of physical plan are 
transmitted from executor to the driver, but we should avoid using executed 
plan as a source to read information, and find alternative to be compatible 
between micro-batch and continuous mode. It doesn't apply only metrics but also 
watermarks.

I'm not sure it is viable, but It could be via RPC or whatever once we can 
aggregate the information from driver. Then each operators can send information 
on driver directly and driver can aggregate them and utilize once a batch or an 
epoch is finished.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[arunmahadevan]

Github user arunmahadevan commented on the issue:

https://github.com/apache/spark/pull/21721
  
I created a follow up PR to move CustomMetrics (and a few other streaming 
specific interfaces in that package) to 'streaming' and mark the interfaces as 
Unstable here - https://github.com/apache/spark/pull/22299


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/21721
  
@arunmahadevan, feel free to pick up the commits in my PR in your followup 
if they have to be changed. I will close mine.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[arunmahadevan]

Github user arunmahadevan commented on the issue:

https://github.com/apache/spark/pull/21721
  
@rxin its for streaming sources and sinks as explained in the [doc](

https://github.com/apache/spark/blob/master/sql/core/src/main/java/org/apache/spark/sql/sources/v2/CustomMetrics.java#L23)

It had to be shared between classes in reader.streaming and 
writer.streaming, so was added in the parent package (similar to other 
streaming specific classes that exists here like 
[StreamingWriteSupportProvider.java 
](https://github.com/apache/spark/blob/master/sql/core/src/main/java/org/apache/spark/sql/sources/v2/StreamingWriteSupportProvider.java)

[MicroBatchReadSupportProvider.java](https://github.com/apache/spark/blob/master/sql/core/src/main/java/org/apache/spark/sql/sources/v2/MicroBatchReadSupportProvider.java))

we could move all of it to a streaming package.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[rxin]

Github user rxin commented on the issue:

https://github.com/apache/spark/pull/21721
  
I'm confused by this api. Is this for streaming only? If yes, why are they 
not in the stream package? If not, I only found streaming implementation. Maybe 
I missed it.



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[rxin]

Github user rxin commented on the issue:

https://github.com/apache/spark/pull/21721
  
Stuff like this merits api discussions. Not just implementation changes ...



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/21721
  
I actually thought those all of them are part of DataSource V2. Why are we 
fine with changing those interfaces but not okay with this one and we consider 
reverting it?

Other things should be clarified if there are some concerns, yea of course. 
In this case, switching it to `Unstable` looks alleviating the concerns listed 
here enough.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/21721
  
Note that, data source v2 API is not stable yet and we may even change the 
abstraction of the APIs. The design of custom metrics may affect the design of 
the streaming source APIs.

I had a hard time to figure out the life cycle of custom metrics. It seems 
like its life cycle should be bound to an epoch, but unfortunately we don't 
have such an interface in continuous streaming to represent an epoch. Is it 
possible that we may end up with 2 sets of custom metrics APIs for micro-batch 
and continuous? The documentation added in this PR is not clear about this.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[arunmahadevan]

Github user arunmahadevan commented on the issue:

https://github.com/apache/spark/pull/21721
  
@HyukjinKwon yes we can mark it unstable. Like I mentioned multiple times 
in previous comments the traits added here like CustomMetrics, 
SupportsCustomReaderMetrics etc have nothing specific to micro batch or 
continuous mode and un-affected when we finally start reporting progress for 
continuous mode. The way to collect and report metrics in continuous mode needs 
to be figured out and I think should be discussed in respective JIRAs.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/21721
  
Can we remove and replace this to another one when we are clear on how to 
deal with continuous mode?

I merged this after having sufficient talks with @HeartSaVioR. The JIRA 
(https://issues.apache.org/jira/browse/SPARK-23887) was by @jose-torres who 
left a sign-off here.

BTW, I hope those discussions should better be made in those PRs or JIRAs 
first. Is this just because some concerns were found later? it looked just good 
and many positive feedbacks but just looks held off for no reasons from my 
point of view.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[arunmahadevan]

Github user arunmahadevan commented on the issue:

https://github.com/apache/spark/pull/21721
  
There are many unknowns to be figured out for continuous mode. Though the 
way to capture the metrics would be different for continuous execution, the 
interface of whats reported is not expected to change. Given that we already 
report progress for micro-batch and as a user of spark the changes in the patch 
are quite useful to report custom metrics for what works right now and since it 
does not impact other parts of DataSourceV2 apis (only the sources that wants 
to report custom metrics would add the traits) IMO, we can keep this and 
continue to investigate in a time-bound manner of how to capture metrics for 
continuous mode.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[HeartSaVioR]

Github user HeartSaVioR commented on the issue:

https://github.com/apache/spark/pull/21721
  
Unfortunate thing is that continuous mode allows different epochs between 
partitions, hence query progress just doesn't fit so hard to address 
SPARK-23887 for now. My 2 cents is continuous mode also should sync epoch in 
query so all partitions deal with same epoch. When we deal with aggregation it 
should be essential.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[HeartSaVioR]

Github user HeartSaVioR commented on the issue:

https://github.com/apache/spark/pull/21721
  
@zsxwing @gatorsmile @cloud-fan 
As you all know, IMHO, the thing is that continuous mode doesn't support 
StreamingQueryProgress for now. We allowed these incomplete state because 
continuous mode is in experiment with filing follow-up issue SPARK-23887, and 
the issue has been open for 3 months. If we would like to take it seriously, 
SPARK-23887 is better to be marked as a "Blocker" and we should deal with the 
issue sooner than later. Otherwise it doesn't sound good if continuous mode 
blocks others.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[zsxwing]

Github user zsxwing commented on the issue:

https://github.com/apache/spark/pull/21721
  
@arunmahadevan yeah, it's better to figure out the solution for continuous 
mode as well. As you mentioned, the current SQL metrics are not updated unless 
the task completes, so we may need to add new APIs to support reporting metrics 
for continuous mode. It would be great that there will be a consistent API for 
all modes. Let's step back and think about it.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[arunmahadevan]

Github user arunmahadevan commented on the issue:

https://github.com/apache/spark/pull/21721
  
The CustomMetrics are traits which can be mixed in if necessary. (see 
https://github.com/apache/spark/pull/21721#issuecomment-403878383) and does not 
affect any other API as such. When query progress is supported for continuous 
mode, changes can be made if necessary. I can further investigate on how to 
report query progress for continuous mode.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[arunmahadevan]

Github user arunmahadevan commented on the issue:

https://github.com/apache/spark/pull/21721
  
@zsxwing @gatorsmile , this PR does not add new APIs as such. It builds on 
the existing StreamingQueryProgress and adds custom metrics to it. 
StreamingQueryProgress as such is not reported for continuous mode. When its 
reported this would be part of that. Are you proposing to not report 
StreamingQueryProgress at all for micro-batch until things are figured out for 
continuous mode.? Otherwise I don't see how adding this would break things. 

cc @jose-torres 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/21721
  
I also agree with @zsxwing . We should revert these two PRs.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[zsxwing]

Github user zsxwing commented on the issue:

https://github.com/apache/spark/pull/21721
  
It’s better to not release such APIs without thinking about how to 
support continuous queries, since it may need to change APIs, which should be 
avoided if possible. I propose to revert this PR. It would be great that there 
is a design doc for streaming source metrics APIs to discuss how to support all 
modes before committing.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[arunmahadevan]

Github user arunmahadevan commented on the issue:

https://github.com/apache/spark/pull/21721
  
For continuous queries, the progress could still be reported by posting 
QueryProgressEvent to the listener for each epoch (instead of micro-batch). The 
`StreamingQueryProgress` also could mostly be the same.

I am not clear on how the continuous query would get the metric updates. We 
may need some mechanism to post metric updates while the query continues to 
run. Right now the SQL metrics relies on accumulator and the accumulators might 
not be updated unless the task completes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/21721
  
Also ping @rxin @marmbrus . Another related PR is 
https://github.com/apache/spark/pull/21819.  


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/21721
  
Since I'm continuously working on data source v2 API, this gets my 
attention. Do we have a story for the metrics in data source v2 streaming API? 
It's weird to me that we add public APIs that only work for micro-batch.

For streaming API, the abstraction is that, we have a logical scan for a 
streaming source in a query(to keep query specific states like offsets), and a 
physical scan to do the actual job, for each micro-batch or for the entire 
continuous query(if `needsReconfigure` is false).

Where does the metrics fit in with the abstraction? It's ok that it only 
works for micro-batch now, but we must have a clear plan that we can and will 
make it work for continuous.

cc @tdas @zsxwing @rdblue 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/21721
  
Merged to master.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21721
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21721
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94320/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21721
  
**[Test build #94320 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94320/testReport)**
 for PR 21721 at commit 
[`01e8451`](https://github.com/apache/spark/commit/01e8451503e84ad471cd93373d2f2d6eaf43251c).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[arunmahadevan]

Github user arunmahadevan commented on the issue:

https://github.com/apache/spark/pull/21721
  
@HyukjinKwon , the master code changed and I had to rebase and fix issues. 
Can you take it forward ? There seems to be unrelated test failures in Kafka 
0.10 integration suite.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21721
  
**[Test build #94320 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94320/testReport)**
 for PR 21721 at commit 
[`01e8451`](https://github.com/apache/spark/commit/01e8451503e84ad471cd93373d2f2d6eaf43251c).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21721
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94200/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21721
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21721
  
**[Test build #94200 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94200/testReport)**
 for PR 21721 at commit 
[`1775c2a`](https://github.com/apache/spark/commit/1775c2a1db2bf790ddf1cad0113c7ead2409ba65).
 * This patch **fails due to an unknown error code, -9**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21721
  
**[Test build #94200 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94200/testReport)**
 for PR 21721 at commit 
[`1775c2a`](https://github.com/apache/spark/commit/1775c2a1db2bf790ddf1cad0113c7ead2409ba65).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/21721
  
retest this please


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21721
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21721
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94147/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21721
  
**[Test build #94147 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94147/testReport)**
 for PR 21721 at commit 
[`1775c2a`](https://github.com/apache/spark/commit/1775c2a1db2bf790ddf1cad0113c7ead2409ba65).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21721
  
**[Test build #94147 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94147/testReport)**
 for PR 21721 at commit 
[`1775c2a`](https://github.com/apache/spark/commit/1775c2a1db2bf790ddf1cad0113c7ead2409ba65).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[arunmahadevan]

Github user arunmahadevan commented on the issue:

https://github.com/apache/spark/pull/21721
  
retest this please


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21721
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94121/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21721
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21721
  
**[Test build #94121 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94121/testReport)**
 for PR 21721 at commit 
[`1775c2a`](https://github.com/apache/spark/commit/1775c2a1db2bf790ddf1cad0113c7ead2409ba65).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21721
  
**[Test build #94121 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94121/testReport)**
 for PR 21721 at commit 
[`1775c2a`](https://github.com/apache/spark/commit/1775c2a1db2bf790ddf1cad0113c7ead2409ba65).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/21721
  
retest this please


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21721
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21721
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94097/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21721
  
**[Test build #94097 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94097/testReport)**
 for PR 21721 at commit 
[`1775c2a`](https://github.com/apache/spark/commit/1775c2a1db2bf790ddf1cad0113c7ead2409ba65).
 * This patch **fails due to an unknown error code, -9**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21721
  
**[Test build #94097 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94097/testReport)**
 for PR 21721 at commit 
[`1775c2a`](https://github.com/apache/spark/commit/1775c2a1db2bf790ddf1cad0113c7ead2409ba65).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/21721
  
retest this please


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/21721
  
Yup, looks the resent Kafka upgrade has an issue.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21721
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21721
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94038/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21721
  
**[Test build #94038 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94038/testReport)**
 for PR 21721 at commit 
[`1775c2a`](https://github.com/apache/spark/commit/1775c2a1db2bf790ddf1cad0113c7ead2409ba65).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[arunmahadevan]

Github user arunmahadevan commented on the issue:

https://github.com/apache/spark/pull/21721
  
The tests keeps failing and looks unrelated. @HyukjinKwon Let me know if 
you think theres something I should look into.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21721
  
**[Test build #94038 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94038/testReport)**
 for PR 21721 at commit 
[`1775c2a`](https://github.com/apache/spark/commit/1775c2a1db2bf790ddf1cad0113c7ead2409ba65).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/21721
  
retest this please


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21721
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94008/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21721
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21721
  
**[Test build #94008 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94008/testReport)**
 for PR 21721 at commit 
[`1775c2a`](https://github.com/apache/spark/commit/1775c2a1db2bf790ddf1cad0113c7ead2409ba65).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21721
  
**[Test build #94008 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94008/testReport)**
 for PR 21721 at commit 
[`1775c2a`](https://github.com/apache/spark/commit/1775c2a1db2bf790ddf1cad0113c7ead2409ba65).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/21721
  
Looks fine otherwise to me too if the test passes


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21721
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93907/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21721
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21721
  
**[Test build #93907 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93907/testReport)**
 for PR 21721 at commit 
[`3e5d9d8`](https://github.com/apache/spark/commit/3e5d9d8ee78176e68b0775a24886a68d021edafa).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21721
  
**[Test build #93907 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93907/testReport)**
 for PR 21721 at commit 
[`3e5d9d8`](https://github.com/apache/spark/commit/3e5d9d8ee78176e68b0775a24886a68d021edafa).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[arunmahadevan]

Github user arunmahadevan commented on the issue:

https://github.com/apache/spark/pull/21721
  
@HyukjinKwon , have addressed the comments and modified SourceProgress and 
SinkProgress to take String instead of JValue so that this can be easily used 
from Java. Regarding the default value in the ctor, I am not sure if its an 
issue because the object is mostly read only and would be an issue only if the 
user tries to construct it from Java. I have added overloaded ctors anyways. 
Please take a look.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[HeartSaVioR]

Github user HeartSaVioR commented on the issue:

https://github.com/apache/spark/pull/21721
  
Looks like we would we also need to add SourceProgress and SinkProgress 
into mima exclude list.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21721
  
**[Test build #93813 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93813/testReport)**
 for PR 21721 at commit 
[`ef65d51`](https://github.com/apache/spark/commit/ef65d519a5f3e7a2f267a8ae93ea80420793c975).
 * This patch **fails MiMa tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21721
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21721
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93813/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21721
  
**[Test build #93813 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93813/testReport)**
 for PR 21721 at commit 
[`ef65d51`](https://github.com/apache/spark/commit/ef65d519a5f3e7a2f267a8ae93ea80420793c975).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[arunmahadevan]

Github user arunmahadevan commented on the issue:

https://github.com/apache/spark/pull/21721
  
@HeartSaVioR thanks for taking time to review. Addressed the comments, can 
you take a look again?

Regarding the mixin interface, would like to take feedback from others.

@jose-torres @tdas @zsxwing could you take a look at the patch and also 
comment on https://github.com/apache/spark/pull/21721#discussion_r206241038 ? 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21721
  
**[Test build #93793 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93793/testReport)**
 for PR 21721 at commit 
[`04c5b6f`](https://github.com/apache/spark/commit/04c5b6fb24331632dea29730f50b4d8f948510c0).
 * This patch **fails MiMa tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21721
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93793/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21721
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21721
  
**[Test build #93793 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93793/testReport)**
 for PR 21721 at commit 
[`04c5b6f`](https://github.com/apache/spark/commit/04c5b6fb24331632dea29730f50b4d8f948510c0).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[arunmahadevan]

Github user arunmahadevan commented on the issue:

https://github.com/apache/spark/pull/21721
  
@jose-torres, addressed initial comments.
@tdas, can you also take a look when possible ?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21721
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92876/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21721
  
**[Test build #92876 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92876/testReport)**
 for PR 21721 at commit 
[`5e732cb`](https://github.com/apache/spark/commit/5e732cba85a5c2e3ed3f0487c70c1ebe4c20b75d).
 * This patch **fails MiMa tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21721
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21721
  
**[Test build #92876 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92876/testReport)**
 for PR 21721 at commit 
[`5e732cb`](https://github.com/apache/spark/commit/5e732cba85a5c2e3ed3f0487c70c1ebe4c20b75d).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21721
  
**[Test build #92874 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92874/testReport)**
 for PR 21721 at commit 
[`bca054f`](https://github.com/apache/spark/commit/bca054f978406b257bfa4c4010e7655144fc820f).
 * This patch **fails Scala style tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21721
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21721
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92874/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21721
  
**[Test build #92874 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92874/testReport)**
 for PR 21721 at commit 
[`bca054f`](https://github.com/apache/spark/commit/bca054f978406b257bfa4c4010e7655144fc820f).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[arunmahadevan]

Github user arunmahadevan commented on the issue:

https://github.com/apache/spark/pull/21721
  
@jose-torres I have removed the Kafka lag metrics out of this PR and added 
writer metrics and the number of rows in the memory sink  as an example.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21721
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92870/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21721
  
**[Test build #92870 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92870/testReport)**
 for PR 21721 at commit 
[`47d802b`](https://github.com/apache/spark/commit/47d802b010d366fe58ea9632fde69a4331c3d65a).
 * This patch **fails Scala style tests**.
 * This patch **does not merge cleanly**.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21721
  
Build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21721
  
**[Test build #92870 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92870/testReport)**
 for PR 21721 at commit 
[`47d802b`](https://github.com/apache/spark/commit/47d802b010d366fe58ea9632fde69a4331c3d65a).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

[jose-torres]

Github user jose-torres commented on the issue:

https://github.com/apache/spark/pull/21721
  
Looks fine to me with a MemorySink example. I don't think a formal 
discussion is super necessary - the major advantage of the mixin model is to 
let us add things like this without impacting the broader API.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org