[GitHub] zentol commented on a change in pull request #6551: [FLINK-10022][network][metrics] add metrics for input/output buffers
zentol commented on a change in pull request #6551:
[FLINK-10022][network][metrics] add metrics for input/output buffers
URL: https://github.com/apache/flink/pull/6551#discussion_r209965354
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/util/MutableIOMetrics.java
##
@@ -179,11 +179,12 @@ public void writeIOMetricsAsJson(JsonGenerator gen)
throws IOException {
gen.writeObjectFieldStart("metrics");
- Long numBytesIn = this.numBytesInLocal + this.numBytesInRemote;
Review comment:
revert changes in this file
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] zentol commented on a change in pull request #6551: [FLINK-10022][network][metrics] add metrics for input/output buffers
zentol commented on a change in pull request #6551: [FLINK-10022][network][metrics] add metrics for input/output buffers URL: https://github.com/apache/flink/pull/6551#discussion_r209965189 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/IOMetrics.java ## @@ -48,6 +48,7 @@ public IOMetrics(Meter recordsIn, Meter recordsOut, Meter bytesLocalIn, Meter by this.numRecordsInPerSecond = recordsIn.getRate(); this.numRecordsOut = recordsOut.getCount(); this.numRecordsOutPerSecond = recordsOut.getRate(); + Review comment: revert This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] zentol commented on a change in pull request #6551: [FLINK-10022][network][metrics] add metrics for input/output buffers
zentol commented on a change in pull request #6551: [FLINK-10022][network][metrics] add metrics for input/output buffers URL: https://github.com/apache/flink/pull/6551#discussion_r209965405 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/rest/messages/job/metrics/IOMetricsInfo.java ## @@ -82,6 +82,7 @@ public IOMetricsInfo( this.bytesReadComplete = bytesReadComplete; this.bytesWritten = bytesWritten; this.bytesWrittenComplete = bytesWrittenComplete; + Review comment: revert This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] zentol commented on a change in pull request #6551: [FLINK-10022][network][metrics] add metrics for input/output buffers
zentol commented on a change in pull request #6551:
[FLINK-10022][network][metrics] add metrics for input/output buffers
URL: https://github.com/apache/flink/pull/6551#discussion_r209932565
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/IOMetrics.java
##
@@ -43,38 +43,74 @@
protected double numBytesInRemotePerSecond;
protected double numBytesOutPerSecond;
- public IOMetrics(Meter recordsIn, Meter recordsOut, Meter bytesLocalIn,
Meter bytesRemoteIn, Meter bytesOut) {
+ protected long numBuffersInLocal;
+ protected long numBuffersInRemote;
+ protected long numBuffersOut;
+
+ protected double numBuffersInLocalPerSecond;
+ protected double numBuffersInRemotePerSecond;
+ protected double numBuffersOutPerSecond;
+
+ public IOMetrics(
Review comment:
it should be fine to remove them from this class, and revert the changes to
the various handlers.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] zentol commented on a change in pull request #6551: [FLINK-10022][network][metrics] add metrics for input/output buffers
zentol commented on a change in pull request #6551:
[FLINK-10022][network][metrics] add metrics for input/output buffers
URL: https://github.com/apache/flink/pull/6551#discussion_r209905769
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/IOMetrics.java
##
@@ -43,38 +43,74 @@
protected double numBytesInRemotePerSecond;
protected double numBytesOutPerSecond;
- public IOMetrics(Meter recordsIn, Meter recordsOut, Meter bytesLocalIn,
Meter bytesRemoteIn, Meter bytesOut) {
+ protected long numBuffersInLocal;
+ protected long numBuffersInRemote;
+ protected long numBuffersOut;
+
+ protected double numBuffersInLocalPerSecond;
+ protected double numBuffersInRemotePerSecond;
+ protected double numBuffersOutPerSecond;
+
+ public IOMetrics(
+ Meter recordsIn,
+ Meter recordsOut,
+ Meter bytesLocalIn,
+ Meter bytesRemoteIn,
+ Meter bytesOut,
+ Meter buffersLocalIn,
+ Meter buffersRemoteIn,
+ Meter buffersOut) {
this.numRecordsIn = recordsIn.getCount();
this.numRecordsInPerSecond = recordsIn.getRate();
this.numRecordsOut = recordsOut.getCount();
this.numRecordsOutPerSecond = recordsOut.getRate();
+
this.numBytesInLocal = bytesLocalIn.getCount();
this.numBytesInLocalPerSecond = bytesLocalIn.getRate();
this.numBytesInRemote = bytesRemoteIn.getCount();
this.numBytesInRemotePerSecond = bytesRemoteIn.getRate();
this.numBytesOut = bytesOut.getCount();
this.numBytesOutPerSecond = bytesOut.getRate();
+
+ this.numBuffersInLocal = buffersLocalIn.getCount();
+ this.numBuffersInLocalPerSecond = buffersLocalIn.getRate();
+ this.numBuffersInRemote = buffersRemoteIn.getCount();
+ this.numBuffersInRemotePerSecond = buffersRemoteIn.getRate();
+ this.numBuffersOut = buffersOut.getCount();
+ this.numBuffersOutPerSecond = buffersOut.getRate();
}
public IOMetrics(
long numBytesInLocal,
long numBytesInRemote,
long numBytesOut,
+ long numBuffersInLocal,
+ long numBuffersInRemote,
+ long numBuffersOut,
long numRecordsIn,
long numRecordsOut,
double numBytesInLocalPerSecond,
double numBytesInRemotePerSecond,
double numBytesOutPerSecond,
+ double numBuffersInLocalPerSecond,
+ double numBuffersInRemotePerSecond,
+ double numBuffersOutPerSecond,
double numRecordsInPerSecond,
double numRecordsOutPerSecond) {
this.numBytesInLocal = numBytesInLocal;
this.numBytesInRemote = numBytesInRemote;
this.numBytesOut = numBytesOut;
+ this.numBuffersInLocal = numBuffersInLocal;
+ this.numBuffersInRemote = numBuffersInRemote;
+ this.numBuffersOut = numBuffersOut;
this.numRecordsIn = numRecordsIn;
this.numRecordsOut = numRecordsOut;
this.numBytesInLocalPerSecond = numBytesInLocalPerSecond;
this.numBytesInRemotePerSecond = numBytesInRemotePerSecond;
this.numBytesOutPerSecond = numBytesOutPerSecond;
+ this.numBuffersInLocalPerSecond = numBuffersInLocalPerSecond;
Review comment:
the `perSecond` metrics appear to be unused. ( I know you probably just
followed the existing pattern )
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] zentol commented on a change in pull request #6551: [FLINK-10022][network][metrics] add metrics for input/output buffers
zentol commented on a change in pull request #6551:
[FLINK-10022][network][metrics] add metrics for input/output buffers
URL: https://github.com/apache/flink/pull/6551#discussion_r209906457
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/IOMetrics.java
##
@@ -43,38 +43,74 @@
protected double numBytesInRemotePerSecond;
protected double numBytesOutPerSecond;
- public IOMetrics(Meter recordsIn, Meter recordsOut, Meter bytesLocalIn,
Meter bytesRemoteIn, Meter bytesOut) {
+ protected long numBuffersInLocal;
+ protected long numBuffersInRemote;
+ protected long numBuffersOut;
+
+ protected double numBuffersInLocalPerSecond;
+ protected double numBuffersInRemotePerSecond;
+ protected double numBuffersOutPerSecond;
+
+ public IOMetrics(
Review comment:
so this here is really the big question mark for me. So far we only kept
metrics in here that are displayed in the UI, but with this PR we'd break that.
This is mostly done to have accurate displays at the end of the job, which in
particular in the case of batch jobs is important.
The question is, do we truly need 100% accurate buffer metrics here? For
_debugging_ purposes there are always other avenues one can take to get
accurate readings. (delay shutdown of job until metrics are updated)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] zentol commented on a change in pull request #6551: [FLINK-10022][network][metrics] add metrics for input/output buffers
zentol commented on a change in pull request #6551:
[FLINK-10022][network][metrics] add metrics for input/output buffers
URL: https://github.com/apache/flink/pull/6551#discussion_r209904582
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/metrics/groups/TaskIOMetricGroup.java
##
@@ -104,7 +130,6 @@ public Meter getNumBytesInRemoteRateMeter() {
public Meter getNumBytesOutRateMeter() {
return numBytesOutRate;
}
-
Review comment:
?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
