CrynetLogistics commented on a change in pull request #17687:
URL: https://github.com/apache/flink/pull/17687#discussion_r743542460
##########
File path:
flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/sink/writer/AsyncSinkWriter.java
##########
@@ -172,6 +229,15 @@ public AsyncSinkWriter(
this.inFlightRequestsCount = 0;
this.bufferedRequestEntriesTotalSizeInBytes = 0;
+
+ this.metrics = context.metricGroup();
+ this.metrics.setCurrentSendTimeGauge(
+ () -> {
+ long time = this.ackTime - this.lastSendTimestamp;
+ return time < 0 ? this.lastSendDuration : time;
Review comment:
It's true, race conditions between the in-flight requests occurs. I've
modified it such that updates to `ackTime` and `lastSendTimestamp` occur in the
same single threaded block now. `lastSendDuration` has been removed since it's
not necessary. Added a test to prove too.
##########
File path:
flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/sink/writer/AsyncSinkWriter.java
##########
@@ -114,9 +134,38 @@
*
* <p>The method is invoked with a set of request entries according to the
buffering hints (and
* the valid limits of the destination). The logic then needs to create
and execute the request
- * against the destination (ideally by batching together multiple request
entries to increase
- * efficiency). The logic also needs to identify individual request
entries that were not
- * persisted successfully and resubmit them using the {@code
requeueFailedRequestEntry} method.
+ * asynchronously against the destination (ideally by batching together
multiple request entries
+ * to increase efficiency). The logic also needs to identify individual
request entries that
+ * were not persisted successfully and resubmit them using the {@code
requestResult} callback.
+ *
+ * <p>From a threading perspective, the mailbox thread will call this
method and initiate the
+ * asynchronous request to persist the {@code requestEntries}. NOTE: The
client must support
+ * asynchronous requests and the method called to persist the records must
asynchronously
Review comment:
As you mentioned in chat, this design is from flip-171 - idea is to have
the concrete sink implementer to pull in an async client. (even though people
will have the option to add a thread pool there and have a sync client if they
so desire)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
