dannycranmer commented on a change in pull request #17687:
URL: https://github.com/apache/flink/pull/17687#discussion_r743024147
##########
File path:
flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/sink/AsyncSinkBase.java
##########
@@ -49,6 +50,28 @@
public abstract class AsyncSinkBase<InputT, RequestEntryT extends Serializable>
implements Sink<InputT, Void, Collection<RequestEntryT>, Void> {
+ protected final ElementConverter<InputT, RequestEntryT> elementConverter;
+ protected final int maxBatchSize;
+ protected final int maxInFlightRequests;
+ protected final int maxBufferedRequests;
+ protected final long flushOnBufferSizeInBytes;
+ protected final long maxTimeInBufferMS;
Review comment:
Why are these protected rather than private?
##########
File path:
flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/sink/AsyncSinkBaseBuilder.java
##########
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.base.sink;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+
+import java.io.Serializable;
+
+/**
+ * Abstract builder for constructing a concrete implementation of {@link
AsyncSinkBase}.
+ *
+ * @param <InputT> type of elements that should be persisted in the destination
+ * @param <RequestEntryT> type of payload that contains the element and
additional metadata that is
+ * required to submit a single element to the destination
+ * @param <ConcreteBuilderT> type of concrete implementation of this builder
class
+ */
+@PublicEvolving
+public abstract class AsyncSinkBaseBuilder<
+ InputT,
+ RequestEntryT extends Serializable,
+ ConcreteBuilderT extends AsyncSinkBaseBuilder<?, ?, ?>> {
+
+ protected ElementConverter<InputT, RequestEntryT> elementConverter;
+ protected Integer maxBatchSize;
+ protected Integer maxInFlightRequests;
+ protected Integer maxBufferedRequests;
+ protected Long flushOnBufferSizeInBytes;
+ protected Long maxTimeInBufferMS;
Review comment:
Why are these protected rather than private?
##########
File path:
flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/sink/writer/AsyncSinkWriter.java
##########
@@ -172,6 +229,15 @@ public AsyncSinkWriter(
this.inFlightRequestsCount = 0;
this.bufferedRequestEntriesTotalSizeInBytes = 0;
+
+ this.metrics = context.metricGroup();
+ this.metrics.setCurrentSendTimeGauge(
+ () -> {
+ long time = this.ackTime - this.lastSendTimestamp;
+ return time < 0 ? this.lastSendDuration : time;
Review comment:
Is this the same as `lastSendDuration`?
##########
File path:
flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/sink/writer/AsyncSinkWriter.java
##########
@@ -172,6 +229,15 @@ public AsyncSinkWriter(
this.inFlightRequestsCount = 0;
this.bufferedRequestEntriesTotalSizeInBytes = 0;
+
+ this.metrics = context.metricGroup();
+ this.metrics.setCurrentSendTimeGauge(
+ () -> {
+ long time = this.ackTime - this.lastSendTimestamp;
+ return time < 0 ? this.lastSendDuration : time;
Review comment:
Is this thread safe when there are multiple in-flight requests? How do
we know `ackTime` and `lastSendTimestamp` are the same request?
##########
File path:
flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/sink/writer/AsyncSinkWriter.java
##########
@@ -114,9 +134,38 @@
*
* <p>The method is invoked with a set of request entries according to the
buffering hints (and
* the valid limits of the destination). The logic then needs to create
and execute the request
- * against the destination (ideally by batching together multiple request
entries to increase
- * efficiency). The logic also needs to identify individual request
entries that were not
- * persisted successfully and resubmit them using the {@code
requeueFailedRequestEntry} method.
+ * asynchronously against the destination (ideally by batching together
multiple request entries
+ * to increase efficiency). The logic also needs to identify individual
request entries that
+ * were not persisted successfully and resubmit them using the {@code
requestResult} callback.
+ *
+ * <p>From a threading perspective, the mailbox thread will call this
method and initiate the
+ * asynchronous request to persist the {@code requestEntries}. NOTE: The
client must support
+ * asynchronous requests and the method called to persist the records must
asynchronously
Review comment:
I am still not 100% on the concrete part handling the async calls,
rather than the base handling threading. Then the concrete implementation just
performing simple sync calls. For instance, in KDS sink we are using an Async
Kinesis client, which is equivalent to a thread pool here and Sync Kinesis
client. Please remind me of the benefit/reason of delegating async to the
concrete implementation?
##########
File path:
flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/sink/AsyncSinkBase.java
##########
@@ -49,6 +50,28 @@
public abstract class AsyncSinkBase<InputT, RequestEntryT extends Serializable>
implements Sink<InputT, Void, Collection<RequestEntryT>, Void> {
+ protected final ElementConverter<InputT, RequestEntryT> elementConverter;
+ protected final int maxBatchSize;
+ protected final int maxInFlightRequests;
+ protected final int maxBufferedRequests;
+ protected final long flushOnBufferSizeInBytes;
+ protected final long maxTimeInBufferMS;
Review comment:
Generally speaking I would be more inclined towards private variables
with protected getters in this case. It is more maintainable for the future
##########
File path:
flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/sink/AsyncSinkBaseBuilder.java
##########
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.base.sink;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+
+import java.io.Serializable;
+
+/**
+ * Abstract builder for constructing a concrete implementation of {@link
AsyncSinkBase}.
+ *
+ * @param <InputT> type of elements that should be persisted in the destination
+ * @param <RequestEntryT> type of payload that contains the element and
additional metadata that is
+ * required to submit a single element to the destination
+ * @param <ConcreteBuilderT> type of concrete implementation of this builder
class
+ */
+@PublicEvolving
+public abstract class AsyncSinkBaseBuilder<
+ InputT,
+ RequestEntryT extends Serializable,
+ ConcreteBuilderT extends AsyncSinkBaseBuilder<?, ?, ?>> {
+
+ protected ElementConverter<InputT, RequestEntryT> elementConverter;
+ protected Integer maxBatchSize;
+ protected Integer maxInFlightRequests;
+ protected Integer maxBufferedRequests;
+ protected Long flushOnBufferSizeInBytes;
+ protected Long maxTimeInBufferMS;
Review comment:
Same as above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
