yashmayya commented on code in PR #13945:
URL: https://github.com/apache/kafka/pull/13945#discussion_r1266275006
##########
connect/file/src/main/java/org/apache/kafka/connect/file/FileStreamSourceConnector.java:
##########
@@ -101,4 +105,49 @@ public ExactlyOnceSupport exactlyOnceSupport(Map<String,
String> props) {
: ExactlyOnceSupport.UNSUPPORTED;
}
+ @Override
+ public boolean alterOffsets(Map<String, String> connectorConfig,
Map<Map<String, ?>, Map<String, ?>> offsets) {
+ AbstractConfig config = new AbstractConfig(CONFIG_DEF,
connectorConfig);
+ String filename = config.getString(FILE_CONFIG);
+ if (filename == null || filename.isEmpty()) {
+ throw new ConnectException("Offsets cannot be modified if the '" +
FILE_CONFIG + "' configuration is unspecified. " +
+ "This is because stdin is used for input and offsets are
not tracked.");
+ }
Review Comment:
Hah this seems very similar to what we've been discussing on the MM2 PR and
has somewhat comparable flexibility - user friendliness tradeoffs 😅
Allowing users to modify offsets when the connector is currently configured
to read from stdin could be useful in order to clean up stale offsets. However,
it could be slightly misleading to accept offsets modification requests and
return a successful response and message when in actuality the operation was a
no-op as far as the connector's task is concerned. I'd also like to note for
posterity that it isn't possible to detect past connector configurations or
previous offsets in the `SourceConnector::alterOffsets` method. Both the
following cases should be uncommon enough that either approach seems acceptable
to me -
- An unfamiliar user attempts to modify offsets for a file stream source
connector configured to read from stdin and expects the connector to re-produce
past data.
- A user configures a file stream source connector to read from a file, then
reconfigures it to read from stdin, and then wants to clean up previous offsets.
I think that the first case seems a bit more likely which is why I think the
existing approach seems preferable, WDYT? Also, the second case can still be
satisfied by temporarily reverting the configuration if the user **_really_**
wants to clean up the offsets.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
