tianz101 commented on code in PR #37326:
URL: https://github.com/apache/beam/pull/37326#discussion_r2710343613
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/action/QueryChangeStreamAction.java:
##########
@@ -179,10 +178,9 @@ public ProcessContinuation run(
final String token = partition.getPartitionToken();
final Timestamp startTimestamp = tracker.currentRestriction().getFrom();
final Timestamp endTimestamp = partition.getEndTimestamp();
Review Comment:
Here endTs from the partition can be
1) bounded (i.e., a specific end ts specified by user) or
2) unbounded(i.e., MAX_INCLUSIVE_END_AT).
If bounded,
1.1) For change stream V1, we can use the bounded endTs to query.
1.2) For change stream V2, we can not use the bounded endTs to query as
validation may fail endTs <= max(startTs, now) + 30m and return exception. So
we should use now+2m instead.
If unbounded,
2.1) For change stream V1, we can use the unbounded endTs to query.
2.2) For change stream V2 we can not use the unbounded endTs to query as
validation may fail endTs <= max(startTs, now) + 30min and return exception.
So we should use now+2m instead.
So in summary, the logic seems to be:
for V1, use endTs no matter bounded or unbounded endTs.
for V2, use now+2m always. Plus if bounded endTs, and now+2m >= bounded
endTs, then the partition is finished.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
