reuvenlax commented on code in PR #38058:
URL: https://github.com/apache/beam/pull/38058#discussion_r3121494735
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/StorageApiWriteUnshardedRecords.java:
##########
@@ -847,11 +896,40 @@ long flush(
+ failedContext.offset);
}
+ // Schema mismatched exceptions can happen if the table was
recently updated. Since
+ // vortex caches schemas
+ // we might see the new schema before vortex does. In this case,
we simply need to
+ // retry.
+ Exceptions.@Nullable StorageException storageException =
+ (error == null) ? null :
Exceptions.toStorageException(error);
+ boolean schemaMismatchError =
+ (storageException instanceof
Exceptions.SchemaMismatchedException);
+ if (!schemaMismatchError && error != null) {
+ // There's no special error code for missing required fields,
and that can also
+ // happen due to vortex
+ // being delayed at seeing a new schema. We're forced to parse
the description to
+ // determine that this
+ // has happened.
+ Status status = Status.fromThrowable(error);
+ if (status.getCode() == Status.Code.INVALID_ARGUMENT) {
+ String description = status.getDescription();
+ schemaMismatchError =
+ description != null &&
description.contains("incompatible fields");
+ }
+ }
+ if (schemaMismatchError) {
+ LOG.info(
+ "Vortex failed stream open due to incompatible fields.
This is likely because the BigTable "
+ + "schema was recently updated and Vortex hasn't
noticed yet, so retrying. error {}",
+ Preconditions.checkStateNotNull(error).toString());
+ }
+
Review Comment:
Yes - unfortunately the logic is subtly different, so we can't just simply
move it into another class. The followon PR to this one begins the process of
refactoring these sinks to make them easier to follow. We can then refactor
some more logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
