carlpayne commented on issue #23291: URL: https://github.com/apache/beam/issues/23291#issuecomment-1325125643
@reuvenlax although this would definitely help in one specific scenario (a required field with default is removed), there are unfortunately other problematic cases where the Avro schema is not in sync with BigQuery (see below for full details, but most notably adding a new optional field is very common for us). **Note**: in all of the below examples where BigQueryIO returns `INVALID_ARGUMENT`, the errors also end up in the retry-loop discussed earlier and are not immediately visible via `getFailedStorageApiInserts`. Ideally, until we get more flexibility around `SchemaUpdateOption` it would be great if these could be caught immediately and then returned via `getFailedStorageApiInserts` Scenario | Supported in Avro (Full Transitive) | Supported in BigQueryIO | Comments -- | -- | -- | -- Add required field with default value | Yes | No | INVALID_ARGUMENT: Input schema has more fields than BigQuery schema Add optional field with default value | Yes | No | INVALID_ARGUMENT: Input schema has more fields than BigQuery schema Add default value to existing required field | Yes | No | BigQueryIO runs, but does not sync the new default value Add default value to existing optional field | Yes | No | BigQueryIO runs, but does not sync the new default value Remove default value from existing required field | Yes (if field existed continuously since v1) | No | BigQueryIO runs, but does not remove the default value Remove default value from existing optional field | Yes (if field existed continuously since v1) | No | BigQueryIO runs, but does not remove the default value Delete optional field | Yes (if default exists) | Yes | Delete required field | Yes (if default exists) | No | INVALID_ARGUMENT: The required BigQuery field .required_field_with_default is missing in the proto message -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
