Re: [PR] Add option to disable validation of cloud bigtable change stream IO [beam]
svetakvsundhar merged PR #31376: URL: https://github.com/apache/beam/pull/31376 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add option to disable validation of cloud bigtable change stream IO [beam]
svetakvsundhar commented on PR #31376: URL: https://github.com/apache/beam/pull/31376#issuecomment-2135603779 `Note`: The failures are unrelated to this change. Going to merge. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add option to disable validation of cloud bigtable change stream IO [beam]
svetakvsundhar commented on PR #31376: URL: https://github.com/apache/beam/pull/31376#issuecomment-2130065700 Sounds good. Will merge after CI completes! Thanks for the quick turnaround! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add option to disable validation of cloud bigtable change stream IO [beam]
tonytanger commented on PR #31376: URL: https://github.com/apache/beam/pull/31376#issuecomment-2130063711 @svetakvsundhar we're ready to merge. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add option to disable validation of cloud bigtable change stream IO [beam]
jackdingilian commented on code in PR #31376:
URL: https://github.com/apache/beam/pull/31376#discussion_r1613798994
##
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java:
##
@@ -2284,25 +2286,77 @@ public ReadChangeStream
withBacklogReplicationAdjustment(Duration adjustment) {
return toBuilder().setBacklogReplicationAdjustment(adjustment).build();
}
+/**
Review Comment:
We should note that this disables metadata table creation / update and that
will need to be done explicitly when this is enabled (and the table isn't
created / up-to-date)
##
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java:
##
@@ -2284,25 +2286,77 @@ public ReadChangeStream
withBacklogReplicationAdjustment(Duration adjustment) {
return toBuilder().setBacklogReplicationAdjustment(adjustment).build();
}
+/**
+ * Disables validation that the table being read and the metadata table
exists, and that the app
+ * profile used is single cluster and single row transcation enabled. Set
this option if the
+ * caller does not have additional Bigtable permissions to validate the
configurations.
+ */
+public ReadChangeStream withoutValidation() {
+ BigtableConfig config = getBigtableConfig();
+ BigtableConfig metadataTableConfig = getMetadataTableBigtableConfig();
+ return toBuilder()
+ .setBigtableConfig(config.withValidate(false))
+
.setMetadataTableBigtableConfig(metadataTableConfig.withValidate(false))
+ .setValidateConfig(false)
+ .build();
+}
+
+@Override
+public void validate(PipelineOptions options) {
+ BigtableServiceFactory factory = new BigtableServiceFactory();
+ if (getBigtableConfig().getValidate()) {
+try {
+ checkArgument(
+ factory.checkTableExists(getBigtableConfig(), options,
getTableId()),
+ "Change Stream table %s does not exist",
+ getTableId());
+} catch (IOException e) {
+ throw new RuntimeException(e);
+}
+ }
+}
+
+// Validate the app profile is single cluster and allows single row
transactions.
+private void validateAppProfile(
+MetadataTableAdminDao metadataTableAdminDao, String appProfileId) {
+ checkArgument(metadataTableAdminDao != null);
+ checkArgument(
+
metadataTableAdminDao.isAppProfileSingleClusterAndTransactional(appProfileId),
+ "App profile id '"
+ + appProfileId
+ + "' provided to access metadata table needs to use
single-cluster routing policy"
+ + " and allow single-row transactions.");
+}
+
+// Update metadata table schema if allowed and required.
+private void createOrUpdateMetadataTable(
+MetadataTableAdminDao metadataTableAdminDao, String metadataTableId) {
+ boolean shouldCreateOrUpdateMetadataTable = true;
+ if (getCreateOrUpdateMetadataTable() != null) {
+shouldCreateOrUpdateMetadataTable = getCreateOrUpdateMetadataTable();
+ }
+ // Only try to create or update metadata table if option is set to true.
Otherwise, just
+ // check if the table exists.
+ if (shouldCreateOrUpdateMetadataTable &&
metadataTableAdminDao.createMetadataTable()) {
+LOG.info("Created metadata table: " + metadataTableId);
+ }
+}
+
@Override
public PCollection> expand(PBegin
input) {
checkArgument(
getBigtableConfig() != null,
"BigtableIO ReadChangeStream is missing required configurations
fields.");
- checkArgument(
- getBigtableConfig().getProjectId() != null, "Missing required
projectId field.");
- checkArgument(
- getBigtableConfig().getInstanceId() != null, "Missing required
instanceId field.");
+ getBigtableConfig().validate();
checkArgument(getTableId() != null, "Missing required tableId field.");
BigtableConfig bigtableConfig = getBigtableConfig();
Review Comment:
Nit: move this before the two calls to getBigtableConfig within this method
above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@beam.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] Add option to disable validation of cloud bigtable change stream IO [beam]
tonytanger commented on PR #31376: URL: https://github.com/apache/beam/pull/31376#issuecomment-2127899927 I added some unit tests for the ReadChangeStream config. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add option to disable validation of cloud bigtable change stream IO [beam]
github-actions[bot] commented on PR #31376: URL: https://github.com/apache/beam/pull/31376#issuecomment-2127438489 Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`: R: @m-trieu for label java. R: @johnjcasey for label io. R: @igorbernstein2 for label bigtable. Available commands: - `stop reviewer notifications` - opt out of the automated review tooling - `remind me after tests pass` - tag the comment author after tests pass - `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers) The PR bot will only process comments in the main thread (not review comments). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add option to disable validation of cloud bigtable change stream IO [beam]
tonytanger commented on PR #31376: URL: https://github.com/apache/beam/pull/31376#issuecomment-2127413740 @jackdingilian for Bigtable review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
