ranjanankur commented on issue #9355: URL: https://github.com/apache/hudi/issues/9355#issuecomment-1696764242
Hi @the-other-tim-brown , @emkornfield In the above article it is mentioned to use the `use-bq-manifest-file` flag while running `BigQuerySyncTool` function to sync Hudi table with BigQuery tables. It is mentioned to use the `hudi-gcp-bundle-0.14.0-SNAPSHOT.jar`. We have upgraded this JAR too but we are finding difficulty to add this `use-bq-manifest-file` in the code. Instead of using spark-summit to run this code, we just call `BigQuerySyncTool` function from our code itself & we are confused where and how to pass this flag. Please let us know where to pass this flag. ``` def getBigQueryProps: Properties = { val props = new Properties() props.setProperty(BIGQUERY_SYNC_PROJECT_ID.key, projectId) props.setProperty(BIGQUERY_SYNC_DATASET_NAME.key, datasetName) props.setProperty(BIGQUERY_SYNC_DATASET_LOCATION.key, datasetLocation) props.setProperty(BIGQUERY_SYNC_TABLE_NAME.key, tableName) props.setProperty(BIGQUERY_SYNC_SOURCE_URI.key, s"$tablePath/$firstLevelPartition=*") props.setProperty(BIGQUERY_SYNC_SOURCE_URI_PREFIX.key, s"$tablePath/") props.setProperty(BIGQUERY_SYNC_SYNC_BASE_PATH.key, tablePath) props.setProperty(BIGQUERY_SYNC_PARTITION_FIELDS.key, partitionKey) props.setProperty(META_SYNC_BASE_PATH.key(), tablePath) props.setProperty(BIGQUERY_SYNC_USE_FILE_LISTING_FROM_METADATA.key, "true") props.setProperty(BIGQUERY_SYNC_ASSUME_DATE_PARTITIONING.key, "false") props } new BigQuerySyncTool(getBigQueryProps).syncHoodieTable() ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org