Marcio Sugar created NIFI-11791: ----------------------------------- Summary: PutBigQuery processor lacks functionality found in PutBigQueryBatch Key: NIFI-11791 URL: https://issues.apache.org/jira/browse/NIFI-11791 Project: Apache NiFi Issue Type: Improvement Components: Extensions Affects Versions: 1.22.0, 2.0.0 Reporter: Marcio Sugar
Before PutBigQuery, we had PutBigQueryBatch and PutBigQueryStream, both now deprecated. Not sure if PutBigQuery was designed to completely replace its older brothers, but it cannot do that yet because of some missing features. For example, we can't use PubBigQuery alone to create snapshot tables, something that was easy to do with PutBigQueryBatch. A snapshot table is a recent copy of a table from a database or a subset of rows/columns of a table. It is used to dynamically replicate data between distributed databases. Using PutBigQueryBatch, we can achieve that by setting the following properties: * Create Disposition = CREATE_IF_NEEDED * Write Disposition = WRITE_TRUNCATE I understand that PutBigQuery uses the newer [BigQuery Storage Write API|https://cloud.google.com/bigquery/docs/write-api], so adding the missing functionality might not be possible. But please note the older BigQuery (core) API (the one I believe PutBigQueryBatch uses) allows the user to submit jobs to load data into BigQuery in a very convenient way. That is something I'd like to see preserved in future versions of NiFi -- This message was sent by Atlassian Jira (v8.20.10#820010)