Hi, Thank you so much for your response. I tried by specifying a temp dataset using the temp_dataset parameter of ReadFromBigQuery and it worked. I was looking at the BigQuerySource class but could not find any such parameters for setting temp_dataset and my job fails by throwing same permission error. So is there any such way of doing using BigQuerySource class.
Different: Could you please help understand the diff. between ReadFromBigQuery and BigQuerySource in terms of how they work [1]. I understand both are used to read from Big Query. [1] https://beam.apache.org/releases/pydoc/2.36.0/apache_beam.io.gcp.bigquery.html#apache_beam.io.gcp.bigquery.ReadFromBigQuery Thanks & Regards Rajnil On Tue, 22 Feb 2022 at 1:56 PM, Israel Herraiz <i...@google.com> wrote: > Yes, if you read from a query, BigQueryIO will write the results of the > query to a temporary table in a temporary dataset. > > If you don't specify that temp dataset, Dataflow will try to create it. > > In Java, if you don't want to give permissions to the service account to > create datasets, you can also specify an existing dataset, to be used as > "temporary" dataset for the results of the queries. See the option > temp_dataset in ReadFromBigQuery: > https://beam.apache.org/releases/pydoc/2.36.0/apache_beam.io.gcp.bigquery.html#apache_beam.io.gcp.bigquery.ReadFromBigQuery > > Dataflow will need permissions to be able to create tables inside that > dataset. >
