Good morning! I'm putting together my Dataflow deployment and am running into another problem I don't know how to deal with: I'm running a pipeline via Dataflow, which contains a "Workflow executor" transform. The workflow contains a number of pipelines that have their run configuration set to Beam-Direct. In principle, this works fine. (Yeah!)
However, in this setup a BigQuery Output fails with a "java.lang.RuntimeException: Failed to create job with prefix beam_bq_job_LOAD_sites_FOO_ID, reached max retries: 3, last failed job: null." I see the the same when running just the pipeline (or any other with BigQuery Output) via Beam-Direct locally, which makes me think that the GCP credentials are not being picked up? Is there something I need to configure? cheers Fabian P.S.: Logs from running locally with Beam-Direct: 2022/08/31 09:30:07 - sites - ERROR: Error starting the Beam pipeline 2022/08/31 09:30:07 - sites - ERROR: org.apache.hop.core.exception.HopException: 2022/08/31 09:30:07 - sites - Error executing pipeline with runner Direct 2022/08/31 09:30:07 - sites - java.lang.RuntimeException: Failed to create job with prefix beam_bq_job_LOAD_sites_65dba39290c04240933e3a982c0c5699_b77cb1586fc969929097729a4a6cdf2a_00001_00000, reached max retries: 3, last failed job: null. 2022/08/31 09:30:07 - sites - 2022/08/31 09:30:07 - sites - at org.apache.hop.beam.engines.BeamPipelineEngine.executePipeline(BeamPipelineEngine.java:258) 2022/08/31 09:30:07 - sites - at org.apache.hop.beam.engines.BeamPipelineEngine.lambda$startThreads$0(BeamPipelineEngine.java:305) 2022/08/31 09:30:07 - sites - at java.base/java.lang.Thread.run(Thread.java:829) 2022/08/31 09:30:07 - sites - Caused by: org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.lang.RuntimeException: Failed to create job with prefix beam_bq_job_LOAD_sites_65dba39290c04240933e3a982c0c5699_b77cb1586fc969929097729a4a6cdf2a_00001_00000, reached max retries: 3, last failed job: null. 2022/08/31 09:30:07 - sites - at org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:373) 2022/08/31 09:30:07 - sites - at org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:341) 2022/08/31 09:30:07 - sites - at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:218) 2022/08/31 09:30:07 - sites - at org.apache.hop.beam.engines.BeamPipelineEngine.executePipeline(BeamPipelineEngine.java:246) 2022/08/31 09:30:07 - sites - ... 2 more 2022/08/31 09:30:07 - sites - Caused by: java.lang.RuntimeException: Failed to create job with prefix beam_bq_job_LOAD_sites_65dba39290c04240933e3a982c0c5699_b77cb1586fc969929097729a4a6cdf2a_00001_00000, reached max retries: 3, last failed job: null. 2022/08/31 09:30:07 - sites - at org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJob.runJob(BigQueryHelpers.java:199) 2022/08/31 09:30:07 - sites - at org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJobManager.waitForDone(BigQueryHelpers.java:152) 2022/08/31 09:30:07 - sites - at org.apache.beam.sdk.io.gcp.bigquery.WriteTables$WriteTablesDoFn.finishBundle(WriteTables.java:380)
