What are the command line arguments that you are using for those direct runner pipelines? For instance, for BigQuery you will need to set --tempLocation to a GCS location for the BQ jobs to work.
On Wed, 31 Aug 2022 at 09:50, Fabian Peters <[email protected]> wrote: > Good morning! > > I'm putting together my Dataflow deployment and am running into another > problem I don't know how to deal with: I'm running a pipeline via Dataflow, > which contains a "Workflow executor" transform. The workflow contains a > number of pipelines that have their run configuration set to Beam-Direct. > In principle, this works fine. (Yeah!) > > However, in this setup a BigQuery Output fails with a > "java.lang.RuntimeException: Failed to create job with prefix > beam_bq_job_LOAD_sites_FOO_ID, reached max retries: 3, last failed job: > null." I see the the same when running just the pipeline (or any other with > BigQuery Output) via Beam-Direct locally, which makes me think that the GCP > credentials are not being picked up? Is there something I need to configure? > > cheers > > Fabian > > P.S.: Logs from running locally with Beam-Direct: > > 2022/08/31 09:30:07 - sites - ERROR: Error starting the Beam pipeline > 2022/08/31 09:30:07 - sites - ERROR: > org.apache.hop.core.exception.HopException: > 2022/08/31 09:30:07 - sites - Error executing pipeline with runner Direct > 2022/08/31 09:30:07 - sites - java.lang.RuntimeException: Failed to create > job with prefix > beam_bq_job_LOAD_sites_65dba39290c04240933e3a982c0c5699_b77cb1586fc969929097729a4a6cdf2a_00001_00000, > reached max retries: 3, last failed job: null. > 2022/08/31 09:30:07 - sites - > 2022/08/31 09:30:07 - sites - at > org.apache.hop.beam.engines.BeamPipelineEngine.executePipeline(BeamPipelineEngine.java:258) > 2022/08/31 09:30:07 - sites - at > org.apache.hop.beam.engines.BeamPipelineEngine.lambda$startThreads$0(BeamPipelineEngine.java:305) > 2022/08/31 09:30:07 - sites - at > java.base/java.lang.Thread.run(Thread.java:829) > 2022/08/31 09:30:07 - sites - Caused by: > org.apache.beam.sdk.Pipeline$PipelineExecutionException: > java.lang.RuntimeException: Failed to create job with prefix > beam_bq_job_LOAD_sites_65dba39290c04240933e3a982c0c5699_b77cb1586fc969929097729a4a6cdf2a_00001_00000, > reached max retries: 3, last failed job: null. > 2022/08/31 09:30:07 - sites - at > org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:373) > 2022/08/31 09:30:07 - sites - at > org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:341) > 2022/08/31 09:30:07 - sites - at > org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:218) > 2022/08/31 09:30:07 - sites - at > org.apache.hop.beam.engines.BeamPipelineEngine.executePipeline(BeamPipelineEngine.java:246) > 2022/08/31 09:30:07 - sites - ... 2 more > 2022/08/31 09:30:07 - sites - Caused by: java.lang.RuntimeException: > Failed to create job with prefix > beam_bq_job_LOAD_sites_65dba39290c04240933e3a982c0c5699_b77cb1586fc969929097729a4a6cdf2a_00001_00000, > reached max retries: 3, last failed job: null. > 2022/08/31 09:30:07 - sites - at > org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJob.runJob(BigQueryHelpers.java:199) > 2022/08/31 09:30:07 - sites - at > org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJobManager.waitForDone(BigQueryHelpers.java:152) > 2022/08/31 09:30:07 - sites - at > org.apache.beam.sdk.io.gcp.bigquery.WriteTables$WriteTablesDoFn.finishBundle(WriteTables.java:380) > >
