The GitHub Actions job "Tests (AMD)" on airflow.git/feature/68279-fix-deferrable-beam-dataflow-job-id has failed. Run started by GitHub user gingeekrishna (triggered by henry3260).
Head commit for run: d8de40e5321202d04d2a45a39fb951cf3f4a1a99 / gingeekrishna <[email protected]> Fix deferrable Beam Dataflow operator failing with 400 when job ID is missing from stdout When the Dataflow launcher process runs with WARNING log level (the default), it does not emit the "Created job with id" line that the Beam operator parses to capture the Dataflow job ID. This left dataflow_job_id as None, causing the deferrable trigger to fail with "400 Request must contain a job and project id". Fix by adding a periodic_callback parameter to run_beam_command() that is invoked roughly every 5 seconds while the launcher subprocess is running. The deferrable Beam operators now pass a callback that polls DataflowHook.fetch_job_id_by_name() to resolve the job ID by name. Once the ID is set, the stdout-reading loop exits early so the operator can truly defer, freeing the Airflow worker while the Dataflow job continues running on Google Cloud. Fixes #68279 Report URL: https://github.com/apache/airflow/actions/runs/28313367881 With regards, GitHub Actions via GitBox --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
