This looks strange. It looks like add_argument is called twice. I am not sure it will work but can you put all your parser to a function like parse_known_args and then call it from run?
e.g., https://github.com/apache/beam/blob/d4dd58b2c4c4b5867ace4bdd34e1bcc32de963cc/sdks/python/apache_beam/examples/per_entity_training.py#L110 On Sat, Oct 5, 2024 at 4:48 PM Sofia’s World <[email protected]> wrote: > HI all > not quite sure what is happening, but i have a job that has been running > with these parameters for the last month - from logs - > > Using launch args: [/template/dataflow_tester.py --limit=10 > --project=datascience-projects > --staging_location=gs://datascience-projects_cloudbuild > --subnetwork=regions/us-central1/subnetworks/default > --temp_location=gs://datascience-projects_cloudbuild --input= > --template_location=gs://datascience-projects_cloudbuild/template_launches/2024-10-05_13_33_33-13068921462026318228/job_object > --fmprepkey=abcdef --output=annual_fundamentals --pat=mypat > --sdk_container_image=gcr.io/datascience-projects/dftester:latest3 > --runner=DataflowRunner --job_name=dftester --region=us-central1 > --service_account_email=682143946483 --period=10 --runtype=premarket] > > and suddenly i am getting this exception > > argparse.ArgumentError: argument --fmprepkey: conflicting option string: > --fmprepkey > > this is the code that kickstart my pipeline... not sure why suddenly i am > getting this error suddenly > > def run(argv: list[str] | None = None): > """Main entry point; defines and runs the wordcount pipeline.""" > > logging.info(f'running with arguments:{argv}') > parser = argparse.ArgumentParser() > parser.add_argument('--fmprepkey') > parser.add_argument('--input') > parser.add_argument('--output') > parser.add_argument('--period') > parser.add_argument('--limit') > parser.add_argument('--pat') > parser.add_argument('--runtype') > > pipeline_options, other_args = parser.parse_known_args(argv) > > > # We use the save_main_session option because one or more DoFn's in this > # workflow rely on global context (e.g., a module imported at module > level). > logging.info(f'Starting tester pipeline:{pipeline_options.fmprepkey}') > > Could any one advice? did i pick up some recent changes? > > Kind regards > Marco > > > > > > > > > > >
