Taragolis commented on PR #29522:
URL: https://github.com/apache/airflow/pull/29522#issuecomment-1474866238

   I've tested on own AWS Account this simple DAG
   
   ```python
   import pendulum
   from airflow import DAG
   from airflow.providers.amazon.aws.operators.batch import BatchOperator
   
   JOB_NAME = "multi-node-sample"
   JOB_DEFINITION = "batch-nodes"
   JOB_QUEUE = "multinode-job-queue"
   CONTAINER_OVERRIDES = None
   ARRAY_PROPERTIES = None
   NODE_OVERRIDES = {
       "numNodes": 5
   }
   
   with DAG(
       dag_id="example_batch_submit_job_multi_node",
       schedule_interval=None,
       start_date=pendulum.datetime(2023, 1, 1, tz="UTC"),
       tags=["example", "amazon-provider", "batch", "multi-node"],
       catchup=False,
   ):
       submit_batch_job = BatchOperator(
           task_id="submit_batch_job",
           job_name=JOB_NAME,
           job_queue=JOB_QUEUE,
           job_definition=JOB_DEFINITION,
           container_overrides=CONTAINER_OVERRIDES,
           array_properties=ARRAY_PROPERTIES,
           node_overrides=NODE_OVERRIDES,
           aws_conn_id=None,
       )
   ```
   
   If only set `NODE_OVERRIDES` then it run mostly successfully, time to time 
one or many nodes fail during run without no reason. But it refers ether miss 
configuration of multinode environment or some internals of AWS.
   
   If I set `CONTAINER_OVERRIDES` to any value rather than `None` (even `{}`), 
I've got:
   ```console
   botocore.errorfactory.ClientException: An error occurred (ClientException) 
when calling the SubmitJob operation: Container overrides and node overrides 
are mutually exclusive, only one can be set.
   ```
   If I set `ARRAY_PROPERTIES` than I've got
   ```console
   botocore.errorfactory.ClientException: An error occurred (ClientException) 
when calling the SubmitJob operation: Multinode Array Job not supported.
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to