Taragolis commented on PR #29522:
URL: https://github.com/apache/airflow/pull/29522#issuecomment-1474866238
I've tested on own AWS Account this simple DAG
```python
import pendulum
from airflow import DAG
from airflow.providers.amazon.aws.operators.batch import BatchOperator
JOB_NAME = "multi-node-sample"
JOB_DEFINITION = "batch-nodes"
JOB_QUEUE = "multinode-job-queue"
CONTAINER_OVERRIDES = None
ARRAY_PROPERTIES = None
NODE_OVERRIDES = {
"numNodes": 5
}
with DAG(
dag_id="example_batch_submit_job_multi_node",
schedule_interval=None,
start_date=pendulum.datetime(2023, 1, 1, tz="UTC"),
tags=["example", "amazon-provider", "batch", "multi-node"],
catchup=False,
):
submit_batch_job = BatchOperator(
task_id="submit_batch_job",
job_name=JOB_NAME,
job_queue=JOB_QUEUE,
job_definition=JOB_DEFINITION,
container_overrides=CONTAINER_OVERRIDES,
array_properties=ARRAY_PROPERTIES,
node_overrides=NODE_OVERRIDES,
aws_conn_id=None,
)
```
If only set `NODE_OVERRIDES` then it run mostly successfully, time to time
one or many nodes fail during run without no reason. But it refers ether miss
configuration of multinode environment or some internals of AWS.
If I set `CONTAINER_OVERRIDES` to any value rather than `None` (even `{}`),
I've got:
```console
botocore.errorfactory.ClientException: An error occurred (ClientException)
when calling the SubmitJob operation: Container overrides and node overrides
are mutually exclusive, only one can be set.
```
If I set `ARRAY_PROPERTIES` than I've got
```console
botocore.errorfactory.ClientException: An error occurred (ClientException)
when calling the SubmitJob operation: Multinode Array Job not supported.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]