dhana-sekhar opened a new issue, #38822:
URL: https://github.com/apache/airflow/issues/38822
### Apache Airflow version
Other Airflow 2 version (please specify below)
### If "Other Airflow 2 version" selected, which one?
2.4.3
### What happened?
we are trying to create a cluster with SageMakerProcessingOperator with
"InstanceCount":2 and passing a custom docker image where it have my spark code.
Now when I try to run the spark code in the container on this cluster, my
spark code is not running on distributed way but I can see from cloudwatch that
sagemaker is able to spin-up 2 instances.
> [!IMPORTANT]
> here is the code i am using to run the SageMakerProcessingOperator
```
def create_sagemaker_processingjob_config(
name_prefix: str,
image_uri: str,
iam_role: str,
entrypoint_cmd: List[str],
instance_type: C.InstanceType,
) -> Dict:
now = pendulum.now(C.timezone)
id_from_time = now.strftime("%Y-%m-%d-%H%M%S-") + str(uuid4())[:8]
conf = {
"ProcessingJobName": f"{name_prefix}-{id_from_time}",
"ProcessingResources": {
"ClusterConfig": {
"InstanceCount": 2,
"InstanceType": instance_type.value,
"VolumeSizeInGB": 60,
}
},
"StoppingCondition": {"MaxRuntimeInSeconds": "432000"},
"AppSpecification": {
"ImageUri": image_uri,
"ContainerEntrypoint": entrypoint_cmd,
},
"Environment": {
"AWS_ACCOUNT_ID": get_aws_account_id(),
"startdate": get_start_date(),
"enddate": get_end_date(),
},
"RoleArn": iam_role,
"NetworkConfig": {
"VpcConfig": {
"SecurityGroupIds": get_security_group_ids(),
"Subnets": get_subnets(),
},
},
}
return conf
def create_sagemaker_processing_operator(
account_id: C.AccountID,
job_name: C.JobName,
instance_type: C.InstanceType,
) -> SageMakerProcessingOperator:
env_name = C.AccountID(account_id).name
iam_role = C.IAMRole[env_name].value
ecr_path = "dkr.ecr.ap-southeast-1.amazonaws.com"
image_uri = f'{account_id}.{ecr_path}/{C.ImageName.image_name}'
cmd = C.ENTRYPOINT_COMMAND[job_name]
out = SageMakerProcessingOperator(
config=create_sagemaker_processingjob_config(
name_prefix=job_name,
image_uri=image_uri,
iam_role=iam_role,
entrypoint_cmd=cmd,
instance_type=instance_type,
),
task_id=job_name,
)
return out
```
### What you think should happen instead?
when I run my code with the SageMakerProcessingOperator with
"InstanceCount":2 it should run in distributed mode
> [!NOTE]
> when I run the job with one instance it is taking 1hr to finish the job,
my expectation is if I use 2 instances with same instance type it should finish
less than 1hr.
### How to reproduce
you can use this code and take any docker image with spark installed. run
some pyspark code witin the container with 2 or more InstanceCount.
```
def create_sagemaker_processingjob_config(
name_prefix: str,
image_uri: str,
iam_role: str,
entrypoint_cmd: List[str],
instance_type: C.InstanceType,
) -> Dict:
now = pendulum.now(C.timezone)
id_from_time = now.strftime("%Y-%m-%d-%H%M%S-") + str(uuid4())[:8]
conf = {
"ProcessingJobName": f"{name_prefix}-{id_from_time}",
"ProcessingResources": {
"ClusterConfig": {
"InstanceCount": 2,
"InstanceType": instance_type.value,
"VolumeSizeInGB": 60,
}
},
"StoppingCondition": {"MaxRuntimeInSeconds": "432000"},
"AppSpecification": {
"ImageUri": image_uri,
"ContainerEntrypoint": entrypoint_cmd,
},
"Environment": {
"AWS_ACCOUNT_ID": get_aws_account_id(),
"startdate": get_start_date(),
"enddate": get_end_date(),
},
"RoleArn": iam_role,
"NetworkConfig": {
"VpcConfig": {
"SecurityGroupIds": get_security_group_ids(),
"Subnets": get_subnets(),
},
},
}
return conf
def create_sagemaker_processing_operator(
account_id: C.AccountID,
job_name: C.JobName,
instance_type: C.InstanceType,
) -> SageMakerProcessingOperator:
env_name = C.AccountID(account_id).name
iam_role = C.IAMRole[env_name].value
ecr_path = "dkr.ecr.ap-southeast-1.amazonaws.com"
image_uri = f'{account_id}.{ecr_path}/{C.ImageName.image_name}'
cmd = C.ENTRYPOINT_COMMAND[job_name]
out = SageMakerProcessingOperator(
config=create_sagemaker_processingjob_config(
name_prefix=job_name,
image_uri=image_uri,
iam_role=iam_role,
entrypoint_cmd=cmd,
instance_type=instance_type,
),
task_id=job_name,
)
return out
```
### Operating System
Amazon Linux
### Versions of Apache Airflow Providers
MWAA version 2.4.3
### Deployment
Amazon (AWS) MWAA
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]