pankajastro opened a new issue, #24318: URL: https://github.com/apache/airflow/issues/24318
### Apache Airflow Provider(s) amazon ### Versions of Apache Airflow Providers _No response_ ### Apache Airflow version 2.3.2 (latest released) ### Operating System os ### Deployment Other ### Deployment details _No response_ ### What happened EmrCreateJobFlowOperator currently have two params for connection `emr_conn_id` and `aws_conn_id`. So it works only when I set `aws_conn_id` containing credentials and an empty `emr_conn_id` and it does not work in the below case - when I set both aws_conn_id and emr_conn_id in the operator and both connection contains credentials i.e it has aws_access_key_id and other params in airflow connection extra ``` Unknown parameter in input: "aws_access_key_id", must be one of: Name, LogUri, LogEncryptionKmsKeyId, AdditionalInfo, AmiVersion, ReleaseLabel, Instances, Steps, BootstrapActions, SupportedProducts, NewSupportedProducts, Applications, Configurations, VisibleToAllUsers, JobFlowRole, ServiceRole, Tags, SecurityConfiguration, AutoScalingRole, ScaleDownBehavior, CustomAmiId, EbsRootVolumeSize, RepoUpgradeOnBoot, KerberosAttributes, StepConcurrencyLevel, ManagedScalingPolicy, PlacementGroupConfigs, AutoTerminationPolicy, OSReleaseLabel ``` - when I set both aws_conn_id and emr_conn_id in the operator and only emr_conn_id connection contains credentials i.e it has aws_access_key_id and other params in airflow connection extra ``` [2022-06-07, 20:49:19 UTC] {taskinstance.py:1826} ERROR - Task failed with exception Traceback (most recent call last): File "/opt/airflow/airflow/providers/amazon/aws/operators/emr.py", line 324, in execute response = emr.create_job_flow(job_flow_overrides) File "/opt/airflow/airflow/providers/amazon/aws/hooks/emr.py", line 87, in create_job_flow response = self.get_conn().run_job_flow(**job_flow_overrides) File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 508, in _api_call return self._make_api_call(operation_name, kwargs) File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 895, in _make_api_call operation_model, request_dict, request_context File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 917, in _make_request return self._endpoint.make_request(operation_model, request_dict) File "/usr/local/lib/python3.7/site-packages/botocore/endpoint.py", line 116, in make_request return self._send_request(request_dict, operation_model) File "/usr/local/lib/python3.7/site-packages/botocore/endpoint.py", line 195, in _send_request request = self.create_request(request_dict, operation_model) File "/usr/local/lib/python3.7/site-packages/botocore/endpoint.py", line 134, in create_request operation_name=operation_model.name, File "/usr/local/lib/python3.7/site-packages/botocore/hooks.py", line 412, in emit return self._emitter.emit(aliased_event_name, **kwargs) File "/usr/local/lib/python3.7/site-packages/botocore/hooks.py", line 256, in emit return self._emit(event_name, kwargs) File "/usr/local/lib/python3.7/site-packages/botocore/hooks.py", line 239, in _emit response = handler(**kwargs) File "/usr/local/lib/python3.7/site-packages/botocore/signers.py", line 103, in handler return self.sign(operation_name, request) File "/usr/local/lib/python3.7/site-packages/botocore/signers.py", line 187, in sign auth.add_auth(request) File "/usr/local/lib/python3.7/site-packages/botocore/auth.py", line 405, in add_auth raise NoCredentialsError() botocore.exceptions.NoCredentialsError: Unable to locate credentials ``` - When I set only aws_conn_id in the operator and it contains credentials ``` Traceback (most recent call last): File "/opt/airflow/airflow/providers/amazon/aws/operators/emr.py", line 324, in execute response = emr.create_job_flow(job_flow_overrides) File "/opt/airflow/airflow/providers/amazon/aws/hooks/emr.py", line 90, in create_job_flow emr_conn = self.get_connection(self.emr_conn_id) File "/opt/airflow/airflow/hooks/base.py", line 67, in get_connection conn = Connection.get_connection_from_secrets(conn_id) File "/opt/airflow/airflow/models/connection.py", line 430, in get_connection_from_secrets raise AirflowNotFoundException(f"The conn_id `{conn_id}` isn't defined") ``` - When I set only emr_conn_id in the operator and it contains credentials ``` [2022-06-07, 20:49:19 UTC] {taskinstance.py:1826} ERROR - Task failed with exception Traceback (most recent call last): File "/opt/airflow/airflow/providers/amazon/aws/operators/emr.py", line 324, in execute response = emr.create_job_flow(job_flow_overrides) File "/opt/airflow/airflow/providers/amazon/aws/hooks/emr.py", line 87, in create_job_flow response = self.get_conn().run_job_flow(**job_flow_overrides) File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 508, in _api_call return self._make_api_call(operation_name, kwargs) File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 895, in _make_api_call operation_model, request_dict, request_context File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 917, in _make_request return self._endpoint.make_request(operation_model, request_dict) File "/usr/local/lib/python3.7/site-packages/botocore/endpoint.py", line 116, in make_request return self._send_request(request_dict, operation_model) File "/usr/local/lib/python3.7/site-packages/botocore/endpoint.py", line 195, in _send_request request = self.create_request(request_dict, operation_model) File "/usr/local/lib/python3.7/site-packages/botocore/endpoint.py", line 134, in create_request operation_name=operation_model.name, File "/usr/local/lib/python3.7/site-packages/botocore/hooks.py", line 412, in emit return self._emitter.emit(aliased_event_name, **kwargs) File "/usr/local/lib/python3.7/site-packages/botocore/hooks.py", line 256, in emit return self._emit(event_name, kwargs) File "/usr/local/lib/python3.7/site-packages/botocore/hooks.py", line 239, in _emit response = handler(**kwargs) File "/usr/local/lib/python3.7/site-packages/botocore/signers.py", line 103, in handler return self.sign(operation_name, request) File "/usr/local/lib/python3.7/site-packages/botocore/signers.py", line 187, in sign auth.add_auth(request) File "/usr/local/lib/python3.7/site-packages/botocore/auth.py", line 405, in add_auth raise NoCredentialsError() botocore.exceptions.NoCredentialsError: Unable to locate credentials ``` ### What you think should happen instead It should work with even one connection id i.e with aws_conn_id or emr_conn_id and should not fail even emr_conn_id has credentials ### How to reproduce - Create EmrCreateJobFlowOperator and pass both aws_conn_id and emr_conn_id or - Create EmrCreateJobFlowOperator and pass aws_conn_id or emr_conn_id ### Anything else _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org