Nidhi Chourasia created AIRFLOW-5271: ----------------------------------------
Summary: EmrCreateJobFlowOperator throwing error in airflow 1.10.4 version Key: AIRFLOW-5271 URL: https://issues.apache.org/jira/browse/AIRFLOW-5271 Project: Apache Airflow Issue Type: Bug Components: operators Affects Versions: 1.10.4 Environment: Operating System details: ubuntu@ip-10-0-1-252:~$ cat /etc/os-release NAME="Ubuntu" VERSION="18.04.1 LTS (Bionic Beaver)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 18.04.1 LTS" VERSION_ID="18.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=bionic UBUNTU_CODENAME=bionic Reporter: Nidhi Chourasia Fix For: 1.10.5 h3. *ERROR LOGS:* {{[2019-08-21 05:39:42,970] \{emr_create_job_flow_operator.py:66} INFO - Creating JobFlow using aws-conn-id: aws_default, emr-conn-id: emr_default [2019-08-21 05:39:42,981] \{logging_mixin.py:95} INFO - [[34m2019-08-21 05:39:42,980[0m] \{[34mconnection.py:[0m296} ERROR[0m - Expecting property name enclosed in double quotes: line 1 column 2 (char 1)[0m [31mTraceback (most recent call last): File "/home/ubuntu/.local/lib/python2.7/site-packages/airflow/models/connection.py", line 294, in extra_dejson obj = json.loads(self.extra) File "/usr/lib/python2.7/json/__init__.py", line 339, in loads return _default_decoder.decode(s) File "/usr/lib/python2.7/json/decoder.py", line 364, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python2.7/json/decoder.py", line 380, in raw_decode obj, end = self.scan_once(s, idx) ValueError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)[0m [2019-08-21 05:39:42,982] \{logging_mixin.py:95} INFO - [[34m2019-08-21 05:39:42,981[0m] \{[34mconnection.py:[0m297} ERROR[0m - Failed parsing the json for conn_id [1maws_default[0m[0m [2019-08-21 05:39:43,054] \{taskinstance.py:1047} ERROR - Parameter validation failed: Unknown parameter in input: "TerminationProtected", must be one of: Name, LogUri, AdditionalInfo, AmiVersion, ReleaseLabel, Instances, Steps, BootstrapActions, SupportedProducts, NewSupportedProducts, Applications, Configurations, VisibleToAllUsers, JobFlowRole, ServiceRole, Tags, SecurityConfiguration, AutoScalingRole, ScaleDownBehavior, CustomAmiId, EbsRootVolumeSize, RepoUpgradeOnBoot, KerberosAttributes Unknown parameter in input: "KeepJobFlowAliveWhenNoSteps", must be one of: Name, LogUri, AdditionalInfo, AmiVersion, ReleaseLabel, Instances, Steps, BootstrapActions, SupportedProducts, NewSupportedProducts, Applications, Configurations, VisibleToAllUsers, JobFlowRole, ServiceRole, Tags, SecurityConfiguration, AutoScalingRole, ScaleDownBehavior, CustomAmiId, EbsRootVolumeSize, RepoUpgradeOnBoot, KerberosAttributes Unknown parameter in input: "Ec2SubnetId", must be one of: Name, LogUri, AdditionalInfo, AmiVersion, ReleaseLabel, Instances, Steps, BootstrapActions, SupportedProducts, NewSupportedProducts, Applications, Configurations, VisibleToAllUsers, JobFlowRole, ServiceRole, Tags, SecurityConfiguration, AutoScalingRole, ScaleDownBehavior, CustomAmiId, EbsRootVolumeSize, RepoUpgradeOnBoot, KerberosAttributes Unknown parameter in input: "Ec2KeyName", must be one of: Name, LogUri, AdditionalInfo, AmiVersion, ReleaseLabel, Instances, Steps, BootstrapActions, SupportedProducts, NewSupportedProducts, Applications, Configurations, VisibleToAllUsers, JobFlowRole, ServiceRole, Tags, SecurityConfiguration, AutoScalingRole, ScaleDownBehavior, CustomAmiId, EbsRootVolumeSize, RepoUpgradeOnBoot, KerberosAttributes}} h3. {{*CORRESPONDING DAG CODE:*}} {{}} {noformat} // code placeholder airflow_test_json=json.load(open(airflow_home+'/test.json')) airflow_asset_analytics_creator = EmrCreateJobFlowOperator( task_id='create_asset_analytics_databricks_test', job_flow_overrides=airflow_test_json['Job'], timeout=10, aws_conn_id='aws_default', emr_conn_id='emr_default', dag=dag ) airflow_asset_analytics_sensor = EmrJobFlowSensor( task_id='check_asset_analytics_databricks_stable', job_flow_id="{{ task_instance.xcom_pull('create_asset_analytics_databricks_test', key='return_value') }}", aws_conn_id='aws_default', dag=dag ) airflow_asset_analytics_sensor.set_upstream(airflow_asset_analytics_creator) {noformat} {{}} Not sure what more details are required,but the exact code worked flawlessly in v1.10.0. -- This message was sent by Atlassian Jira (v8.3.2#803003)