[ 
https://issues.apache.org/jira/browse/AIRFLOW-6824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17038385#comment-17038385
 ] 

ASF GitHub Bot commented on AIRFLOW-6824:
-----------------------------------------

baolsen commented on pull request #7443: [AIRFLOW-6824] EMRAddStepsOperator 
problem with multi-step XCom
URL: https://github.com/apache/airflow/pull/7443
 
 
   ---
   Issue link: WILL BE INSERTED BY 
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Commit message/PR title starts with `[AIRFLOW-NNNN]`. AIRFLOW-NNNN = 
JIRA ID<sup>*</sup>
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   <sup>*</sup> For document-only changes commit message can start with 
`[AIRFLOW-XXXX]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   
 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> EMRAddStepsOperator does not work well with multi-step XCom
> -----------------------------------------------------------
>
>                 Key: AIRFLOW-6824
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-6824
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: aws
>    Affects Versions: 1.10.9
>            Reporter: Bjorn Olsen
>            Assignee: Bjorn Olsen
>            Priority: Minor
>
> EmrAddStepsOperator allows you to add several steps to EMR for processing - 
> the steps must be supplied as a list.
> This works well when passing an actual Python list as the 'steps' value, but 
> we want to be able to generate the list of steps from a previous task - using 
> an XCom.
> We must use the operator as follows, for the templating to work correctly and 
> for it to resolve the XCom:
>  
> {code:java}
> add_steps_task = EmrAddStepsOperator(
>  task_id='add_steps',
>  job_flow_id=job_flow_id,
>  aws_conn_id='aws_default',
>  provide_context=True,
>  steps="{{task_instance.xcom_pull(task_ids='generate_steps')}}"
>  ){code}
>  
> The value in XCom from the 'generate_steps' task looks like (simplified):
> {code:java}
> [{'Name':'Step1'}, {'Name':'Step2'}]
> {code}
> However this is passed as a string to the operator, which cannot be passed to 
> the underlying boto3 library which expects a list object.
> The following won't work either:
> {code:java}
> add_steps_task = EmrAddStepsOperator(
>  task_id='add_steps',
>  job_flow_id=job_flow_id,
>  aws_conn_id='aws_default',
>  provide_context=True,
>  steps={{task_instance.xcom_pull(task_ids='generate_steps')}}
>  ){code}
> Since this is not valid Python.
> We have to pass the steps as a string to the operator, and then convert it 
> into a list after the render_template_fields has happened (immediately before 
> the execute). Therefore the only option is to do the conversion from string 
> to list in the operator's execute method.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Reply via email to