[GitHub] [airflow] nuclearpinguin commented on a change in pull request #6898: [AIRFLOW-6432] fix EmrAddStepsOperator broken ref & faulty test & add test case for missing cluster

2019-12-30 Thread GitBox 
nuclearpinguin commented on a change in pull request #6898: [AIRFLOW-6432] fix 
EmrAddStepsOperator broken ref & faulty test & add test case for missing cluster
URL: https://github.com/apache/airflow/pull/6898#discussion_r362162056
 
 

 ##
 File path: tests/contrib/operators/test_emr_add_steps_operator.py
 ##
 @@ -107,23 +108,45 @@ def test_execute_returns_step_id(self):
 def test_init_with_cluster_name(self):
 expected_job_flow_id = 'j-1231231234'
 
-self.emr_client_mock.get_cluster_id_by_name.return_value = 
expected_job_flow_id
 self.emr_client_mock.add_job_flow_steps.return_value = 
ADD_STEPS_SUCCESS_RETURN
 
 with patch('boto3.session.Session', self.boto3_session_mock):
+with 
patch('airflow.contrib.hooks.emr_hook.EmrHook.get_cluster_id_by_name') \
+as mock_get_cluster_id_by_name:
+mock_get_cluster_id_by_name.return_value = expected_job_flow_id
+
+operator = EmrAddStepsOperator(
+task_id='test_task',
+job_flow_name='test_cluster',
+cluster_states=['RUNNING', 'WAITING'],
+aws_conn_id='aws_default',
+dag=DAG('test_dag_id', default_args=self.args)
+)
+
+operator.execute(self.mock_context)
+
+ti = self.mock_context['ti']
+
+ti.xcom_push.assert_called_once_with(key='job_flow_id', 
value=expected_job_flow_id)
+
+def test_init_with_nonexistent_cluster_name(self):
+cluster_name = 'test_cluster'
+
+with 
patch('airflow.contrib.hooks.emr_hook.EmrHook.get_cluster_id_by_name') \
+as mock_get_cluster_id_by_name:
+mock_get_cluster_id_by_name.return_value = None
+
 operator = EmrAddStepsOperator(
 task_id='test_task',
-job_flow_name='test_cluster',
+job_flow_name=cluster_name,
 cluster_states=['RUNNING', 'WAITING'],
 aws_conn_id='aws_default',
 dag=DAG('test_dag_id', default_args=self.args)
 )
 
-operator.execute(self.mock_context)
-
-ti = self.mock_context['ti']
-
-ti.xcom_push.assert_any_call(key='job_flow_id', 
value=expected_job_flow_id)
+with self.assertRaises(AirflowException) as error:
+operator.execute(self.mock_context)
+self.assertEqual(str(error.exception), 'No cluster found for name 
= test_cluster')
 
 Review comment:
   ```suggestion
   self.assertEqual(str(error.exception), f'No cluster found for 
name:  {cluster_name}')
   ```
   The test need an adjustment after the change in core :)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] nuclearpinguin commented on a change in pull request #6898: [AIRFLOW-6432] fix EmrAddStepsOperator broken ref & faulty test & add test case for missing cluster

2019-12-28 Thread GitBox 
nuclearpinguin commented on a change in pull request #6898: [AIRFLOW-6432] fix 
EmrAddStepsOperator broken ref & faulty test & add test case for missing cluster
URL: https://github.com/apache/airflow/pull/6898#discussion_r361808039
 
 

 ##
 File path: airflow/contrib/operators/emr_add_steps_operator.py
 ##
 @@ -68,12 +68,16 @@ def __init__(
 self.steps = steps
 
 def execute(self, context):
-emr = EmrHook(aws_conn_id=self.aws_conn_id).get_conn()
+emr_hook = EmrHook(aws_conn_id=self.aws_conn_id)
+
+emr = emr_hook.get_conn()
 
 job_flow_id = self.job_flow_id
 
 if not job_flow_id:
-job_flow_id = emr.get_cluster_id_by_name(self.job_flow_name, 
self.cluster_states)
+job_flow_id = emr_hook.get_cluster_id_by_name(self.job_flow_name, 
self.cluster_states)
+if not job_flow_id:
 
 Review comment:
   There is duplicated check, what about:
   ```python
   job_flow_id = job_flow_id  or 
emr_hook.get_cluster_id_by_name(self.job_flow_name, self.cluster_states)
   if not job_flow_id:
   raise...
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] nuclearpinguin commented on a change in pull request #6898: [AIRFLOW-6432] fix EmrAddStepsOperator broken ref & faulty test & add test case for missing cluster

2019-12-28 Thread GitBox 
nuclearpinguin commented on a change in pull request #6898: [AIRFLOW-6432] fix 
EmrAddStepsOperator broken ref & faulty test & add test case for missing cluster
URL: https://github.com/apache/airflow/pull/6898#discussion_r361807913
 
 

 ##
 File path: airflow/contrib/hooks/emr_hook.py
 ##
 @@ -51,11 +51,12 @@ def get_cluster_id_by_name(self, emr_cluster_name, 
cluster_states):
 
 if len(matching_clusters) == 1:
 cluster_id = matching_clusters[0]['Id']
-self.log.info('Found cluster name = %s id = %s' % 
(emr_cluster_name, cluster_id))
+self.log.info(f'Found cluster name = {emr_cluster_name} id = 
{cluster_id}')
 return cluster_id
 elif len(matching_clusters) > 1:
-raise AirflowException('More than one cluster found for name = %s' 
% emr_cluster_name)
+raise AirflowException(f'More than one cluster found for name 
{emr_cluster_name}')
 else:
+self.log.info(f'No cluster found for name {emr_cluster_name}')
 
 Review comment:
   ```suggestion
   self.log.info('No cluster found for name %s', emr_cluster_name)
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] nuclearpinguin commented on a change in pull request #6898: [AIRFLOW-6432] fix EmrAddStepsOperator broken ref & faulty test & add test case for missing cluster

2019-12-28 Thread GitBox 
nuclearpinguin commented on a change in pull request #6898: [AIRFLOW-6432] fix 
EmrAddStepsOperator broken ref & faulty test & add test case for missing cluster
URL: https://github.com/apache/airflow/pull/6898#discussion_r361807954
 
 

 ##
 File path: airflow/contrib/operators/emr_add_steps_operator.py
 ##
 @@ -68,12 +68,16 @@ def __init__(
 self.steps = steps
 
 def execute(self, context):
-emr = EmrHook(aws_conn_id=self.aws_conn_id).get_conn()
+emr_hook = EmrHook(aws_conn_id=self.aws_conn_id)
+
+emr = emr_hook.get_conn()
 
 job_flow_id = self.job_flow_id
 
 if not job_flow_id:
-job_flow_id = emr.get_cluster_id_by_name(self.job_flow_name, 
self.cluster_states)
+job_flow_id = emr_hook.get_cluster_id_by_name(self.job_flow_name, 
self.cluster_states)
+if not job_flow_id:
+raise AirflowException('No cluster found for name = %s' % 
self.job_flow_name)
 
 Review comment:
   ```suggestion
   raise AirflowException(f'No cluster found for name: 
{self.job_flow_name}')
   ```
   But here we can use f-strings ✅


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] nuclearpinguin commented on a change in pull request #6898: [AIRFLOW-6432] fix EmrAddStepsOperator broken ref & faulty test & add test case for missing cluster

2019-12-28 Thread GitBox 
nuclearpinguin commented on a change in pull request #6898: [AIRFLOW-6432] fix 
EmrAddStepsOperator broken ref & faulty test & add test case for missing cluster
URL: https://github.com/apache/airflow/pull/6898#discussion_r361807885
 
 

 ##
 File path: airflow/contrib/hooks/emr_hook.py
 ##
 @@ -51,11 +51,12 @@ def get_cluster_id_by_name(self, emr_cluster_name, 
cluster_states):
 
 if len(matching_clusters) == 1:
 cluster_id = matching_clusters[0]['Id']
-self.log.info('Found cluster name = %s id = %s' % 
(emr_cluster_name, cluster_id))
+self.log.info(f'Found cluster name = {emr_cluster_name} id = 
{cluster_id}')
 
 Review comment:
   ```suggestion
   self.log.info('Found cluster name = %s id = %s', 
emr_cluster_name, cluster_id)
   ```
   
   In case of logging we do formatting this way.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] nuclearpinguin commented on a change in pull request #6898: [AIRFLOW-6432] fix EmrAddStepsOperator broken ref & faulty test & add test case for missing cluster

2019-12-28 Thread GitBox 
nuclearpinguin commented on a change in pull request #6898: [AIRFLOW-6432] fix 
EmrAddStepsOperator broken ref & faulty test & add test case for missing cluster
URL: https://github.com/apache/airflow/pull/6898#discussion_r361807897
 
 

 ##
 File path: airflow/contrib/hooks/emr_hook.py
 ##
 @@ -51,11 +51,12 @@ def get_cluster_id_by_name(self, emr_cluster_name, 
cluster_states):
 
 if len(matching_clusters) == 1:
 cluster_id = matching_clusters[0]['Id']
-self.log.info('Found cluster name = %s id = %s' % 
(emr_cluster_name, cluster_id))
+self.log.info(f'Found cluster name = {emr_cluster_name} id = 
{cluster_id}')
 return cluster_id
 elif len(matching_clusters) > 1:
-raise AirflowException('More than one cluster found for name = %s' 
% emr_cluster_name)
+raise AirflowException(f'More than one cluster found for name 
{emr_cluster_name}')
 
 Review comment:
   ```suggestion
   raise AirflowException('More than one cluster found for name 
%s', emr_cluster_name)
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services