SameerMesiah97 commented on code in PR #77069:
URL: https://github.com/apache/airflow/pull/77069#discussion_r4617857751


##########
providers/google/src/airflow/providers/google/cloud/hooks/dataflow.py:
##########
@@ -460,13 +461,33 @@ def fetch_job_id(self) -> str | None:
             self.log.info("Google Cloud DataFlow job not available yet..")
         return self._job_id
 
-    def _fetch_all_jobs(self, jobs_filter: str | None = None) -> list[dict]:
+    def fetch_latest_job_by_name(self, job_name: str) -> dict | None:
+        """
+        Return the latest created job with the given name, in any state.
+
+        Dataflow API does not support getting a single job by name, so the 
jobs in the location
+        are listed with the name passed to the list request.
+
+        :param job_name: Name of the job. It is converted to lowercase because 
Dataflow job names
+            are always lowercase.
+        :return: The job, or None if no job has this name.
+        """
+        job_name = job_name.lower()
+        return max(
+            (job for job in self._fetch_all_jobs(name=job_name) if 
job.get("name") == job_name),

Review Comment:
   Is the if job.get("name") == job_name fitler needed here?



##########
providers/google/tests/unit/google/cloud/hooks/test_dataflow.py:
##########
@@ -1939,6 +1959,19 @@ async def test_list_jobs(self, initialize_client_mock, 
hook, make_mock_awaitable
         initialize_client_mock.assert_called_once()
         client.list_jobs.assert_called_once_with(request=request)
 
+    @pytest.mark.asyncio
+    @mock.patch(DATAFLOW_STRING.format("AsyncDataflowHook.initialize_client"))
+    async def test_list_jobs_with_name(self, initialize_client_mock, hook):

Review Comment:
   Why add a dedicated test for just one additional parameter instead of 
modifying `test_list_jobs`?



##########
providers/google/src/airflow/providers/google/cloud/hooks/dataflow.py:
##########
@@ -1602,10 +1609,17 @@ async def get_latest_job_by_name(
         :return: The job, or None if no job has this name.
         """
         job_name = job_name.lower()
-        jobs = await self.list_jobs(project_id=project_id, location=location)
-        matching_jobs = [job async for job in jobs if job.name == job_name]
-        # create_time is typed as protobuf Timestamp, but proto-plus returns 
it as a datetime.
-        return max(matching_jobs, key=lambda job: cast("datetime", 
job.create_time), default=None)
+        jobs = await self.list_jobs(project_id=project_id, location=location, 
name=job_name)
+        latest_job: Job | None = None
+        async for job in jobs:
+            if job.name != job_name:
+                continue
+            # create_time is typed as protobuf Timestamp, but proto-plus 
returns it as a datetime.
+            if latest_job is None or cast("datetime", job.create_time) > cast(
+                "datetime", latest_job.create_time
+            ):
+                latest_job = job
+        return latest_job

Review Comment:
   This looks messy. The before was much cleaner. Why did  you change it? And 
remove the redundant if  job.name == job_name filter as you are passing the 
job_name to the API.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to