DrYampy commented on issue #36963:
URL: https://github.com/apache/airflow/issues/36963#issuecomment-2669566660

   IF you want to change the 
[CloudRunExecuteJobOperator](https://github.com/apache/airflow/blob/0adb5ad1cbee03309ffeffa91ec61722723713f1/providers/google/src/airflow/providers/google/cloud/operators/cloud_run.py#L352)
 to actually fetch and print the logs - this does so after completion:
   
   ```
       def _wait_for_operation(self, operation: 'operation.Operation'):
           try:
               return operation.result(timeout=self.timeout_seconds)
           except Exception:
               error = operation.exception(timeout=self.timeout_seconds)
               raise AirflowException(error)
           finally:
               
               # This was a process to fetch logs from Google Cloud Logging - 
it hits the GCP project quota for 60 logging calls/min
               # This can be raised with qupta management, however I opted for 
just a link to the logs in both Datadog and Logging 
               
               from google.cloud import logging_v2
               client = logging_v2.Client(project=self.project_id)
               execution_name = operation.metadata.name.split('/')[-1]
               print(f"Isolated Execution ID: {execution_name}")
               
               # Define log filter for the specific Cloud Run job execution 
(run)
               filter_str = f"""
                   resource.type="cloud_run_job"
                   resource.labels.project_id="{self.project_id}"
                   resource.labels.location="{self.region}"
                   resource.labels.job_name="{self.job_name}"
                   labels."run.googleapis.com/execution_name"="{execution_name}"
               """
               print(f"Full Filter String:\n{filter_str}")
                       
               # Execute the query to fetch logs
               print(f"EXECUTION LOGS BELOW!")
               log_entries = []
               entries = client.list_entries(filter_=filter_str)
               for entry in entries:
                   if hasattr(entry, "payload"):
                       if isinstance(entry.payload, str):
                           log_entries.append(entry.payload)
                       elif isinstance(entry.payload, OrderedDict):
                           
log_entries.append(entry.payload["status"]["message"])
                       else:
                           log_entries.append(
                               "CORNER: Entry Payload is not a String or 
OrderedDict:"
                           )
                           log_entries.append(entry.payload)
                   else:
                       log_entries.append("CORNER: No Payload in Log Entry:")
                       log_entries.append(entry)
   
               # Print log entries to Airflow task logs
               for log_entry in log_entries:
                   print(log_entry)
   ```
   But again, I dont recommend this bevcause if you consider using this 
operator at scale where you need to fetch more than 60 executions a minute, 
youre gonna run afoul of the log api quota, and you will need to request a 
raise on it


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to