Bowei-Gao-ING opened a new issue, #43253:
URL: https://github.com/apache/airflow/issues/43253

   ### Apache Airflow Provider(s)
   
   google
   
   ### Versions of Apache Airflow Providers
   
   2.9.3
   
   ### Apache Airflow version
   
   Apache Airflow Providers Google 10.12.0
   
   ### Operating System
   
   Ubuntu
   
   ### Deployment
   
   Google Cloud Composer
   
   ### Deployment details
   
   _No response_
   
   ### What happened
   
   We’re trying to use AwaitMessageSensor from Airflow provider to get messages 
from Kafka. AwaitMessageSensor has a parameter “apply_function”. This parameter 
is mandatory, and describes the function is used to process Kafka messages. The 
value of this parameter should be a dotted string and is the path to the 
function.
   
   No matter which path we try, the sensor never found the function.
   
   PYTHONPATH has the following paths:
   /opt/python3.11/bin
   /opt/python3.11/lib/python311.zip
   /opt/python3.11/lib/python3.11
   /opt/python3.11/lib/python3.11/lib-dynload
   /opt/python3.11/lib/python3.11/site-packages
   /home/airflow/gcs/dags
   /etc/airflow/config
   /home/airflow/gcs/plugins
    
   Path of the file: /home/airflow/gcs/dags
   Path the DAG is running in: /home/airflow
   
   We tried different values for the apply_function
   process_messages
   event_based_triggering.process_messages
   dags.event_based_triggering.process_messages
   gcs.dags.event_based_triggering.process_messages
   airflow.gcs.dags.event_based_triggering.process_messages
   home.airflow.gcs.dags.event_based_triggering.process_messages
   
   ### What you think should happen instead
   
   Below is the error in log:
   
   ```
   ERROR - Trigger failed:
   Traceback (most recent call last):
     File 
"/opt/python3.11/lib/python3.11/site-packages/airflow/jobs/triggerer_job_runner.py",
 line 529, in cleanup_finished_triggers
       result = details["task"].result()
                ^^^^^^^^^^^^^^^^^^^^^^^^
     File 
"/opt/python3.11/lib/python3.11/site-packages/airflow/jobs/triggerer_job_runner.py",
 line 602, in run_trigger
       async for event in trigger.run():
     File 
"/opt/python3.11/lib/python3.11/site-packages/airflow/providers/apache/kafka/triggers/await_message.py",
 line 99, in run
       processing_call = import_string(self.apply_function)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File 
"/opt/python3.11/lib/python3.11/site-packages/airflow/utils/module_loading.py", 
line 39, in import_string
       module = import_module(module_path)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/opt/python3.11/lib/python3.11/importlib/__init__.py", line 126, in 
import_module
       return _bootstrap._gcd_import(name[level:], package, level)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
     File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
     File "<frozen importlib._bootstrap>", line 1126, in _find_and_load_unlocked
     File "<frozen importlib._bootstrap>", line 241, in 
_call_with_frames_removed
     File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
     File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
     File "<frozen importlib._bootstrap>", line 1140, in _find_and_load_unlocked
   ModuleNotFoundError: No module named 'name_of_a_module'
   ```
   
   We did some digging and tests -
   If we use the module we created as an import at the top of the DAG file, the 
import statement works, and we are able to call the function in the DAG.
   `from util import my_process`
    
   But, in the AwaitMessageSensor, it uses 
importlib.import_module("util.my_package") which does not work.
   
   We have found a workaround for our problem. In ConsumeFromTopicOperator, we 
can use both dotted string and callable as the value of "apply_function". The 
apply_function of ConsumeFromTopicOperator works fine. Why not do the same for 
AwaitMessageSensor? When you compare ConsumeFromTopicOperator with 
AwaitMessageTrigger, you will see that the change to be made is only:
   
   ```
   if isinstance(self.apply_function, str):
       process_message = import_string(self.apply_function)
   else:
       process_message self.apply_function
   ```
   
   Instead of:
   ```
   process_message = import_string(self.apply_function)
   ```
   
   And also need to change the type definition of apply_function in 
AwaitMessageSensor and AwaitMessageTrigger. Overall, the change is small.
   
   ### How to reproduce
   
   Use AwaitMessageSensor in Gooogle Cloud Composer and the problem will 
produce.
   
   ### Anything else
   
   The problem happens every time.
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to