dirrao commented on PR #36492: URL: https://github.com/apache/airflow/pull/36492#issuecomment-1872469573
> > Nice work. Can't we access the variables and connections from the trigger run? What is the recommended pattern for accessing the sensitive information from the trigger run? > > We can - when we instantiate trigger, not when it runs (when it runs it runs in async-io event loop and it can only access disk, networking and any other resources via asynchronous calls - which means that accessign them has to follow async/await semantics. And When Trigger is deserialized by the Triggerer, the deserializing does not run regular `__init__` calll on the Trigger class - it just `deserializes` it from the DB. > > And yes - we could potentially implement some `_post_deserialize` call to retrieve sensitive information from secrets or variables - however in a number of cases this means that Triggerer would have to access **some** authentication information to retrieve it. For example it would have to have the same "account" credentials that worker has or sometimes it might mean that it would have to have access to (say) same kubernetes config or .aws or .gcs authentication in order to retrieve that data. > > While it is possible, it adds additional requirement on `triggerer` process - it would have to have the same authentication information set in its environment as `worker` has. But this is not necessary in most cases - as long as "serialization/deserialization" has all the necessary information. Thanks for the detailed explanation. We usually access secrets from either disk or connections or encrypted variables in airflow. We can access the same in the trigger while it is running the event loop. Check the same in the below S3 Trigger. We usually don't generate new secrets at runtime in the worker and share them with trigger https://github.com/apache/airflow/blob/2bc34ffcb5e830544e024e085f36481a33852f49/airflow/providers/amazon/aws/triggers/s3.py#L86 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
