Hi all,
I'm doing some work to get Google Cloud Logging into Airflow, but without
it actually being implemented into Airflow. For this I used a simple
pluggable discovery system using standard Python setuptools. Here is what
I'm talking about:
In Airflow I reworked all longing config in an AirflowInternalLogging
component. The AirflowInternalLogging is written as a discoverable
compoment and needed to be added to the setup.py:
entry_points={
'airflow.logging': [
'AirflowInternalLogging = airflow.defaults:AirflowInternalLogging'
]
That's all that is needed to make it discoverable. In the code I just need
to load it via:
def load_airflow_component(group, name):
from pkg_resources import iter_entry_points
print "Discovery of '{}' components".format(group)
for entry_point in iter_entry_points(group=group, name=None):
if entry_point.name == name:
component = entry_point
print("- {}".format(entry_point))
print "Loading '{}' for '{}'".format(name, group)
print component
instance = component.load()
return instance()
That's it. Now in a completly other project with it's own setup I can add
other implementations (with the same methods) of the component and make
them selectable via the config of that project:
setup(name='airflow_plugin.google_cloud',
version='0.1.0',
packages=['airflow_plugin.google_cloud'],
entry_points={
'airflow.logging': [
'GoogleCloudLogging =
airflow_plugin.google_cloud:GoogleCloudLogging'
]
})
I would generatlise this so we could make other parts (like for example
connections) also pluggable.
What do you think. Seems to work fine so far... and doesn't add any
dependencies.
Thoughts?
--
_/
_/ Alex Van Boxel
