If I have multiple airflow dags with some overlapping python package
dependencies, how can I keep each of these project deps. decoupled? Eg. if
I had project A and B on same server I would run each of them with
something like...
source /path/to/virtualenv_a/activate
python script_a.py
deactivate
source /path/to/virtualenv_b/activate
python script_b.py
deactivate
Basically, would like to run dags with the same situation (eg. each dag
uses python scripts that have may have overlapping package deps. that I
would like to develop separately (ie. not have to update all code using a
package when want to update the package just for one project)). Note, I've
been using the BashOperator to run python tasks like...
do_stuff = BashOperator(
task_id='my_task',
bash_command='python /path/to/script.py'),
execution_timeout=timedelta(minutes=30),
dag=dag)
Is there a way to get this working? Is there some other best-practice way
that airflow intendeds for people to address (or avoid) these kinds of
problems?
--
This electronic message is intended only for the named
recipient, and may
contain information that is confidential or
privileged. If you are not the
intended recipient, you are
hereby notified that any disclosure, copying,
distribution or
use of the contents of this message is strictly
prohibited. If
you have received this message in error or are not the
named
recipient, please notify us immediately by contacting the
sender at
the electronic mail address noted above, and delete
and destroy all copies
of this message. Thank you.
