Airflow does not have a built-in mechanism, but if you have the opportunity to create a contribution then you should look in the direction of BaseTaskRunner.
On Thu, Oct 17, 2019 at 9:57 PM Reed Villanueva <[email protected]> wrote: > > Is there a way to have dags use separate environ and sys.path values? > > For example, I have files dag1.py and dag2.py that both want to import > modules and run files as tasks from directories /files/for/dag1 and > /files/for/dag2. The files are named the same in both dirs but do very > different things. Currently, if I set environ (eg. PROJECT_HOME) and sys.path > in one dag file, I see that it affects all other dag files in ~/airflow/dag. > This causes problems / mixups when trying to import modules or reference > files in dirs where files have same names. > > Is there a way to get this working without using packaged dags? Can't use > packaged dags, since have some bash scripts that run as tasks that use > environ variables (like PROJECT_HOME) to reference local dirs for other > things and as I understand it a packaged dag will only help co-locating the > files, but not isolate env vars. > > This electronic message is intended only for the named > recipient, and may contain information that is confidential or > privileged. If you are not the intended recipient, you are > hereby notified that any disclosure, copying, distribution or > use of the contents of this message is strictly prohibited. If > you have received this message in error or are not the named > recipient, please notify us immediately by contacting the > sender at the electronic mail address noted above, and delete > and destroy all copies of this message. Thank you.
