This is a problem I have been grappling with for some time, and am intrigued by your proposal! For departments that are under the same "umbrella" department but are comprised of many different decentralized smaller groups, the ability to see and interact with other people's dags via the UI - while also managing their own dependencies - would be a feature, not a bug. The Airflow UI, with descriptive dag and task ids, are a visual cue for the subgroups to speak with one another to leverage each other's work as a base instead of reinventing the wheel.
How would this work if one were using the KubernetesExecutor? Cheers, Constance / CONSTANCE MARTINEAU | Développeuse Principale, Platformes et Exploitation | Tél: 514-847-7992 | comartin...@cdpq.com -----Original Message----- From: Tomasz Urbaszek <turbas...@apache.org> Sent: Friday, January 8, 2021 11:48 AM To: dev@airflow.apache.org Subject: Re: [DISCUSS] [AIP-37] Virtualenv management inside Airflow Thanks Zacharya! I think this is an interesting idea. It seems to be a simple way to add "multi tenancy" to Airflow. However, I'm afraid that having separate venvs for separate tasks (=teams) solves only dependency management problem not managing multiple teams using the same cluster (different teams still can view, delete, update DAGs of other teams and they share the same db). As far as I know there's no "production grade" way to deploy a single Airflow instance that can be shared by teams. Although I know there are ways to achieve that using GKE cluster with multiple instances of Airflow. Now, focusing on your proposition. If we will decide that we want to support such a feature we will need to think about retention policy for the venvs. Having multiple venvs will increase disk/memory allocation and we should try to remove venvs that are no longer used. In general we will need a mechanism to manage those venvs (define, create, update, delete). That said I'm not convinced if Airflow should take care of its environment/deployment. In my opinion this is a users' task to make sure that their tasks are executed in the right environment. And I think this is easily achievable with multiple workers/queues and docker images that can be build on CI/CD systems. Cheers, Tomek On Fri, Jan 8, 2021 at 5:26 PM Zacharya Haitin <zachary...@gmail.com> wrote: > > Hi everyone, > > Currently, there is no easy way to manage an Airflow cluster that contains > multiple teams and python based DAGS. > I submitted the following AIP with a suggestion how to make the venv > management part of the Airflow executors lifecycle and would love to get your > feedback. > > My suggestion should be pretty easy to implement and will help the users with > python packages deployments. > > AIP: > https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwik > i.apache.org%2Fconfluence%2Fdisplay%2FAIRFLOW%2FAIP-37%2BVirtualenv%2B > management%2Binside%2BAirflow&data=04%7C01%7Ccomartineau%40cdpq.co > m%7C23c6c1ef2783438ee9a708d8b3f5326d%7C0bdbe0278f504ec3843fe27c41a6395 > 7%7C1%7C0%7C637457212977264140%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjA > wMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=O > uFDY%2F9PQpYZ7xuW%2BX65GwlQ2OA3HyY8G1GojLQknG8%3D&reserved=0 > Issue: > https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgith > ub.com%2Fapache%2Fairflow%2Fissues%2F13364&data=04%7C01%7Ccomartin > eau%40cdpq.com%7C23c6c1ef2783438ee9a708d8b3f5326d%7C0bdbe0278f504ec384 > 3fe27c41a63957%7C1%7C0%7C637457212977264140%7CUnknown%7CTWFpbGZsb3d8ey > JWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C100 > 0&sdata=Pukf%2FxlUdF%2BCk3Q9NEbNaSEkPBUzh4e3DkntvGY0P0c%3D&res > erved=0 > > Please let me know if you have any questions or suggestions on how to improve > this solution, or if you disagree with my approach. > > Thanks, > Zacharya. ________________________________ Avis de confidentialité : Ce courriel et les pièces qui y sont jointes contiennent de l'information confidentielle et peuvent être protégés par le secret professionnel ou constituer de l’information privilégiée. Ils sont destinés à l'usage exclusif de la (des) personne(s) à qui ils sont adressés. Si vous n'êtes pas le destinataire visé ou la personne chargée de transmettre ce document à son destinataire, vous êtes avisé par la présente que toute divulgation, reproduction, copie, distribution ou autre utilisation de cette information est strictement interdite. Si vous avez reçu ce courriel par erreur, veuillez en aviser immédiatement l’expéditeur par téléphone ainsi que détruire et effacer l'information que vous avez reçue de tout disque dur ou autre média sur lequel elle peut être enregistrée et ne pas en conserver de copie. Merci de votre collaboration. ________________________________ Notice of Confidentiality: This electronic mail message, including any attachments, is confidential and may be privileged and protected by professional secrecy. They are intended for the exclusive use of the addressee. If you are not the intended addressee or the person responsible for delivering this document to the intended addressee, you are hereby advised that any disclosure, reproduction, copy, distribution or other use of this information is strictly forbidden. If you have received this document by mistake, please immediately inform the sender by telephone, destroy and delete the information received from any hard disk or any media on which it may have been registered and do not keep any copy. Thank you for your cooperation.
