Thanks Jarek and Shubham for the clarifications. Looking forward to this one!
Thanks & Regards, Amogh Desai On Sat, Mar 16, 2024 at 10:10 AM Mehta, Shubham <shu...@amazon.com.invalid> wrote: > Jarek - I totally agree. We had a similar conversation in Dec 2022, and > Filip K. from Google suggested [1] calling them "workspaces." But I think > most of our users and contributors are used to the word "tenant." Trying a > new term like "workspaces" just seems to make things more confusing. For > example, when I tried using it with a couple of developers at AWS, who were > somewhat familiar with Airflow, it immediately prompted questions about its > relation to "tenants." > > I also liked how you explained it in the AIP response. It's like when > Kubernetes talks about "multi-tenant" [2] and they mean it could be for > different customers or different teams. What we’re doing with Airflow is > for teams, not really different customers. But it's simpler to keep calling > it "multi-tenant," just like Kubernetes does, and make sure we explain that > we're talking about different teams. > > Reference: > 1. > https://docs.google.com/document/d/1n23h26p4_8F5-Cd0JGLPEnF3gumJ5hw3EpwUljz7HcE/edit?disco=AAAAlo3bv6Q > 2. https://kubernetes.io/docs/concepts/security/multi-tenancy/ > > Shubham > > On 2024-03-15, 2:50 AM, "Jarek Potiuk" <ja...@potiuk.com <mailto: > ja...@potiuk.com>> wrote: > > > CAUTION: This email originated from outside of the organization. Do not > click links or open attachments unless you can confirm the sender and know > the content is safe. > > > > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que > le contenu ne présente aucun risque. > > > > > > > Thaks Shubham - that's a really nice and succinct description of the vision > behind tht 'Multi-tenant deployment' AIP. The case you described is spot-on > where I see there is enough value for the organisation that would like to > create such multi-tenant deployment of Airflow. > > > And I think one comment fro TP in the AIP that summaries it well - this is > not really `Making Airflow multi-tenant'. It's merely enabling airflow > components to be put together in a single multi-tenant deployment - > consisting with a number of smaller sub-deployments that achieve a nice > IMHO way of being able to decompose monolithic Airflow installation Ina a > set of loosely cooperating 'sub-deployments' - achieving some (interesting) > centralisation property - while giving enough freedom for tenants to manage > their part separately > > > In a way it might be a response to the claims that airflow is this old > monolithic style application and it's not the new, cool micro-service > /serverless world. Which I think trades some problems with a set of other > problems mainly around managing all those micro-sevices to cooperate with > each other and introduces a lot of performance and service management > problems. > > > But with the architecture described here it is a bit connecting best things > if both worlds - splitting Airflow into 'midi-services' - where the > architecture of your deploynent nicely follows Conway's Law and > decomposition is done at the right boundaries - boundaries where Tenant > Deployment Managers might claim as 'theirs' while the 'Airflow Platform > Team' keeps the whole Airflow deployment under control because there are > clear boundaries what the Platform team is responsible for, and where each > Tenant has their own 'kingdom'. > > > I think once we get this one approved and some deployments out there in the > wild - we will be able to claim that the architecture is actually the next > gen after monolithic, micro-sevices and serverless - bit done better, > taking into account some real-life user scenarios and Making it relatively > easy to adopt such deployment because of Conway's Law limitations. > > > I thought a lot about Conway's Law playing a role in this whole design and > I think your Business Case, Shubham, is a perfect reflection of how well > the proposes architecture fits in the Business structures of organisation > that would like to deploy it. > > > J. > > > czw., 14 mar 2024, 20:21 użytkownik Mehta, Shubham > <shu...@amazon.com.inva <mailto:shu...@amazon.com.inva>lid> napisał: > > > > Hi folks, > > > > Firstly, thanks Jarek for putting together such a thorough and > > well-thought-out proposal. > > > > > > > > I am very much in support of the multi-tenancy proposal. Having discussed > > this with over 30 customers (AWS and non-AWS), there's a clear desire to > > shift focus from the complex management of multiple Airflow environments > to > > enhancing their capabilities, such as enabling data quality checks and > > lineage. This proposal is a significant step towards achieving that goal. > > > > Acknowledging that not every Airflow user has enough time to thoroughly > > review the AIP, I have drafted a user scenario that encapsulates what's > > possible with the implementation of multi-tenancy support: > > > > *---- Scenario: Multi-Tenancy in Apache Airflow at [Rocket] ----* > > > > [Rocket], a leading [mobile gaming platform], has adeptly structured its > > cloud operations using Apache Airflow to provide an efficient and secure > > multi-tenant environment for orchestrating their complex workflows. This > > approach caters to the diverse needs of their three main user groups: the > > Data Engineering team, the Data Science team, and the Data Analytics > team. > > > > All teams share basic Airflow components like the Scheduler and > Webserver, > > providing centralized management with shared cost. Each team has its own > > distinct tenant cluster, offering self-sufficiency, flexibility, and > > isolation. The Data Engineering team builds ETL/ELT pipelines and > produces > > user profile, telemetry, and marketing data. The Analytics team works > with > > marketing data and user information to build comprehensive dashboards. > The > > Data Science team uses Kubernetes as their execution environment for > > heavy-duty machine learning tasks, producing a churn prediction dataset. > > > > Members of each team can only see and work with their own workflows. > > However, Data engineers are granted access to all tenants, enabling them > to > > assist with DAG troubleshooting and optimization across all teams. Upon > > logging in, users are presented with a tenant-specific view, displaying > > only the relevant DAGs and artifacts. For those with multi-tenant access, > > seamless navigation between different tenant views is available without > the > > need for re-authentication. > > > > > > This setup lets each team work independently with their own tools and > > data, while also getting help from data engineers when needed. It's > secure, > > efficient, and user-friendly. > > > > *Image:* https://imgur.com/gallery/uQNqiVc < > https://imgur.com/gallery/uQNqiVc> (highly recommend reviewing > > the image to understand the underlying setup) > > > > > *-----------------------------------------------------------------------------------* > > > > I’d suggest that interested Airflow users review the scenario and share > > your support or concerns on this concept in this thread or AIP. For those > > interested in diving deeper into the details, the AIP is available here - > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-tenant+deployment+of+Airflow+components > < > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-tenant+deployment+of+Airflow+components > > > > > > > > Thanks > > Shubham > > Product Manager - Amazon MWAA > > > > > > > > *From: *Jarek Potiuk <ja...@potiuk.com <mailto:ja...@potiuk.com>> > > *Reply-To: *"us...@airflow.apache.org <mailto:us...@airflow.apache.org>" > <us...@airflow.apache.org <mailto:us...@airflow.apache.org>> > > *Date: *Monday, March 11, 2024 at 4:05 PM > > *To: *"dev@airflow.apache.org <mailto:dev@airflow.apache.org>" < > dev@airflow.apache.org <mailto:dev@airflow.apache.org>>, " > > us...@airflow.apache.org <mailto:us...@airflow.apache.org>" < > us...@airflow.apache.org <mailto:us...@airflow.apache.org>> > > *Subject: *RE: [EXTERNAL] [COURRIEL EXTERNE] [DISCUSS] DRAFT AIP-67 > > Multi-tenant deployment of Airflow components > > > > > > > > *CAUTION*: This email originated from outside of the organization. Do not > > click links or open attachments unless you can confirm the sender and > know > > the content is safe. > > > > > > > > *AVERTISSEMENT*: Ce courrier électronique provient d’un expéditeur > > externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si > vous > > ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas > > certain que le contenu ne présente aucun risque. > > > > > > > > I have iterated and already got a LOT of comments from a LOT of people > > (Thanks everyone who spent time on it ). I'd say the document is out of > > draft already, it very much describes the idea of multi-tenancy that I > hope > > we will be voting on some time in the future. > > > > > > > > Taking into account that ~ 30% of people in our survey said they want > > "mutl-tenancy" - what I am REALLY interested in is to get honest feedback > > about the proposal. Manly: > > > > > > > > **"Is this the multi-tenancy you were looking for?" * > > > > > > > > Or were you looking for different droids (err, tenants) maybe?. > > > > > > > > I do not want to exercise my Jedi skills to influence your opinion, > that's > > why the document is there (and some people say it's nice, readable and > > pretty complete) so that you can judge yourself and give feedback. > > > > > > > > The document is here: > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-tenant+deployment+of+Airflow+components > < > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-tenant+deployment+of+Airflow+components > > > > > > > > > > > > Feel free to comment here, or in the document. I would love to hear more > > voices, and have some ideas what to do next to validate the idea, so > please > > - engage for now - but also expect some follow-ups. > > > > > > > > J. > > > > > > > > > > > > On Wed, Mar 6, 2024 at 9:16 AM Jarek Potiuk <ja...@potiuk.com <mailto: > ja...@potiuk.com>> wrote: > > > > Sooo.. Seems that it's an AIP time :D I've just published a Draft of > > AIP-67: > > > > Multi-tenant deployment of Airflow components > > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/%5BDRAFT%5D+AIP-67+Multi-tenant+deployment+of+Airflow+components > < > https://cwiki.apache.org/confluence/display/AIRFLOW/%5BDRAFT%5D+AIP-67+Multi-tenant+deployment+of+Airflow+components > > > > > > This AIP is a bit lighter in detail than the others you could see > > from Jed , Nikolas and Maciej. This is really a DRAFT / High Level > > idea of Multi-Tenancy that could be implemented as the follow-up after > > previous steps of Multi-Tenancy implemented (or being implemented) > > right now. > > > > I decided to - rather than describe all the details now - focus on > > the concept of Multitenancy that I wanted to propose. Most of all > > explaining the concept, comparing it to current ways of achieving some > > forms of multi-tenancy and showing benefits and drawbacks of the > > solution and connected costs (i.e. what complexity we need to add to > > achieve it). > > > > When thinking about Multi-tenancy, I realized few things: > > > > * everyone might understand multi-tenancy differently > > * some forms of multi-tenancy are achievable even today > > * but - most of all - I started to question myself "Is this what we > > can do, enough for some, sufficiently numerous groups of users to call > > it a useful feature for them". > > > > So before we get into more details - my aim is to make sure we are all > > at the same page on what we CAN do as a multi-tenancy, and eventually > > to decide whether we SHOULD do it. > > > > Have fun. Bring in comments and feedback. > > > > More about all the currently active AIPs at today's Town Hall > > > > BTW. Do you think it's a surprise that 5 AIPS were announced just > > before the Town Hall? I think not :D > > > > J. > > > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > For additional commands, e-mail: dev-h...@airflow.apache.org >
