Hello Everyone, I would like to call for a meeting where we can talk about the multi-tenancy AIP (AIPs?)
I prepared a doodle to find a good time next week https://doodle.com/poll/nrfaqbmcwfqqhbsw?utm_source=poll&utm_medium=link The initial agenda I plan for now: 1) I would like to start the kinds of "isolation" and various "scopes" of multi-tenancy 2) We will have a "draft" review of the example architecture that needs the "multitenancy" - this is from the Google Composer team - Mateusz Henc will walk through the composer team goals. That will give us a well thought and discussed "use" of the multitenancy features. 3) Possibly others (Ian? anyone else?) would like to get their case laid out similarly (maybe just focusing on what's different - because after studying earlier discussions I think we all have similar ideas on what needs to be done). 4) Discussing proposals on the scope of the AIP(s) we want to write and rough approach we can take for implementation and initial commitment on who will do what Please mark your good time in the doodle, or if the time is not great and you would like to propose other times - let us know here. J. On Mon, Nov 15, 2021 at 11:11 PM Jarek Potiuk <[email protected]> wrote: > > Cool. I will propose next steps tomorrow most likely. > > On Sat, Nov 13, 2021 at 12:37 PM Ian Buss <[email protected]> wrote: > > > > Thanks Jarek! Will join the channel. This is still something close to my > > heart and an area where I think we could make great progress. I agree that > > we should get together and start a draft AIP around this to gather the > > various strands of this together. > > On 7 Nov 2021, 13:19 +0000, Jarek Potiuk <[email protected]>, wrote: > > > > For now I created a channel for the SIG: > > https://apache-airflow.slack.com/archives/C02M551UDA4 - feel free to > > join anyone and in the next weeks once all the people involved so far > > expressed their interest, we should set some plan on getting the > > AIP(s)? drafted/discussed and start implementing it. > > > > On Sat, Nov 6, 2021 at 11:22 PM Xinbin Huang <[email protected]> wrote: > > > > > > Hi Jarek, > > > > The plan sounds great! And +1 to a special interest group. Please add me to > > the group if you do create one. > > > > Here is the doc ( Airflow Multi-tenancy discussion ) we used to discuss > > back in April. It's not a note per-se, but I think it can shed some light > > on what we talked about. Other folks may have an actual note or even a > > draft proposal on this topic. > > > > I'm excited for us to move forward with this. > > > > Bin > > > > On Fri, Nov 5, 2021 at 10:38 AM Jarek Potiuk <[email protected]> wrote: > > > > > > Hello Ian, Everyone, > > > > I wonder if there are any notes from the meeting in April? Has there > > been any more work on that one from Cloudera to formalize and plan > > work on it? > > > > I was not able to participate, but I think it's about the time to > > seriously start work on that and I am super happy to take more lead on > > this project and involve all the interested parties. The ideas > > described in the email and discussed after are I think super > > reasonable and definitely necessary to get to the multi-tenancy and I > > believe that there are already ideas that can be turned into reality > > rather soon. I had a talk today also with the Google Composer team and > > they are also fully on board with dedicating a lot of effort on this > > one (and their ideas are I think super-aligned with Cloudera's), so I > > think we have a critical mass and engineering power to make it happen > > :) > > > > I plan to put quite a lot of focus on that one over the coming months > > and I am happy to lead or co-lead the AIP and take a big part in > > implementation. > > > > Possibly we should create a special interest group around that and > > start drafting the AIP proposals in a smaller group of people who are > > interested and start planning the work. I already have some ideas > > where we could start gradually implementing it (of course after we > > prepare the AIP and get it through the community's approval process). > > > > How does it sound? > > > > J. > > > > On Wed, Apr 21, 2021 at 8:56 AM Ian Buss <[email protected]> wrote: > > > > > > Yes, no invite required. See you tomorrow! > > On 21 Apr 2021, 07:46 +0100, Sumit Maheshwari <[email protected]>, wrote: > > > > I'll join as well (I believe the zoom link will work without an invite) > > > > On Wed, Apr 21, 2021 at 10:48 AM Dimitris Stafylarakis <[email protected]> > > wrote: > > > > > > hi all, > > > > great to read about this, I'd like to join in! Can I just join using the > > zoom link tomorrow or do I need an invitation? (If I do need one, please > > invite me :)) > > > > cheers > > > > > > On Wed, Apr 14, 2021 at 8:15 PM Daniel Imberman <[email protected]> > > wrote: > > > > > > Thank you Ian, > > > > I’ve invited everyone on this thread to the meeting with that zoom link. > > Anyone else who wants to join can add the calendar event here > > calendar.google.com/event?action=TEMPLATE&tmeid=Mm4zN2Q3MnFwNnBqbW9hMmNocXMyNzJpdHYgZGFuaWVsQGFzdHJvbm9tZXIuaW8&[email protected] > > > > On Wed, Apr 14, 2021 at 11:05 AM, Ian Buss <[email protected]> wrote: > > > > If this works for everyone, here's a zoom link for Thursday 8AM PST: > > https://cloudera.zoom.us/j/99928254235?pwd=VTFlQk4vQjQ5Z2JzUDM3ZWZKKy9MQT09 > > > > Happy to move or use an alternate method as needed. > > > > On Wed, Apr 14, 2021 at 6:58 PM Daniel Imberman <[email protected]> > > wrote: > > > > > > Thursday works for me! > > > > On Wed, Apr 14, 2021 at 10:05 AM, Ian Buss <[email protected]> wrote: > > > > Hi all, > > > > I actually can’t do Wednesday next week as I’m moving house :) Any chance > > we could do Thursday or Friday at the same time? > > > > Cheers > > > > Ian > > On 14 Apr 2021, 17:49 +0100, Kaxil Naik <[email protected]>, wrote: > > > > Just few comments here: > > > > Currently -- atleast for the foreseeable future Airflow workers will need > > access to the DAG Files, so workers can not run using the Serialized DAGs. > > > > Also serialized DAGs do not even have all the info needed for it to run it. > > Currently the serialization happens in the parsing process in the scheduler > > which can be in future separated as a separator "parsining" component, but > > that won't solve the "isolation" problem you are trying to solve. The only > > current way it can be solved is pickling -- and we have strictly decided > > against using pickling for DAGs. > > > > The idea in Statement (2) & (3) would help solve the isolation problem in > > (1) and can be done with some work now. > > > > Happy to talk about it in more detail here or on call, the time Daniel > > suggested works for me. > > > > Regards, > > Kaxil > > > > On Wed, Apr 14, 2021 at 5:35 PM Daniel Imberman <[email protected]> > > wrote: > > > > > > How about Wednesday, April 21 at 8:00AM PST? > > > > On Wed, Apr 14, 2021 at 9:33 AM, Xinbin Huang <[email protected]> wrote: > > > > I am available any days. > > > > On Wed, Apr 14, 2021, 9:32 AM Daniel Imberman <[email protected]> > > wrote: > > > > > > Hi everyone! > > > > Would people be available around 8AM/9AM PST some point next week? I’m in > > PST and Ian is UTC+1 so would be great to find a timezone that accomodates > > everyone. > > > > Daniel > > On Wed, Apr 14, 2021 at 6:26 AM, Ryan Hatter <[email protected]> wrote: > > > > I’d also like to be added please :) > > > > On Apr 13, 2021, at 21:27, Xinbin Huang <[email protected]> wrote: > > > > > > Hi Daniel & Ian, > > > > I am also interested in the idea of a serialization representation that can > > be executed by workers directly. Can you also add me to the call? > > > > Thanks > > Bin > > > > On Tue, Apr 13, 2021 at 2:49 PM Ian Buss <[email protected]> wrote: > > > > > > Daniel, > > > > Thanks for your warm welcome and quick response and the advice on > > providers! Will certainly check out the examples you sent. > > > > 1. An "airflow register" command definitely sounds promising, would love to > > collaborate on an AIP there so let's set something up. > > 2. We use KubernetesExecutor exclusively as well. We've noticed significant > > additional load on the metadata DB as we scale up task pods so I've also > > thought about an API-based approach. Such an API could also open up the > > possibility of per-task security tokens which are injected by the > > scheduler, which should improve the security of such a system. Food for > > thought at least. I will start putting some of these thoughts down on paper > > in a sharable format. > > > > Ian > > > > On Tue, Apr 13, 2021 at 7:46 PM Daniel Imberman <[email protected]> > > wrote: > > > > > > Hi Ian, > > > > > > Firstly, welcome to the Airflow community :). I'm glad to hear you've had a > > positive experience so far. It's great to hear that you want to contribute > > back, and I think that multi-tenancy/DAG isolation is a pretty fantastic > > project for the community as a whole (a lot of things are are things we > > want but are limited by hours in a day). > > > > > > 1. I've personally been kicking around some ideas lately about an "airflow > > register" command that would write the DAG into the metadata DB in a way > > that could be "gettable" by the workers via the API. This work is very > > early. I'd love to get some help on it. Perhaps we can set up a zoom chat > > to discuss drafting an AIP? > > > > > > 2. Limiting worker access to the DB is not only good security practice; it > > also opens up the door to a lot of valuable features. This feature would be > > especially close to my heart as it would make the KubernetesExecutor > > significantly more efficient. It should be possible to set up a system > > where the workers only ever speak to an API server and never need to touch > > the DB. > > > > > > 3. This is not something I personally have insight into, but I think it > > sounds like a good idea. > > > > > > Finally, addressing your question about a Cloudera provider. If anything, > > it would probably give the provider _more_ legitimacy if you hosted it > > under the Cloudera GitHub org (we very purposely created the provider > > packages with this workflow in mind). There are multiple places where we > > can work to surface this provider so it is easy to find and use. > > > > > > Astronomer has a pretty good sample provider here. One example of it > > running in the wild is the Great Expectations provider here. I'd also be > > glad to get you in contact with people who have built providers in the past > > to help you with that process. > > > > > > Looking forward to seeing some of these things come to fruition! > > > > > > Daniel > > > > > > On Tue, Apr 13, 2021 at 9:43 AM, Ian Buss <[email protected]> wrote: > > > > Hi all, > > > > First a quick introduction: I'm an engineer with Cloudera working on our > > Data Engineering product (CDE). Airflow is working great for us so far. > > We've been looking into how we can enhance the multi-tenancy story of > > Apache Airflow as we currently deploy it. We have the following areas which > > we'd like (with community consensus) to work on and contribute back to > > Apache Airflow to enhance the isolation between tenants in a single Airflow > > deployment. > > > > 1. Isolating code execution and parsing of DAG files. At the moment, DAG > > files are parsed in a few locations in Airflow, including the scheduler and > > in tasks. There is already the concept of DAG serialization (and we're > > using that for the web component) but we'd be interested to see if we can > > sandbox the execution of arbitrary user code to a locked down > > process/container without full access to the metadata DB and connection > > secrets etc. The idea would be to parse and serialize the DAG in this > > isolated container and pass back a serialized representation for > > persistence in the DB. Has anyone explored this idea? > > > > 2. Limiting task access to the metadata DB. It would be great if we could > > remove the requirement for tasks to have full access to the metadata DB and > > to report task status in a different (but still scalable) way. We'd need to > > tackle access or injection of connection, variable and xcom data as well > > for each task naturally. > > > > 3. Finer-grained access controls on connection secrets. Right now, although > > there are nice at-rest encryption options with Fernet or Vault, IIUC any > > DAG can access any connection (and thus any secret). Since the "run as" > > user is largely defined within the DAG and its tasks, this is challenging > > for a multi-tenant environment (see caveat below) > > > > Caveat: It's definitely noted that to some extent we should assume that an > > Airflow deployment is a "trusted" environment and that best practices such > > as git+PR workflows are the gold standard and that any malicious code and > > dependencies should be identified through this process. Also that there is > > a clear admin role for connection management etc. > > > > We have some ideas informally sketched out as to how to address the above > > but would be keen to hear the community opinion on this and to see if > > anyone is keen to collaborate on designs and implementation, or to hear if > > anything is already in the works. In particular I noticed that the very > > first improvement proposal (AIP-1) addresses much of the above :). However, > > it seems fairly dormant at the moment. > > > > One other question: we have a provider (operators and hooks) for > > interacting with Cloudera components that we'd like to contribute to the > > project. The provider FAQs indicate that new provider contributions are > > still welcome in the project in 2.x, is that accurate? > > > > Thanks in advance! > > > > Ian
