Agreed on your points @andrey.ans...@taragol.is <andrey.ans...@taragol.is>
On Fri, 17 May 2024 at 15:01, Andrey Anshin <andrey.ans...@taragol.is> wrote: > IMHO, In case if we decide to keep only Postgres support we need to have > really powerful arguments to provide an interface which helps integrate > with other DBs. > > In this case, we must clearly understand what the community is responsible > for in this case and how it can be sure that nothing is broken > > Especially if we take in account the Airflow has very tight integrations > with specific Databases, requires a lot of effort to support additional > ones (MS SQL case), and the DB part is not a part of the Public Interface > of Airflow [1]. > > So I would consider that this should be two separate decisions: > 1. Keep only Postgres (vanila, not forks) as supported/tested backend in > Production. SQLite remains as development DB. > 2. Provide public interface to DB integrations between Airflow and DB for > third parties > > [1]: > > https://airflow.apache.org/docs/apache-airflow/stable/public-airflow-interface.html > > > On Tue, 14 May 2024 at 12:15, Kaxil Naik <kaxiln...@gmail.com> wrote: > > > Yeah, that works for me > > > > Can we have it possible to have two (or maybe three - > > > like a sub-committee) co-owners of topics? > > > > > > On Tue, 14 May 2024 at 06:15, Vikram Koka <vik...@astronomer.io.invalid> > > wrote: > > > > > Definitely a fast moving thread on the mailing list. I haven’t been > able > > to > > > respond for a few days and feel very far behind already. > > > > > > A few comments on topics discussed the last few days: > > > - Jarek, in response to your comments around being more aggressive than > > in > > > Airflow 2 about deprecation and drops of functionality, I am very > > > supportive of that stance. I completely agree that we could have been > > more > > > aggressive as part of Airflow 2. > > > However, I would like to ask that as we go forward, we make sure that > we > > > have clean interfaces to be able to add support, even if we choose a > > single > > > implementation. For example, with respect to dropping MySQL support. I > > can > > > understand the perspective of the project that this should be > deprecated > > > from an Airflow OSS perspective. However, even if the only OSS > supported > > DB > > > is Postgres, I would like to ensure that a clean interface exists for > > > interaction with the DB, so that other databases such as MySQL or > others > > > CAN be supported by a third party or at a later date. > > > I realize that this may seem onerous, but I believe that it enables us > to > > > be more flexible in the long run, rather than locking us into a single > DB > > > implementation. > > > > > > - Bolke, Daniel Standish, Ash, et al on the task execution contract, > > > definitely looking forward to this. > > > > > > - To those that I proposed a couple of more detailed write ups, I still > > > plan to do that, at the latest by early next week. > > > > > > Vikram > > > > > > > > > > > > On Mon, May 13, 2024 at 9:30 PM Jarek Potiuk <ja...@potiuk.com> wrote: > > > > > > > Super-excited about that. > > > > > > > > Question/Proposal: Can we have it possible to have two (or maybe > three > > - > > > > like a sub-committee) co-owners of topics? I think it's a lot to put > on > > > > one's head to "own" a topic and given circumstances/ volunteer time > of > > > > people, interruptions (and life intervening), it might be a bit risky > > to > > > > put it on one's shoulders only. > > > > > > > > I know it's against the rule ("if it is owned by many, it's not owned > > by > > > > anyone") - but I think in our case there are at least some topics > that > > > > could benefit from having more than one owner. Especially when we > know > > > and > > > > trust that we can work together on some topics that we are passionate > > > > about. It might also encourage getting out of people's comfort zones. > > > > > > > > For example - I'd absolutely love to volunteer to co-own the > > "streamline > > > > the development" with Andrey if he would be willing to of course :D > > > (sorry > > > > Andrey for "volunteering you" on that one :D) - and maybe we could > get > > > > someone else to join us. > > > > > > > > That might have the added benefit of being able to break with the way > > > > we've been doing things. If I am owning it for one - I'd likely > > gravitate > > > > towards past choices, but with others joining me and taking decisions > > > (and > > > > responsibility in making sure we implement them) together, we could > > make > > > > better decisions and reduce bus factor for dev tooling/ CI in the > > future. > > > > > > > > BTW. Shameless promotion: tomorrow I am giving a talk about that > very > > > > topic (in the context of last few years not yet Airflow 3.0) at the > NY > > > > meetup hosted at Astronomer NY headquarters > > > > https://www.meetup.com/nyc-apache-airflow-meetup/events/300017228/ - > > so > > > if > > > > you are in NY or around - I think you can stil sign up :D. I am also > > > > getting to PyCon US in Pittsburgh next week so don't expect too much > > from > > > > me. I will be gearing up for streamlining the development by talking > to > > > the > > > > right people and listening to the latest things and best practices of > > the > > > > larger Python community :). > > > > > > > > J. > > > > > > > > On Tue, May 14, 2024 at 12:03 AM Kaxil Naik <kaxiln...@gmail.com> > > wrote: > > > > > > > > > Thank you all, I am very happy about the discussions. > > > > > > > > > > The mailing list moves fast :). The main reason I recommended > > starting > > > > the > > > > > dev calls in early June was to have some of these discussions on > the > > > > > mailing list. > > > > > > > > > > Since Michal already scheduled a call, let's start there to discuss > > > > > various ideas. For the week after that, I have created an Airflow > > > 2-style > > > > > recurring open dev calls for anyone to join, info below: > > > > > > > > > > *Date & Time: *Recurring every 2 weeks on Thursday at* 4pm BST *( 3 > > PM > > > > > GMT/UTC | 11 AM EST | 8 AM PST); starting* May 30, 2024 04:00 PM > BST* > > > and > > > > > then > > > > > *One-time registration Link*: > > > > > > > > > > > > > > > > > > > > https://astronomer.zoom.us/meeting/register/tZAsde2vqDwpE9XrBAbCeIFHA_l7OLywrWkG > > > > > *Add to your calendar*: > > > > > > > > > > > > > > > > > > > > https://astronomer.zoom.us/meeting/tZAsde2vqDwpE9XrBAbCeIFHA_l7OLywrWkG/calendar/google/add > > > > > > > > > > I will post the meeting notes on the dev mailing list as well as > > > > Confluence > > > > > for archival purposes (example > > > > > <https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes > > >). > > > > > > > > > > Once we discuss various proposals next week, I recommend that for > > each > > > > > "workstream", we have an owner who would want to lead that > > workstream. > > > > For > > > > > items, that does not have an owner we can put those into Airflow 3 > > Meta > > > > > issue <https://github.com/apache/airflow/issues/39593> or > cross-link > > > > over > > > > > there so someone in the community can take it on. If we don't have > an > > > > owner > > > > > who will commit to working on it, we park that item until we find > the > > > > > owner. > > > > > > > > > > At the end of each call, I would solicit ideas for the agenda for > the > > > > next > > > > > call and propose it to the broader group on the mailing list. > > > > > > > > > > Some of the items that should be discussed in the upcoming calls > IMO: > > > > > > > > > > - Agreeing on Principles > > > > > > > > > > Based on the discussions, some potential items (all up for > debate) > > > > > - Considering Airflow 3.0 for early adopters and* breaking > (and > > > > > removing) things for AF 3.0*. Things can be re-added as > needed > > in > > > > > upcoming minor releases > > > > > - Optimize to get *foundational pieces in* and not "let > perfect > > > be > > > > > the enemy of good" > > > > > - Working on features that solidify Airflow as the* modern > > > > > Orchestrator* that also has state of the art *support for > Data, > > > AI > > > > & > > > > > ML workloads*. This includes scalability & performance > > discussion > > > > > - Set up the codebase for the next 5 years. This encompasses > > all > > > > the > > > > > things we are discussing e.g removing MySQL to reduce the > test > > > > > matrix, > > > > > simplifying things architecturally, consolidating > serialization > > > > > methods, etc > > > > > > > > > > - Workstream & Stream Owners > > > > > - Airflow 2 support policy including scope (feature vs bug > fixes + > > > > > security only) & support period > > > > > - Separate discussions for each big workstream including one for > > > items > > > > > to remove & refactor (e.g dropping MySQL) > > > > > - Discussion to streamline the development of Airflow 3 > > > > > - Separating dev for Providers & Airflow (something Jarek > > already > > > > > kick-started), and > > > > > - Separate branch for Airflow 2 > > > > > - CI changes for the above > > > > > - Finalize Scope + Timelines > > > > > - Migration Utilities > > > > > - Progress check-ins > > > > > > > > > > Looking forward to the exciting months ahead. > > > > > > > > > > Regards, > > > > > Kaxil > > > > > > > > > > On Mon, 13 May 2024 at 21:40, Bolke de Bruin <bdbr...@gmail.com> > > > wrote: > > > > > > > > > > > Declaring connections prior to task execution was already > proposed > > in > > > > > AIP-1 > > > > > > :-). At that time, I had in mind to communicate over IPC to the > > task > > > > the > > > > > > required settings. Registration could then happen with a > manifest. > > > > Maybe > > > > > > during DAG serialization this could be obtained unobtrusively? > The > > > > > benefit > > > > > > is that tasks become truly atomic or independent from Airflow as > > long > > > > as > > > > > > they communicate their exit codes (success, failed, and I think > Ash > > > > had a > > > > > > couple of others in mind - the fewer the better). > > > > > > > > > > > > If you want two-way communication, maybe for variables as they > can > > > > change > > > > > > during scheduling, this can happen with AIP-44. Although, I'd > > prefer > > > it > > > > > to > > > > > > happen with the *executor* rather than some centralized service. > If > > > the > > > > > > executor is used, IPC is the logical choice. The benefit of this > is > > > > that > > > > > > you have better resiliency and you can start to think about no > > > downtime > > > > > > upgrades > > > > > > > > > > > > So I hope Ash takes this to 2024 :-). > > > > > > > > > > > > B. > > > > > > > > > > > > > > > > > > On Mon, 13 May 2024 at 19:27, Ash Berlin-Taylor <a...@apache.org> > > > > wrote: > > > > > > > > > > > > > > That would require some mechanism of declaring prior to task > > > > > execution > > > > > > > what connections would be used > > > > > > > > > > > > > > That’s exactly what I’m proposing in the proposal doc I’m > working > > > on > > > > > > (It’s > > > > > > > part of also overhauling and re-designing the “Task Execution > > > > > interface” > > > > > > > that also gives us the ability to nicely have support for > running > > > > tasks > > > > > > in > > > > > > > other languages — much more than just BashOperator) > > > > > > > > > > > > > > This is a bit of a fundamental shift in thinking about task > > > execution > > > > > in > > > > > > > Airflow, but I think it gives us some really nice properties > that > > > the > > > > > > > project is currently missing. > > > > > > > > > > > > > > Tl;dr; lets discuss this in my doc when it comes our (next week > > > most > > > > > > > likely) please :) > > > > > > > > > > > > > > -ash > > > > > > > > > > > > > > > On 13 May 2024, at 18:15, Daniel Standish > > > > > > > <daniel.stand...@astronomer.io.INVALID> wrote: > > > > > > > > > > > > > > > > re > > > > > > > > > > > > > > > > As tasks require connection access, I assume connection data > > will > > > > > > somehow > > > > > > > >> be passed as part of the > > > > > > > >> metadata to task execution - whether it's part of the > executor > > > > > > protocol > > > > > > > or > > > > > > > >> in some other way (I'm > > > > > > > >> not an expert on that part of Airflow). Then, provided it's > > > > > accessible > > > > > > > as > > > > > > > >> part of some execution > > > > > > > >> context, and not only passed to the task's execute method, > > > > > OpenLineage > > > > > > > >> could utilize it. > > > > > > > >> > > > > > > > > > > > > > > > > It's not strictly necessary that connection info be passed > "as > > > part > > > > > of > > > > > > > task > > > > > > > > matadata". That would require some mechanism of declaring > > prior > > > to > > > > > > task > > > > > > > > execution what connections would be used. This is a thought > > that > > > > has > > > > > > > come > > > > > > > > up when thinking about execution of non-python tasks. But > it's > > > not > > > > > > > > required from a technical perspective by AIP-44 because the > > > > > > > > `get_connection` function can be made to be an RPC call so a > > task > > > > > could > > > > > > > > continue to retrieve connections at runtime. > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > -- > > > > > > Bolke de Bruin > > > > > > bdbr...@gmail.com > > > > > > > > > > > > > > > > > > > > >
