> > @Kaxil-> I think it will be better to make a decision before we make > Backport Operators release if we think it is a good direction. The change > is relatively easy to do and automate, and I think the main idea of > backport operators is to make it easy for people to switch to 2.0 operators > in their DAGs - so if make a change later, we again introduce another DAG > migration.
Yeah sure :-) On Mon, Jun 1, 2020 at 8:22 AM Jarek Potiuk <jarek.pot...@polidea.com> wrote: > Should we start voting on it? I think if we do it now and we agree on > moving - we have the opportunity to release RC4 on Wednesday and release > final version on Monday next week. > > I'd love to have all the moves already in place with the final Backport > release. That would make it really easy for people to migrate to 2.0 later > on. > > J. > > On Sat, May 30, 2020 at 7:25 PM Jarek Potiuk <jarek.pot...@polidea.com> > wrote: > > > @Ash - Another example - the talk we gave with Tomek in Mexico for one > > (but we gave it at least 5 times already) - link to the very place we > talk > > about it: https://youtu.be/06tYjcQ12i4?t=1210 :). > > > > @Kaxil-> I think it will be better to make a decision before we make > > Backport Operators release if we think it is a good direction. The change > > is relatively easy to do and automate, and I think the main idea of > > backport operators is to make it easy for people to switch to 2.0 > operators > > in their DAGs - so if make a change later, we again introduce another > DAG > > migration. > > > > With Backports/2.0 I think we have quite a unique opportunity to set some > > standards in naming that we can enforce and maintain in the foreseeable > > future. Also - it makes it easier for our automation around > documentation, > > installation, testing etc. Only because we set some standards and enforce > > them we are - for one able to produce fully automated release notes that > > are nice and readable and useful for 56 packages. That's not a small > feat I > > think. > > > > That's why I think even if it delays release by a few days, let's better > > do it right and consistent. > > > > J. > > > > On Sat, May 30, 2020 at 7:05 PM Ash Berlin-Taylor <a...@apache.org> > wrote: > > > >> Ah https://airflow.apache.org/docs/1.10.10/_api/index.html#operators -- > >> I haven't looked at that content closely in a long while. > >> > >> On May 30 2020, at 5:40 pm, Ash Berlin-Taylor <a...@apache.org> wrote: > >> > >> > Oh really, can you point at any talks? I've not seen them referrred to > >> > as a first class concept before. > >> > > >> > (Sensors are also different because they can be rescheduled - > >> > TransferOperators aren't currently in any way more special to Airflow > >> > than any other operator.) > >> > > >> > -a > >> > > >> > On May 30 2020, at 5:31 pm, Jarek Potiuk <jarek.pot...@polidea.com> > >> wrote: > >> > > >> >> Well. Sensors are based on BaseSensorOperator which is based on > >> >> BaseOperator. So they are technically also operators (just > >> >> grand-children). In the future maybe we will have > BaseTransferOperator > >> >> as well (as per our discussion in Slack). I think this might make > >> >> sense to have transfer at the same level as operator and sensor. > >> >> > >> >> In many talks we already speak about those > (Sensor/Transfer/Operators) > >> >> as different kinds of beasts so having packages at the same level > >> >> makes more sense IMHO.. > >> >> > >> >> J. > >> >> > >> >> > >> >> . > >> >> > >> >> > >> >> On Sat, May 30, 2020 at 5:54 PM Ash Berlin-Taylor <a...@apache.org> > >> wrote: > >> >>> > >> >>> I understand what you mean, and some separation could make sense, > but > >> >>> transfer operators are still operators (unlike sensors which are a > >> >>> different interface/base class). > >> >>> > >> >>> How would people feel about airflow.providers.*. > operators.transfer.X? > >> >>> > >> >>> On 30 May 2020 15:36:42 BST, Felix Uellendall > >> >>> <felue...@pm.me.INVALID> wrote: > >> >>> >I really like the idea, too. It makes the users search for > >> >>> transfers a > >> >>> >lot easier. > >> >>> > > >> >>> >Best Regards, > >> >>> >Felix > >> >>> > > >> >>> >Sent from ProtonMail Mobile > >> >>> > > >> >>> >On Sat, May 30, 2020 at 15:59, Jarek Potiuk < > >> jarek.pot...@polidea.com> > >> >>> >wrote: > >> >>> > > >> >>> >> I definitely like the idea - if we can discussion and voting on > it > >> >>> >before > >> >>> >> we prepare backport package final (???) RC candidate, that would > be > >> >>> >great. > >> >>> >> > >> >>> >> On Sat, May 30, 2020 at 3:31 PM Tomasz Urbaszek > >> >>> ><turbas...@apache.org> > >> >>> >> wrote: > >> >>> >> > >> >>> >>> Oh, makes even more sense. I like the idea. > >> >>> >>> > >> >>> >>> T. > >> >>> >>> > >> >>> >>> On Sat, May 30, 2020 at 3:14 PM Kamil Breguła > >> >>> ><kamil.breg...@polidea.com> > >> >>> >>> wrote: > >> >>> >>> > >> >>> >>> > I don't want to create package > >> >>> >>> > airflow.providers.transfer > >> >>> >>> > but package > >> >>> >>> > airflow.providers.*.transfer > >> >>> >>> > All classes will be in the same provider package but in a > >> >>> >different > >> >>> >>> nested > >> >>> >>> > package. You will need to import packages from the transfer > >> >>> >package, not > >> >>> >>> > the operator package. > >> >>> >>> > > >> >>> >>> > > >> >>> >>> > On Sat, May 30, 2020, 15:05 Tomasz Urbaszek < > >> turbas...@apache.org> > >> >>> >>> wrote: > >> >>> >>> > > >> >>> >>> > > I am in favor of this change. However, do I correctly > >> understand > >> >>> >that > >> >>> >>> > > providers.transfer will need many other providers (GCS=gcp, > >> >>> >S3=aws, > >> >>> >>> > > etc)? > >> >>> >>> > > > >> >>> >>> > > T. > >> >>> >>> > > > >> >>> >>> > > > >> >>> >>> > > On Sat, May 30, 2020 at 2:28 PM Kamil Breguła < > >> >>> >>> kamil.breg...@polidea.com > >> >>> >>> > > > >> >>> >>> > > wrote: > >> >>> >>> > > > > >> >>> >>> > > > Hello, > >> >>> >>> > > > > >> >>> >>> > > > We currently have transfer operators and other operators > in > >> >>> >one > >> >>> >>> > > > package - operators. > >> >>> >>> > > > This causes some packages to be large and will grow all > the > >> >>> >time. For > >> >>> >>> > > > example: airflow.providers.google.cloud.operators have 52 > >> >>> >modules > >> >>> >>> > > > For this reason, the idea arose to move these operators to > >> the > >> >>> >new > >> >>> >>> > > package. > >> >>> >>> > > > As a result, we will have the following provider package > >> >>> >structure. > >> >>> >>> > > > > >> >>> >>> > > > airflow.providers.*.example_dags > >> >>> >>> > > > airflow.providers.*.hooks > >> >>> >>> > > > airflow.providers.*.operators > >> >>> >>> > > > airflow.providers.*.secrets > >> >>> >>> > > > airflow.providers.*.sensors > >> >>> >>> > > > airflow.providers.*.transfer > >> >>> >>> > > > airflow.providers.*.utils > >> >>> >>> > > > > >> >>> >>> > > > Such a division already exists in the documentation. We > >> >>> >distinguish > >> >>> >>> > > > operators, transfer operators, and sensors. > >> >>> >>> > > > > >> >>> >https://airflow.readthedocs.io/en/latest/_api/index.html#operators > >> >>> >>> > > > > >> >>> >>> > >> >>> > > >> https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html > >> >>> >>> > > > In my opinion, this change has many advantages: > >> >>> >>> > > > * the documentation and the user habits would more suit > the > >> >>> >structure > >> >>> >>> > > > of the code, > >> >>> >>> > > > * would facilitate browsing of operators, > >> >>> >>> > > > * would allow more comfortable documentation generation > >> based > >> >>> >on the > >> >>> >>> > > > directory structure. It cannot be determined now whether > >> >>> >>> > > > cloud_speech_to_text is transfer operators or a regular > >> >>> >operator > >> >>> >>> based > >> >>> >>> > > > on the module name. > >> >>> >>> > > > > >> >>> >>> > > > The only minus is the need to change users' habits. Users > >> will > >> >>> >have > >> >>> >>> to > >> >>> >>> > > > start looking for operators in the new package, but if > they > >> >>> >find this > >> >>> >>> > > > package, they will find the module faster. > >> >>> >>> > > > > >> >>> >>> > > > Best regards, > >> >>> >>> > > > Kamil > >> >>> >>> > > > >> >>> >>> > > >> >>> >>> > >> >>> >> > >> >>> >> -- > >> >>> >> > >> >>> >> Jarek Potiuk > >> >>> >> Polidea <https://www.polidea.com/> | Principal Software Engineer > >> >>> >> > >> >>> >> M: +48 660 796 129 <+48660796129> > >> >>> >> [image: Polidea] <https://www.polidea.com/> > >> >> > >> >> > >> >> > >> >> -- > >> >> > >> >> Jarek Potiuk > >> >> Polidea | Principal Software Engineer > >> >> > >> >> M: +48 660 796 129 > >> >> > >> > > > > > > -- > > > > Jarek Potiuk > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > M: +48 660 796 129 <+48660796129> > > [image: Polidea] <https://www.polidea.com/> > > > > > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> >
