Should we start voting on it? I think if we do it now and we agree on moving - we have the opportunity to release RC4 on Wednesday and release final version on Monday next week.
I'd love to have all the moves already in place with the final Backport release. That would make it really easy for people to migrate to 2.0 later on. J. On Sat, May 30, 2020 at 7:25 PM Jarek Potiuk <jarek.pot...@polidea.com> wrote: > @Ash - Another example - the talk we gave with Tomek in Mexico for one > (but we gave it at least 5 times already) - link to the very place we talk > about it: https://youtu.be/06tYjcQ12i4?t=1210 :). > > @Kaxil-> I think it will be better to make a decision before we make > Backport Operators release if we think it is a good direction. The change > is relatively easy to do and automate, and I think the main idea of > backport operators is to make it easy for people to switch to 2.0 operators > in their DAGs - so if make a change later, we again introduce another DAG > migration. > > With Backports/2.0 I think we have quite a unique opportunity to set some > standards in naming that we can enforce and maintain in the foreseeable > future. Also - it makes it easier for our automation around documentation, > installation, testing etc. Only because we set some standards and enforce > them we are - for one able to produce fully automated release notes that > are nice and readable and useful for 56 packages. That's not a small feat I > think. > > That's why I think even if it delays release by a few days, let's better > do it right and consistent. > > J. > > On Sat, May 30, 2020 at 7:05 PM Ash Berlin-Taylor <a...@apache.org> wrote: > >> Ah https://airflow.apache.org/docs/1.10.10/_api/index.html#operators -- >> I haven't looked at that content closely in a long while. >> >> On May 30 2020, at 5:40 pm, Ash Berlin-Taylor <a...@apache.org> wrote: >> >> > Oh really, can you point at any talks? I've not seen them referrred to >> > as a first class concept before. >> > >> > (Sensors are also different because they can be rescheduled - >> > TransferOperators aren't currently in any way more special to Airflow >> > than any other operator.) >> > >> > -a >> > >> > On May 30 2020, at 5:31 pm, Jarek Potiuk <jarek.pot...@polidea.com> >> wrote: >> > >> >> Well. Sensors are based on BaseSensorOperator which is based on >> >> BaseOperator. So they are technically also operators (just >> >> grand-children). In the future maybe we will have BaseTransferOperator >> >> as well (as per our discussion in Slack). I think this might make >> >> sense to have transfer at the same level as operator and sensor. >> >> >> >> In many talks we already speak about those (Sensor/Transfer/Operators) >> >> as different kinds of beasts so having packages at the same level >> >> makes more sense IMHO.. >> >> >> >> J. >> >> >> >> >> >> . >> >> >> >> >> >> On Sat, May 30, 2020 at 5:54 PM Ash Berlin-Taylor <a...@apache.org> >> wrote: >> >>> >> >>> I understand what you mean, and some separation could make sense, but >> >>> transfer operators are still operators (unlike sensors which are a >> >>> different interface/base class). >> >>> >> >>> How would people feel about airflow.providers.*. operators.transfer.X? >> >>> >> >>> On 30 May 2020 15:36:42 BST, Felix Uellendall >> >>> <felue...@pm.me.INVALID> wrote: >> >>> >I really like the idea, too. It makes the users search for >> >>> transfers a >> >>> >lot easier. >> >>> > >> >>> >Best Regards, >> >>> >Felix >> >>> > >> >>> >Sent from ProtonMail Mobile >> >>> > >> >>> >On Sat, May 30, 2020 at 15:59, Jarek Potiuk < >> jarek.pot...@polidea.com> >> >>> >wrote: >> >>> > >> >>> >> I definitely like the idea - if we can discussion and voting on it >> >>> >before >> >>> >> we prepare backport package final (???) RC candidate, that would be >> >>> >great. >> >>> >> >> >>> >> On Sat, May 30, 2020 at 3:31 PM Tomasz Urbaszek >> >>> ><turbas...@apache.org> >> >>> >> wrote: >> >>> >> >> >>> >>> Oh, makes even more sense. I like the idea. >> >>> >>> >> >>> >>> T. >> >>> >>> >> >>> >>> On Sat, May 30, 2020 at 3:14 PM Kamil Breguła >> >>> ><kamil.breg...@polidea.com> >> >>> >>> wrote: >> >>> >>> >> >>> >>> > I don't want to create package >> >>> >>> > airflow.providers.transfer >> >>> >>> > but package >> >>> >>> > airflow.providers.*.transfer >> >>> >>> > All classes will be in the same provider package but in a >> >>> >different >> >>> >>> nested >> >>> >>> > package. You will need to import packages from the transfer >> >>> >package, not >> >>> >>> > the operator package. >> >>> >>> > >> >>> >>> > >> >>> >>> > On Sat, May 30, 2020, 15:05 Tomasz Urbaszek < >> turbas...@apache.org> >> >>> >>> wrote: >> >>> >>> > >> >>> >>> > > I am in favor of this change. However, do I correctly >> understand >> >>> >that >> >>> >>> > > providers.transfer will need many other providers (GCS=gcp, >> >>> >S3=aws, >> >>> >>> > > etc)? >> >>> >>> > > >> >>> >>> > > T. >> >>> >>> > > >> >>> >>> > > >> >>> >>> > > On Sat, May 30, 2020 at 2:28 PM Kamil Breguła < >> >>> >>> kamil.breg...@polidea.com >> >>> >>> > > >> >>> >>> > > wrote: >> >>> >>> > > > >> >>> >>> > > > Hello, >> >>> >>> > > > >> >>> >>> > > > We currently have transfer operators and other operators in >> >>> >one >> >>> >>> > > > package - operators. >> >>> >>> > > > This causes some packages to be large and will grow all the >> >>> >time. For >> >>> >>> > > > example: airflow.providers.google.cloud.operators have 52 >> >>> >modules >> >>> >>> > > > For this reason, the idea arose to move these operators to >> the >> >>> >new >> >>> >>> > > package. >> >>> >>> > > > As a result, we will have the following provider package >> >>> >structure. >> >>> >>> > > > >> >>> >>> > > > airflow.providers.*.example_dags >> >>> >>> > > > airflow.providers.*.hooks >> >>> >>> > > > airflow.providers.*.operators >> >>> >>> > > > airflow.providers.*.secrets >> >>> >>> > > > airflow.providers.*.sensors >> >>> >>> > > > airflow.providers.*.transfer >> >>> >>> > > > airflow.providers.*.utils >> >>> >>> > > > >> >>> >>> > > > Such a division already exists in the documentation. We >> >>> >distinguish >> >>> >>> > > > operators, transfer operators, and sensors. >> >>> >>> > > > >> >>> >https://airflow.readthedocs.io/en/latest/_api/index.html#operators >> >>> >>> > > > >> >>> >>> >> >>> > >> https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html >> >>> >>> > > > In my opinion, this change has many advantages: >> >>> >>> > > > * the documentation and the user habits would more suit the >> >>> >structure >> >>> >>> > > > of the code, >> >>> >>> > > > * would facilitate browsing of operators, >> >>> >>> > > > * would allow more comfortable documentation generation >> based >> >>> >on the >> >>> >>> > > > directory structure. It cannot be determined now whether >> >>> >>> > > > cloud_speech_to_text is transfer operators or a regular >> >>> >operator >> >>> >>> based >> >>> >>> > > > on the module name. >> >>> >>> > > > >> >>> >>> > > > The only minus is the need to change users' habits. Users >> will >> >>> >have >> >>> >>> to >> >>> >>> > > > start looking for operators in the new package, but if they >> >>> >find this >> >>> >>> > > > package, they will find the module faster. >> >>> >>> > > > >> >>> >>> > > > Best regards, >> >>> >>> > > > Kamil >> >>> >>> > > >> >>> >>> > >> >>> >>> >> >>> >> >> >>> >> -- >> >>> >> >> >>> >> Jarek Potiuk >> >>> >> Polidea <https://www.polidea.com/> | Principal Software Engineer >> >>> >> >> >>> >> M: +48 660 796 129 <+48660796129> >> >>> >> [image: Polidea] <https://www.polidea.com/> >> >> >> >> >> >> >> >> -- >> >> >> >> Jarek Potiuk >> >> Polidea | Principal Software Engineer >> >> >> >> M: +48 660 796 129 >> >> >> > > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
