@Ash - Another example - the talk we gave with Tomek in Mexico for one (but we gave it at least 5 times already) - link to the very place we talk about it: https://youtu.be/06tYjcQ12i4?t=1210 :).
@Kaxil-> I think it will be better to make a decision before we make Backport Operators release if we think it is a good direction. The change is relatively easy to do and automate, and I think the main idea of backport operators is to make it easy for people to switch to 2.0 operators in their DAGs - so if make a change later, we again introduce another DAG migration. With Backports/2.0 I think we have quite a unique opportunity to set some standards in naming that we can enforce and maintain in the foreseeable future. Also - it makes it easier for our automation around documentation, installation, testing etc. Only because we set some standards and enforce them we are - for one able to produce fully automated release notes that are nice and readable and useful for 56 packages. That's not a small feat I think. That's why I think even if it delays release by a few days, let's better do it right and consistent. J. On Sat, May 30, 2020 at 7:05 PM Ash Berlin-Taylor <a...@apache.org> wrote: > Ah https://airflow.apache.org/docs/1.10.10/_api/index.html#operators -- > I haven't looked at that content closely in a long while. > > On May 30 2020, at 5:40 pm, Ash Berlin-Taylor <a...@apache.org> wrote: > > > Oh really, can you point at any talks? I've not seen them referrred to > > as a first class concept before. > > > > (Sensors are also different because they can be rescheduled - > > TransferOperators aren't currently in any way more special to Airflow > > than any other operator.) > > > > -a > > > > On May 30 2020, at 5:31 pm, Jarek Potiuk <jarek.pot...@polidea.com> > wrote: > > > >> Well. Sensors are based on BaseSensorOperator which is based on > >> BaseOperator. So they are technically also operators (just > >> grand-children). In the future maybe we will have BaseTransferOperator > >> as well (as per our discussion in Slack). I think this might make > >> sense to have transfer at the same level as operator and sensor. > >> > >> In many talks we already speak about those (Sensor/Transfer/Operators) > >> as different kinds of beasts so having packages at the same level > >> makes more sense IMHO.. > >> > >> J. > >> > >> > >> . > >> > >> > >> On Sat, May 30, 2020 at 5:54 PM Ash Berlin-Taylor <a...@apache.org> > wrote: > >>> > >>> I understand what you mean, and some separation could make sense, but > >>> transfer operators are still operators (unlike sensors which are a > >>> different interface/base class). > >>> > >>> How would people feel about airflow.providers.*. operators.transfer.X? > >>> > >>> On 30 May 2020 15:36:42 BST, Felix Uellendall > >>> <felue...@pm.me.INVALID> wrote: > >>> >I really like the idea, too. It makes the users search for > >>> transfers a > >>> >lot easier. > >>> > > >>> >Best Regards, > >>> >Felix > >>> > > >>> >Sent from ProtonMail Mobile > >>> > > >>> >On Sat, May 30, 2020 at 15:59, Jarek Potiuk <jarek.pot...@polidea.com > > > >>> >wrote: > >>> > > >>> >> I definitely like the idea - if we can discussion and voting on it > >>> >before > >>> >> we prepare backport package final (???) RC candidate, that would be > >>> >great. > >>> >> > >>> >> On Sat, May 30, 2020 at 3:31 PM Tomasz Urbaszek > >>> ><turbas...@apache.org> > >>> >> wrote: > >>> >> > >>> >>> Oh, makes even more sense. I like the idea. > >>> >>> > >>> >>> T. > >>> >>> > >>> >>> On Sat, May 30, 2020 at 3:14 PM Kamil Breguła > >>> ><kamil.breg...@polidea.com> > >>> >>> wrote: > >>> >>> > >>> >>> > I don't want to create package > >>> >>> > airflow.providers.transfer > >>> >>> > but package > >>> >>> > airflow.providers.*.transfer > >>> >>> > All classes will be in the same provider package but in a > >>> >different > >>> >>> nested > >>> >>> > package. You will need to import packages from the transfer > >>> >package, not > >>> >>> > the operator package. > >>> >>> > > >>> >>> > > >>> >>> > On Sat, May 30, 2020, 15:05 Tomasz Urbaszek < > turbas...@apache.org> > >>> >>> wrote: > >>> >>> > > >>> >>> > > I am in favor of this change. However, do I correctly > understand > >>> >that > >>> >>> > > providers.transfer will need many other providers (GCS=gcp, > >>> >S3=aws, > >>> >>> > > etc)? > >>> >>> > > > >>> >>> > > T. > >>> >>> > > > >>> >>> > > > >>> >>> > > On Sat, May 30, 2020 at 2:28 PM Kamil Breguła < > >>> >>> kamil.breg...@polidea.com > >>> >>> > > > >>> >>> > > wrote: > >>> >>> > > > > >>> >>> > > > Hello, > >>> >>> > > > > >>> >>> > > > We currently have transfer operators and other operators in > >>> >one > >>> >>> > > > package - operators. > >>> >>> > > > This causes some packages to be large and will grow all the > >>> >time. For > >>> >>> > > > example: airflow.providers.google.cloud.operators have 52 > >>> >modules > >>> >>> > > > For this reason, the idea arose to move these operators to > the > >>> >new > >>> >>> > > package. > >>> >>> > > > As a result, we will have the following provider package > >>> >structure. > >>> >>> > > > > >>> >>> > > > airflow.providers.*.example_dags > >>> >>> > > > airflow.providers.*.hooks > >>> >>> > > > airflow.providers.*.operators > >>> >>> > > > airflow.providers.*.secrets > >>> >>> > > > airflow.providers.*.sensors > >>> >>> > > > airflow.providers.*.transfer > >>> >>> > > > airflow.providers.*.utils > >>> >>> > > > > >>> >>> > > > Such a division already exists in the documentation. We > >>> >distinguish > >>> >>> > > > operators, transfer operators, and sensors. > >>> >>> > > > > >>> >https://airflow.readthedocs.io/en/latest/_api/index.html#operators > >>> >>> > > > > >>> >>> > >>> >https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html > >>> >>> > > > In my opinion, this change has many advantages: > >>> >>> > > > * the documentation and the user habits would more suit the > >>> >structure > >>> >>> > > > of the code, > >>> >>> > > > * would facilitate browsing of operators, > >>> >>> > > > * would allow more comfortable documentation generation based > >>> >on the > >>> >>> > > > directory structure. It cannot be determined now whether > >>> >>> > > > cloud_speech_to_text is transfer operators or a regular > >>> >operator > >>> >>> based > >>> >>> > > > on the module name. > >>> >>> > > > > >>> >>> > > > The only minus is the need to change users' habits. Users > will > >>> >have > >>> >>> to > >>> >>> > > > start looking for operators in the new package, but if they > >>> >find this > >>> >>> > > > package, they will find the module faster. > >>> >>> > > > > >>> >>> > > > Best regards, > >>> >>> > > > Kamil > >>> >>> > > > >>> >>> > > >>> >>> > >>> >> > >>> >> -- > >>> >> > >>> >> Jarek Potiuk > >>> >> Polidea <https://www.polidea.com/> | Principal Software Engineer > >>> >> > >>> >> M: +48 660 796 129 <+48660796129> > >>> >> [image: Polidea] <https://www.polidea.com/> > >> > >> > >> > >> -- > >> > >> Jarek Potiuk > >> Polidea | Principal Software Engineer > >> > >> M: +48 660 796 129 > >> > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
