Awesome Ash ??, expecting beta2. And don't forget some minor bug still in beta1
https://issues.apache.org/jira/browse/AIRFLOW-3623?focusedCommentId=16803866&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16803866 Best wish. -- Jiajie ________________________________ From: Ash Berlin-Taylor <a...@apache.org> Sent: Friday, March 29, 2019 23:16 To: dev@airflow.apache.org Subject: Re: API Reference - current confusion and improvement plan It took pulling in about another 30 commits to get it without conflicts but I've pulled this in to the v1-10-stable branch so it will be in the 1.10.3 too! (There are 68 commits to the branch already since 1.10.3b1. Time for a beta2 I think) -a > On 29 Mar 2019, at 13:28, Jiajie Zhong <zhongjiajie...@hotmail.com> wrote: > > Thanks Kamil, really a great change in out documentation > > > Best wish. > -- Jiajie > > ________________________________ > From: Driesprong, Fokko <fo...@driesprong.frl> > Sent: Friday, March 29, 2019 19:16 > To: dev@airflow.apache.org > Subject: Re: API Reference - current confusion and improvement plan > > Awesome work Kamil. Thanks for giving some love to the documentation. It > really needed some :-) > > Don't forget to remove the line from the Github template: When adding new > operators/hooks/sensors, the autoclass documentation generation needs to be > added. > https://github.com/apache/airflow/blob/master/.github/PULL_REQUEST_TEMPLATE.md > > Cheers, Fokko > > Op wo 27 mrt. 2019 om 05:59 schreef Kamil Breguła <kamil.breg...@polidea.com >> : > >> Hi. >> >> Work on this has been completed. >> New documentation is available: >> https://airflow.readthedocs.io/en/latest/_api/index.html >> >> Greetings >> Kamil Breguła >> >> On Wed, Feb 27, 2019 at 12:51 PM Kamil Breguła >> <kamil.breg...@polidea.com> wrote: >>> >>> Hi. >>> >>> Me and Jarek Potiuk have recently worked to finish these changes. As a >> result, a PR series was created: >>> >>> - [AIRFLOW-XXX][1/3] Syntax docs improvements - >> https://github.com/apache/airflow/pull/4789 >>> - [AIRFLOW-3968][2/3] Refactor base GCP hook - >> https://github.com/apache/airflow/pull/4790 >>> - [AIRFLOW-3811][3/3] Add automatic generation of API Reference - >> https://github.com/apache/airflow/pull/4788 >>> >>> I invite you to review. Preview is available in the description of each >> PR >>> >>> Greets, >>> Kamil Breguła >>> >>> On Wed, Feb 6, 2019 at 2:09 PM Szymon Przedwojski < >> szymon.przedwoj...@polidea.com> wrote: >>>> >>>> +1 >>>> I also like the new docs layout and the big win is that it’s generated >> automatically from all files and we won’t have to modify code.rst / >> integration.rst manually anymore. >>>> >>>> Szymon Przedwojski >>>> Polidea | Software Engineer >>>> >>>> M: +48 500 330 790 >>>> E: szymon.przedwoj...@polidea.com >>>> >>>>> On 5 Feb 2019, at 21:33, Ash Berlin-Taylor <a...@apache.org> wrote: >>>>> >>>>> I have idly wondered about something like this as a layout >>>>> >>>>> from airflow.$something.aws.operators import EmrAddStepOperator >>>>> >>>>> - Grouping by service provider is more helpful >>>>> - Having more than one operator per module >>>>> - Not having `_operator` (etc.) suffix on the modue, and the class, >> and the module path >>>>> >>>>> Perhaps a bigger change - though to make it much less painful on our >> users we could keep the old names with a deprecation warning or two (even >> past 2.0, to say 2.1) Out of scope for current discussion. >>>>> >>>>> -ash >>>>> >>>>>> On 5 Feb 2019, at 20:22, Kamil Breguła <kamil.breg...@polidea.com> >> wrote: >>>>>> >>>>>> I think that we should group operators by service (ex. Amazon Web >> Service: >>>>>> Simple Cloud Storage). One module to one service. it will be much >> easier to >>>>>> navigate through them. A similar problem occurs with the Google Cloud >>>>>> Storage service, but we have a solution (PR: >>>>>> https://github.com/apache/airflow/pull/3000 ). A large part and >> future >>>>>> operators, which are written in accordance with the recommendations ( >>>>>> >> https://lists.apache.org/thread.html/e8534d82be611ae7bcb21ba371546a4278aad117d5e50361fd8f14fe@%3Cdev.airflow.apache.org%3E >> ), >>>>>> follow these rules. >>>>>> >>>>>> The problem will be with operators that integrate two services at >> the same >>>>>> time. I think that we can leave them in a separate module and link >> to this >>>>>> class in the description of the module. >>>>>> >>>>>> However, this is not a current problem. I just wanted to mark future >>>>>> improvements, which is possible if we introduce the proposed >> solution. >>>>>> >>>>>> On Tue, Feb 5, 2019 at 8:57 PM Ash Berlin-Taylor <a...@apache.org> >> wrote: >>>>>> >>>>>>> I like the API reference v2 layout a lot! Much easier to navigate >> and see >>>>>>> what classes are available, for me at least >>>>>>> >>>>>>> Documenting modules will help somewhat with a few things but, lets >> say the >>>>>>> "AWS" section of the integration doc is across the following >> modules: >>>>>>> >>>>>>> airflow.contrib.operators.aws_athena_operator < >>>>>>> >> http://level-can.surge.sh/html/autoapi/airflow/contrib/operators/aws_athena_operator/index.html >>>>>>>> >>>>>>> airflow.contrib.operators.awsbatch_operator < >>>>>>> >> http://level-can.surge.sh/html/autoapi/airflow/contrib/operators/awsbatch_operator/index.html >>>>>>>> >>>>>>> airflow.contrib.operators.ecs_operator < >>>>>>> >> http://level-can.surge.sh/html/autoapi/airflow/contrib/operators/ecs_operator/index.html >>>>>>>> >>>>>>> airflow.contrib.operators.emr_add_steps_operator < >>>>>>> >> http://level-can.surge.sh/html/autoapi/airflow/contrib/operators/emr_add_steps_operator/index.html >>>>>>>> >>>>>>> airflow.contrib.operators.emr_create_job_flow_operator < >>>>>>> >> http://level-can.surge.sh/html/autoapi/airflow/contrib/operators/emr_create_job_flow_operator/index.html >>>>>>>> >>>>>>> airflow.contrib.operators.emr_terminate_job_flow_operator < >>>>>>> >> http://level-can.surge.sh/html/autoapi/airflow/contrib/operators/emr_terminate_job_flow_operator/index.html >>>>>>>> >>>>>>> airflow.contrib.operators.s3_copy_object_operator < >>>>>>> >> http://level-can.surge.sh/html/autoapi/airflow/contrib/operators/s3_copy_object_operator/index.html >>>>>>>> >>>>>>> airflow.contrib.operators.s3_delete_objects_operator < >>>>>>> >> http://level-can.surge.sh/html/autoapi/airflow/contrib/operators/s3_delete_objects_operator/index.html >>>>>>>> >>>>>>> airflow.contrib.operators.s3_list_operator < >>>>>>> >> http://level-can.surge.sh/html/autoapi/airflow/contrib/operators/s3_list_operator/index.html >>>>>>>> >>>>>>> airflow.contrib.operators.s3_to_gcs_operator < >>>>>>> >> http://level-can.surge.sh/html/autoapi/airflow/contrib/operators/s3_to_gcs_operator/index.html >>>>>>>> >>>>>>> airflow.contrib.operators.s3_to_gcs_transfer_operator < >>>>>>> >> http://level-can.surge.sh/html/autoapi/airflow/contrib/operators/s3_to_gcs_transfer_operator/index.html >>>>>>>> >>>>>>> airflow.contrib.operators.s3_to_sftp_operator < >>>>>>> >> http://level-can.surge.sh/html/autoapi/airflow/contrib/operators/s3_to_sftp_operator/index.html >>>>>>>> >>>>>>> airflow.contrib.operators.sagemaker_base_operator < >>>>>>> >> http://level-can.surge.sh/html/autoapi/airflow/contrib/operators/sagemaker_base_operator/index.html >>>>>>>> >>>>>>> airflow.contrib.operators.sagemaker_endpoint_config_operator < >>>>>>> >> http://level-can.surge.sh/html/autoapi/airflow/contrib/operators/sagemaker_endpoint_config_operator/index.html >>>>>>>> >>>>>>> airflow.contrib.operators.sagemaker_endpoint_operator < >>>>>>> >> http://level-can.surge.sh/html/autoapi/airflow/contrib/operators/sagemaker_endpoint_operator/index.html >>>>>>>> >>>>>>> airflow.contrib.operators.sagemaker_model_operator < >>>>>>> >> http://level-can.surge.sh/html/autoapi/airflow/contrib/operators/sagemaker_model_operator/index.html >>>>>>>> >>>>>>> airflow.contrib.operators.sagemaker_training_operator < >>>>>>> >> http://level-can.surge.sh/html/autoapi/airflow/contrib/operators/sagemaker_training_operator/index.html >>>>>>>> >>>>>>> airflow.contrib.operators.sagemaker_transform_operator < >>>>>>> >> http://level-can.surge.sh/html/autoapi/airflow/contrib/operators/sagemaker_transform_operator/index.html >>>>>>>> >>>>>>> airflow.contrib.operators.sagemaker_tuning_operator < >>>>>>> >> http://level-can.surge.sh/html/autoapi/airflow/contrib/operators/sagemaker_tuning_operator/index.html >>>>>>>> >>>>>>> airflow.contrib.operators.segment_track_event_operator < >>>>>>> >> http://level-can.surge.sh/html/autoapi/airflow/contrib/operators/segment_track_event_operator/index.html >>>>>>>> >>>>>>> airflow.operators.redshift_to_s3_operator < >>>>>>> >> http://level-can.surge.sh/html/autoapi/airflow/operators/redshift_to_s3_operator/index.html >>>>>>>> >>>>>>> airflow.operators.s3_file_transform_operator < >>>>>>> >> http://level-can.surge.sh/html/autoapi/airflow/operators/s3_file_transform_operator/index.html >>>>>>>> >>>>>>> airflow.operators.s3_to_hive_operator < >>>>>>> >> http://level-can.surge.sh/html/autoapi/airflow/operators/s3_to_hive_operator/index.html >>>>>>>> >>>>>>> airflow.operators.s3_to_redshift_operator < >>>>>>> >> http://level-can.surge.sh/html/autoapi/airflow/operators/s3_to_redshift_operator/index.html >>>>>>>> >>>>>>> airflow.sensors.s3_key_sensor < >>>>>>> >> http://level-can.surge.sh/html/autoapi/airflow/sensors/s3_key_sensor/index.html >>>>>>>> >>>>>>> airflow.sensors.s3_prefix_sensor < >>>>>>> >> http://level-can.surge.sh/html/autoapi/airflow/sensors/s3_prefix_sensor/index.html >>>>>>>> >>>>>>> airflow.contrib.sensors.emr_base_sensor < >>>>>>> >> http://level-can.surge.sh/html/autoapi/airflow/contrib/sensors/emr_base_sensor/index.html >>>>>>>> >>>>>>> airflow.contrib.sensors.emr_job_flow_sensor < >>>>>>> >> http://level-can.surge.sh/html/autoapi/airflow/contrib/sensors/emr_job_flow_sensor/index.html >>>>>>>> >>>>>>> airflow.contrib.sensors.emr_step_sensor < >>>>>>> >> http://level-can.surge.sh/html/autoapi/airflow/contrib/sensors/emr_step_sensor/index.html >>>>>>>> >>>>>>> >>>>>>> And that was just before I got bored of looking for them :) >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> On 5 Feb 2019, at 16:25, Kamil Breguła <kamil.breg...@polidea.com> >>>>>>> wrote: >>>>>>>> >>>>>>>> I already have a POC: :-) >>>>>>>> >>>>>>>> Available at: http://level-can.surge.sh/html/autoapi/index.html >>>>>>>> >>>>>>>> I would like to point out that in addition to class documentation, >> you >>>>>>> can >>>>>>>> also document modules. >>>>>>>> >>>>>>> >> http://level-can.surge.sh/html/autoapi/airflow/executors/local_executor/index.html >>>>>>>> Currently, the `howto/operators.rst` file is used for this >> (Related link: >>>>>>>> >>>>>>> >> https://airflow.readthedocs.io/en/latest/howto/operator.html#cloudsqlqueryoperator >>>>>>>> ) >>>>>>>> >>>>>>>> >>>>>>>> On Tue, Feb 5, 2019 at 5:18 PM Ash Berlin-Taylor <a...@apache.org> >> wrote: >>>>>>>> >>>>>>>>>> We want to rewrite the `integration.rst` file so that it does not >>>>>>> contain >>>>>>>>>> duplicates from `code.rst ' (API Reference). In the next step, >>>>>>> introduce >>>>>>>>>> the reference API generation based on the source code that will >> replace >>>>>>>>> the >>>>>>>>>> `code.rst` file. >>>>>>>>> >>>>>>>>> :100: Yes please! >>>>>>>>> >>>>>>>>> >>>>>>>>> Given a number of integrations are across multiple files (n >> operators, >>>>>>> and >>>>>>>>> m hooks) my first thought is that something in integration.rst, >> or a >>>>>>> file >>>>>>>>> elsewhere in the docs/ tree is the place to put this. >>>>>>>>> >>>>>>>>> On epydoc vs a sphinx extension I lean very heavily towards the >> sphinx >>>>>>>>> extension, as we are already using much of sphinx. >>>>>>>>> >>>>>>>>> Can you create a _small_ example of what you'd propse for no.4 (I >> don't >>>>>>>>> want you to do a lot of work that might be wasted) >>>>>>>>> >>>>>>>>> -ash >>>>>>>>> >>>>>>>>> >>>>>>>>>> On 5 Feb 2019, at 15:55, Kamil Breguła < >> kamil.breg...@polidea.com> >>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> Hello community, >>>>>>>>>> >>>>>>>>>> While working on the documentation for the GCP operators, my >> team at >>>>>>>>>> Polidea encountered some confusion related to the structure of >> the >>>>>>>>>> documentation. >>>>>>>>>> >>>>>>>>>> Short story: >>>>>>>>>> >>>>>>>>>> We want to rewrite the `integration.rst` file so that it does not >>>>>>> contain >>>>>>>>>> duplicates from `code.rst ' (API Reference). In the next step, >>>>>>> introduce >>>>>>>>>> the reference API generation based on the source code that will >> replace >>>>>>>>> the >>>>>>>>>> `code.rst` file. >>>>>>>>>> >>>>>>>>>> Long story: >>>>>>>>>> >>>>>>>>>> Currently, the documentation contains two places where the >> description >>>>>>> of >>>>>>>>>> classes related to operators is included. They are `code.rst` and >>>>>>>>>> `integration.rst` files. >>>>>>>>>> >>>>>>>>>> The `integration.rst` file contains information about >> integration, in >>>>>>>>>> particular for Azure: Microsoft Azure, AWS: Amazon Web Services, >>>>>>>>>> Databricks, GCP: Google Cloud Platform, Qubole. Other >> integrations, >>>>>>>>>> however, do not have descriptions. >>>>>>>>>> >>>>>>>>>> The `code.rst` file contains “API Reference” which contains >> information >>>>>>>>>> about *all* classes including those included in the file >>>>>>>>> `integration.rst`. >>>>>>>>>> >>>>>>>>>> Such duplication, however, is problematic for several reasons: >>>>>>>>>> >>>>>>>>>> 1. >>>>>>>>>> >>>>>>>>>> Users may feel lost and may not know which section they should >> look >>>>>>>>> into. >>>>>>>>>> 2. >>>>>>>>>> >>>>>>>>>> Changes must be made in many places which leads to >> desynchronization. >>>>>>>>>> Most often, changes are made only in the source code, so they do >> not >>>>>>>>> appear >>>>>>>>>> in the generated documentation. >>>>>>>>>> 3. >>>>>>>>>> >>>>>>>>>> Linking to classes using the `class` directive for Sphinx is >>>>>>>>>> inconclusive - if the code is embedded both in `integration.rst` >> and >>>>>>>>>> `code.rst` using the `autoclass` directive, we’re not sure where >> the >>>>>>>>> user >>>>>>>>>> will be navigated. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> There are several solutions:: >>>>>>>>>> >>>>>>>>>> 1. >>>>>>>>>> >>>>>>>>>> Leave it as is. Then we need to agree on which `autoclass` >> directive >>>>>>>>>> should have the `no-index` flags. >>>>>>>>>> 2. >>>>>>>>>> >>>>>>>>>> Delete duplicates from the `code.rst` file and add a note about >> the >>>>>>>>>> `integration.rst` file in the `code.rst` file. >>>>>>>>>> 3. >>>>>>>>>> >>>>>>>>>> Delete duplicates from the `integration.rst` file and add a note >> about >>>>>>>>>> the `code.rst` file in the `integration.rst` file. >>>>>>>>>> 4. >>>>>>>>>> >>>>>>>>>> Delete information from both files and generate the API >> documentation >>>>>>>>>> always based only on the source code. This solution means that we >>>>>>> would >>>>>>>>>> have to write less documentation. >>>>>>>>>> There are ready tools that we can use: >>>>>>>>>> 1. >>>>>>>>>> >>>>>>>>>> epydoc - http://epydoc.sourceforge.net/ ; >>>>>>>>>> 2. >>>>>>>>>> >>>>>>>>>> autoapi extension for Sphinx - >>>>>>>>> https://github.com/rtfd/sphinx-autoapi >>>>>>>>>> ; >>>>>>>>>> 3. >>>>>>>>>> >>>>>>>>>> other - https://wiki.python.org/moin/DocumentationTools >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> The first, second, third solution does not solve all problems. In >>>>>>>>>> particular, we still need to complete the `code.rst` and >>>>>>>>> `integration.rst` >>>>>>>>>> files. The fourth solution solves all problems, but is the most >>>>>>> complex. >>>>>>>>> It >>>>>>>>>> is worth noting that mixing solutions is possible. For example, >> we can >>>>>>>>>> delete information from the file `integration.rst` as short term >>>>>>> solution >>>>>>>>>> and start working on creating another form of documentation as a >> long >>>>>>>>> term >>>>>>>>>> solution. This is the best option in our opinion. >>>>>>>>>> >>>>>>>>>> I’ve recently done a few activities related to this topic. >>>>>>>>>> >>>>>>>>>> First, I added the noindex flag to autoclass directives for all >>>>>>> operators >>>>>>>>>> in `integration.rst` file. In rare cases (If any), this caused >> links >>>>>>> that >>>>>>>>>> were previously directed to the file `integration.rst` to be >> redirected >>>>>>>>> to >>>>>>>>>> the `code.rst` file. Elements had to be linked using `:class:` >> instead >>>>>>>>> of a >>>>>>>>>> section link. Each operator is included in the new section in >> this >>>>>>> file. >>>>>>>>>> >>>>>>>>>> PR: https://github.com/apache/airflow/pull/4585 >>>>>>>>>> <https://github.com/apache/airflow/pull/4585/files> >>>>>>>>>> >>>>>>>>>> Second, I completed the `code.rst` file with the missing classes. >>>>>>>>>> >>>>>>>>>> PR: https://github.com/apache/airflow/pull/4644 >>>>>>>>>> >>>>>>>>>> I would like to ask which solution is the best in your opinion? >> What >>>>>>>>> steps >>>>>>>>>> should we take to make the documentation more enjoyable? >>>>>>>>>> >>>>>>>>>> Greetings >>>>>>>>>> >>>>>>>>>> Kamil Breguła >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> Kamil Breguła >>>>>>>> Polidea <https://www.polidea.com/> | Software Engineer >>>>>>>> >>>>>>>> M: +48 505 458 451 <+48505458451> >>>>>>>> E: kamil.breg...@polidea.com >>>>>>>> [image: Polidea] <https://www.polidea.com/> >>>>>>>> >>>>>>>> We create human & business stories through technology. >>>>>>>> Check out our projects! <https://www.polidea.com/our-work> >>>>>>>> [image: Github] <https://github.com/Polidea> [image: Facebook] >>>>>>>> <https://www.facebook.com/Polidea.Software> [image: Twitter] >>>>>>>> <https://twitter.com/polidea> [image: Linkedin] >>>>>>>> <https://www.linkedin.com/company/polidea> [image: Instagram] >>>>>>>> <https://instagram.com/polidea> [image: Behance] >>>>>>>> <https://www.behance.net/polidea> >>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> Kamil Breguła >>>>>> Polidea <https://www.polidea.com/> | Software Engineer >>>>>> >>>>>> M: +48 505 458 451 <+48505458451> >>>>>> E: kamil.breg...@polidea.com >>>>>> [image: Polidea] <https://www.polidea.com/> >>>>>> >>>>>> We create human & business stories through technology. >>>>>> Check out our projects! <https://www.polidea.com/our-work> >>>>>> [image: Github] <https://github.com/Polidea> [image: Facebook] >>>>>> <https://www.facebook.com/Polidea.Software> [image: Twitter] >>>>>> <https://twitter.com/polidea> [image: Linkedin] >>>>>> <https://www.linkedin.com/company/polidea> [image: Instagram] >>>>>> <https://instagram.com/polidea> [image: Behance] >>>>>> <https://www.behance.net/polidea> >>>>> >>>> >>> >>> >>> -- >>> >>> Kamil Breguła >>> Polidea | Software Engineer >>> >>> M: +48 505 458 451 >>> E: kamil.breg...@polidea.com >>> >>> We create human & business stories through technology. >>> Check out our projects! >> >> >> >> -- >> >> Kamil Breguła >> Polidea | Software Engineer >> >> M: +48 505 458 451 >> E: kamil.breg...@polidea.com >> >> We create human & business stories through technology. >> Check out our projects! >>
