- The structure of the documentation requires rethinking. A lot of information is contained on one subpage - "concepts". Some information should not be on this subpage e.x. .airflowignore Other information should be extended, but then they will occupy too much space on this page e. x. creating relation including via chain method https://github.com/apache/airflow/pull/4779
- Difference between Kubernetes Executor and Operator The difference is not obvious for beginners. - How to setup Kubernetes Executor. It causes problems. https://apache-airflow.slack.com/archives/CCV3FV9KL/p1554319091033300 https://apache-airflow.slack.com/archives/CCV3FV9KL/p1553708569192000 - How Airflow distributes tasks. Inspiration: https://blog.sicara.com/using-airflow-with-celery-workers-54cb5212d405 - Description of Airflow architecture and description of components Scheduler/webserver/worker/metadata db I would like the part of this document to be a diagram similar to this: https://imgur.com/a/YGpg5Wa - Setup Prometheus and Grafana - simple introduction - Recommendations on how to create new operators. When reviewing new integration code(hook/operators) very often the same mistakes are made. I would like to have one document to which I could refer. - Xcom and macros are a lot of problems for new users. The documentation is not well described. I am sending other people references to describe the fields in the class when I want to explain this concept. https://apache-airflow.slack.com/archives/CCR6P6JRL/p1551967217291400 I think it's worth reviewing the list of articles that the community wrote. This can be the source of inspiration for new pages. https://github.com/jghoman/awesome-apache-airflow I will try to write down the problems that appear in the community. In the next step, we can check whether the solution is in the documentation and is clearly explained. I think PMCs should look more closely at the documentation when accepting a new piece of code. I look at a large amount of PR and very often suggest to write new documentation. I feel that I am alone in this matter. Any new functionality that has only docstring can be accepted. For example: https://github.com/apache/airflow/commit/b78193240ad4266a7a22dced3832f51a9dce1897 A new function has been added, but the description in doc is missing. https://airflow.readthedocs.io/en/latest/concepts.html#variables On Wed, Apr 10, 2019, 3:01 AM Gerardo Curiel <[email protected]> wrote: > - Deployment in {AWS, Azure, GCP} > - Monitoring, health checks, etc > - Best practices for {dags, subdags, dag scheduling} > > Gerard Toonstra has some amazing docs here for some of these topics as > well: https://gtoonstra.github.io/etl-with-airflow/ > > <https://gtoonstra.github.io/etl-with-airflow/> > > On 10 April 2019 at 5:56:52 am, Ash Berlin-Taylor ([email protected]) wrote: > > - how to configure different types of connections > - how to rerun a failed task. > - tips for designing dags > - writing custom operators (or tweaking the behaviour of a built-in one) > without needing plug in. > > Ash > > On 9 April 2019 20:52:02 BST, Aizhamal Nurmamat kyzy > <[email protected]> wrote: > >Hello all, > > > >Let's put together ideas for the Season of Docs. Can you please share > >any > >documentation pain points that you are aware of as a user or as a > >contributor? Anything where you think the Airflow docs could be > >improved > >significantly? > > > >Some ideas that members of the community have shared with me are: > > > >- Document how to unit-test DAGs > >- Document how to set up a local testing environment > >- Go through JIRA documentation requests. > > > >Please share any other ideas that you may have. There are no bad ideas > >here. We want to generate as many as possible, and prioritize the most > >important ones later. > > > >Thank you, > >Aizhamal > > -- > Gerardo Curiel // https://gerar.do >
