In matter of documentation it would be nice to add PlantUML to Sphinx, so we can visualise a lot of complexities of Airflow (see here: https://pypi.org/project/sphinxcontrib-plantuml/).
Especially that sometimes one diagram can speak much clearly than wall of text. Not sure how others see that, but I found PlantUML easy to use and helping a lot dealing with complexity of documenting more advanced concepts. I can prepare JIRA ticket and PR if there is interest in that. Thanks Kamil On Wed, Apr 10, 2019 at 8:04 AM Kamil Breguła <kamil.breg...@polidea.com> wrote: > - The structure of the documentation requires rethinking. > A lot of information is contained on one subpage - "concepts". Some > information should not be on this subpage e.x. .airflowignore > Other information should be extended, but then they will occupy too much > space on this page e. x. creating relation including via chain method > https://github.com/apache/airflow/pull/4779 > > - Difference between Kubernetes Executor and Operator > The difference is not obvious for beginners. > > - How to setup Kubernetes Executor. > It causes problems. > https://apache-airflow.slack.com/archives/CCV3FV9KL/p1554319091033300 > https://apache-airflow.slack.com/archives/CCV3FV9KL/p1553708569192000 > > - How Airflow distributes tasks. > Inspiration: > https://blog.sicara.com/using-airflow-with-celery-workers-54cb5212d405 > > - Description of Airflow architecture and description of components > Scheduler/webserver/worker/metadata db > I would like the part of this document to be a diagram similar to this: > https://imgur.com/a/YGpg5Wa > > - Setup Prometheus and Grafana - simple introduction > > - Recommendations on how to create new operators. > When reviewing new integration code(hook/operators) very often the same > mistakes are made. I would like to have one document to which I could > refer. > > - Xcom and macros are a lot of problems for new users. > The documentation is not well described. I am sending other people > references to describe the fields in the class when I want to explain this > concept. > https://apache-airflow.slack.com/archives/CCR6P6JRL/p1551967217291400 > > I think it's worth reviewing the list of articles that the community > wrote. This can be the source of inspiration for new pages. > https://github.com/jghoman/awesome-apache-airflow > > I will try to write down the problems that appear in the community. In the > next step, we can check whether the solution is in the documentation and is > clearly explained. > > I think PMCs should look more closely at the documentation when accepting a > new piece of code. I look at a large amount of PR and very often suggest to > write new documentation. I feel that I am alone in this matter. Any new > functionality that has only docstring can be accepted. > For example: > > https://github.com/apache/airflow/commit/b78193240ad4266a7a22dced3832f51a9dce1897 > A new function has been added, but the description in doc is missing. > https://airflow.readthedocs.io/en/latest/concepts.html#variables > > > > > > On Wed, Apr 10, 2019, 3:01 AM Gerardo Curiel <gera...@gerar.do> wrote: > > > - Deployment in {AWS, Azure, GCP} > > - Monitoring, health checks, etc > > - Best practices for {dags, subdags, dag scheduling} > > > > Gerard Toonstra has some amazing docs here for some of these topics as > > well: https://gtoonstra.github.io/etl-with-airflow/ > > > > <https://gtoonstra.github.io/etl-with-airflow/> > > > > On 10 April 2019 at 5:56:52 am, Ash Berlin-Taylor (a...@apache.org) > wrote: > > > > - how to configure different types of connections > > - how to rerun a failed task. > > - tips for designing dags > > - writing custom operators (or tweaking the behaviour of a built-in one) > > without needing plug in. > > > > Ash > > > > On 9 April 2019 20:52:02 BST, Aizhamal Nurmamat kyzy > > <aizha...@google.com.INVALID> wrote: > > >Hello all, > > > > > >Let's put together ideas for the Season of Docs. Can you please share > > >any > > >documentation pain points that you are aware of as a user or as a > > >contributor? Anything where you think the Airflow docs could be > > >improved > > >significantly? > > > > > >Some ideas that members of the community have shared with me are: > > > > > >- Document how to unit-test DAGs > > >- Document how to set up a local testing environment > > >- Go through JIRA documentation requests. > > > > > >Please share any other ideas that you may have. There are no bad ideas > > >here. We want to generate as many as possible, and prioritize the most > > >important ones later. > > > > > >Thank you, > > >Aizhamal > > > > -- > > Gerardo Curiel // https://gerar.do > > >
