One more idea - providing that we will implement AIP-7 Simplified Development Workflow ( https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-7+Simplified+development+workflow) - maybe it would be worthwhile to take a look at updates in the "Contributors" document and "Breeze" documentation. That's the new local development environment which allows to replicate Travis CI tests locally and have it ready in < 10 minutes from the scratch).
There are two sections/areas there that might need a bit of care and touch from Technical Writer: - I reorganised the "Setting up a development environment" chapter in "contributing" document https://github.com/PolideaInternal/airflow/blob/simplified-development-workflow/CONTRIBUTING.md - But especially I documented a lot about convenient and fast development with Breeze - including screencast, screenshots and a lot hints (including some from the recent Ash's Hangout): https://github.com/PolideaInternal/airflow/blob/simplified-development-workflow/BREEZE.rst Those are documents I imagine might be super-important for new contributors to Airflow and it would be great to have them structured and worded in the way that will be easy to follow. I am super-happy to iterate and cooperate if we can get this part included! J. On Fri, Apr 12, 2019 at 7:37 PM Aizhamal Nurmamat kyzy <aizha...@google.com.invalid> wrote: > Thank you all for sharing the ideas! > > We will start moving them to a separate doc and start scoping each idea > with mentors. > > If you have more ideas, keep posting in this thread. > > Thank you, > Aizhamal > > > On Wed, Apr 10, 2019 at 6:34 AM Driesprong, Fokko <fo...@driesprong.frl> > wrote: > > > Thanks Aizhamal for opening the discussion. > > > > I would like to see more documentation on the Kubernetes executor, which > is > > under heavy development, but the documentation is still lacking. I would > > love to see some more extended documentation on how to set it up, and how > > to make the most out of it. I think this the k8s executor has a lot of > > potential, but isn't fully utilized due to lack of documentation. > > > > Cheers, Fokko > > > > Op wo 10 apr. 2019 om 11:29 schreef Kaxil Naik <kaxiln...@gmail.com>: > > > > > +1 for using plantuml > > > > > > On Wed, Apr 10, 2019, 10:19 Kamil Gałuszka <ka...@flyrlabs.com> wrote: > > > > > > > In matter of documentation it would be nice to add PlantUML to > Sphinx, > > so > > > > we can visualise a lot of complexities of Airflow (see here: > > > > https://pypi.org/project/sphinxcontrib-plantuml/). > > > > > > > > Especially that sometimes one diagram can speak much clearly than > wall > > of > > > > text. > > > > > > > > Not sure how others see that, but I found PlantUML easy to use and > > > helping > > > > a lot dealing with complexity of documenting more advanced concepts. > I > > > can > > > > prepare JIRA ticket and PR if there is interest in that. > > > > > > > > Thanks > > > > Kamil > > > > > > > > On Wed, Apr 10, 2019 at 8:04 AM Kamil Breguła < > > kamil.breg...@polidea.com > > > > > > > > wrote: > > > > > > > > > - The structure of the documentation requires rethinking. > > > > > A lot of information is contained on one subpage - "concepts". Some > > > > > information should not be on this subpage e.x. .airflowignore > > > > > Other information should be extended, but then they will occupy too > > > much > > > > > space on this page e. x. creating relation including via chain > method > > > > > https://github.com/apache/airflow/pull/4779 > > > > > > > > > > - Difference between Kubernetes Executor and Operator > > > > > The difference is not obvious for beginners. > > > > > > > > > > - How to setup Kubernetes Executor. > > > > > It causes problems. > > > > > > > https://apache-airflow.slack.com/archives/CCV3FV9KL/p1554319091033300 > > > > > > > https://apache-airflow.slack.com/archives/CCV3FV9KL/p1553708569192000 > > > > > > > > > > - How Airflow distributes tasks. > > > > > Inspiration: > > > > > > > https://blog.sicara.com/using-airflow-with-celery-workers-54cb5212d405 > > > > > > > > > > - Description of Airflow architecture and description of components > > > > > Scheduler/webserver/worker/metadata db > > > > > I would like the part of this document to be a diagram similar to > > this: > > > > > https://imgur.com/a/YGpg5Wa > > > > > > > > > > - Setup Prometheus and Grafana - simple introduction > > > > > > > > > > - Recommendations on how to create new operators. > > > > > When reviewing new integration code(hook/operators) very often the > > > same > > > > > mistakes are made. I would like to have one document to which I > could > > > > > refer. > > > > > > > > > > - Xcom and macros are a lot of problems for new users. > > > > > The documentation is not well described. I am sending other people > > > > > references to describe the fields in the class when I want to > explain > > > > this > > > > > concept. > > > > > > > https://apache-airflow.slack.com/archives/CCR6P6JRL/p1551967217291400 > > > > > > > > > > I think it's worth reviewing the list of articles that the > community > > > > > wrote. This can be the source of inspiration for new pages. > > > > > https://github.com/jghoman/awesome-apache-airflow > > > > > > > > > > I will try to write down the problems that appear in the community. > > In > > > > the > > > > > next step, we can check whether the solution is in the > documentation > > > and > > > > is > > > > > clearly explained. > > > > > > > > > > I think PMCs should look more closely at the documentation when > > > > accepting a > > > > > new piece of code. I look at a large amount of PR and very often > > > suggest > > > > to > > > > > write new documentation. I feel that I am alone in this matter. Any > > new > > > > > functionality that has only docstring can be accepted. > > > > > For example: > > > > > > > > > > > > > > > > > > > > https://github.com/apache/airflow/commit/b78193240ad4266a7a22dced3832f51a9dce1897 > > > > > A new function has been added, but the description in doc is > missing. > > > > > https://airflow.readthedocs.io/en/latest/concepts.html#variables > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Apr 10, 2019, 3:01 AM Gerardo Curiel <gera...@gerar.do> > > wrote: > > > > > > > > > > > - Deployment in {AWS, Azure, GCP} > > > > > > - Monitoring, health checks, etc > > > > > > - Best practices for {dags, subdags, dag scheduling} > > > > > > > > > > > > Gerard Toonstra has some amazing docs here for some of these > topics > > > as > > > > > > well: https://gtoonstra.github.io/etl-with-airflow/ > > > > > > > > > > > > <https://gtoonstra.github.io/etl-with-airflow/> > > > > > > > > > > > > On 10 April 2019 at 5:56:52 am, Ash Berlin-Taylor ( > a...@apache.org) > > > > > wrote: > > > > > > > > > > > > - how to configure different types of connections > > > > > > - how to rerun a failed task. > > > > > > - tips for designing dags > > > > > > - writing custom operators (or tweaking the behaviour of a > built-in > > > > one) > > > > > > without needing plug in. > > > > > > > > > > > > Ash > > > > > > > > > > > > On 9 April 2019 20:52:02 BST, Aizhamal Nurmamat kyzy > > > > > > <aizha...@google.com.INVALID> wrote: > > > > > > >Hello all, > > > > > > > > > > > > > >Let's put together ideas for the Season of Docs. Can you please > > > share > > > > > > >any > > > > > > >documentation pain points that you are aware of as a user or as > a > > > > > > >contributor? Anything where you think the Airflow docs could be > > > > > > >improved > > > > > > >significantly? > > > > > > > > > > > > > >Some ideas that members of the community have shared with me > are: > > > > > > > > > > > > > >- Document how to unit-test DAGs > > > > > > >- Document how to set up a local testing environment > > > > > > >- Go through JIRA documentation requests. > > > > > > > > > > > > > >Please share any other ideas that you may have. There are no bad > > > ideas > > > > > > >here. We want to generate as many as possible, and prioritize > the > > > most > > > > > > >important ones later. > > > > > > > > > > > > > >Thank you, > > > > > > >Aizhamal > > > > > > > > > > > > -- > > > > > > Gerardo Curiel // https://gerar.do > > > > > > > > > > > > > > > > > > > > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> E: jarek.pot...@polidea.com
