Hi everyone, Here are a few updates on our application for Season of Docs:
1. I updated the document with more descriptions for each project idea and specified the deliverables[1]. 2. To submit our application, we need to publish our project ideas list. For this I will open Jira tickets with “gsod2019” tag for each project idea on Friday. So please, update the document if you want to add any improvements to your project idea. 3. We should also think of adding a small page in Airflow website or Wiki that contains all the ideas in one place. I was thinking maybe somewhere here [2] or at the top level in the Wiki[3]. Please let me know your thoughts/preferences on this. 4. All mentors, please complete the mentor registration form [4] by next Tuesday, April 23rd to successfully enroll into the program. Thank you, Aizhamal [1] https://docs.google.com/document/d/1HoL_yjNYiTAP9IxSlhx3EUnPFU4l9WOT9EnwBZjCZo0/edit# [2] https://airflow.apache.org/project.html [3] https://cwiki.apache.org/confluence/display/AIRFLOW/ [4] https://docs.google.com/forms/d/e/1FAIpQLSe-JjGvaKKGWZOXxrorONhB8qN3mjPrB9ZVkcsntR73Cv_K7g/viewform On Sat, Apr 13, 2019 at 3:46 AM Jarek Potiuk <[email protected]> wrote: > One more idea - providing that we will implement AIP-7 Simplified > Development Workflow ( > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-7+Simplified+development+workflow > ) > - maybe it would be worthwhile to take a look at updates in the > "Contributors" document and "Breeze" documentation. That's the new local > development environment which allows to replicate Travis CI tests locally > and have it ready in < 10 minutes from the scratch). > > There are two sections/areas there that might need a bit of care and touch > from Technical Writer: > > - I reorganised the "Setting up a development environment" chapter in > "contributing" document > > > https://github.com/PolideaInternal/airflow/blob/simplified-development-workflow/CONTRIBUTING.md > > - But especially I documented a lot about convenient and fast > development with Breeze - including screencast, screenshots and a lot > hints > (including some from the recent Ash's Hangout): > > > https://github.com/PolideaInternal/airflow/blob/simplified-development-workflow/BREEZE.rst > > Those are documents I imagine might be super-important for new contributors > to Airflow and it would be great to have them structured and worded in the > way that will be easy to follow. I am super-happy to iterate and cooperate > if we can get this part included! > > J. > > > On Fri, Apr 12, 2019 at 7:37 PM Aizhamal Nurmamat kyzy > <[email protected]> wrote: > > > Thank you all for sharing the ideas! > > > > We will start moving them to a separate doc and start scoping each idea > > with mentors. > > > > If you have more ideas, keep posting in this thread. > > > > Thank you, > > Aizhamal > > > > > > On Wed, Apr 10, 2019 at 6:34 AM Driesprong, Fokko <[email protected]> > > wrote: > > > > > Thanks Aizhamal for opening the discussion. > > > > > > I would like to see more documentation on the Kubernetes executor, > which > > is > > > under heavy development, but the documentation is still lacking. I > would > > > love to see some more extended documentation on how to set it up, and > how > > > to make the most out of it. I think this the k8s executor has a lot of > > > potential, but isn't fully utilized due to lack of documentation. > > > > > > Cheers, Fokko > > > > > > Op wo 10 apr. 2019 om 11:29 schreef Kaxil Naik <[email protected]>: > > > > > > > +1 for using plantuml > > > > > > > > On Wed, Apr 10, 2019, 10:19 Kamil Gałuszka <[email protected]> > wrote: > > > > > > > > > In matter of documentation it would be nice to add PlantUML to > > Sphinx, > > > so > > > > > we can visualise a lot of complexities of Airflow (see here: > > > > > https://pypi.org/project/sphinxcontrib-plantuml/). > > > > > > > > > > Especially that sometimes one diagram can speak much clearly than > > wall > > > of > > > > > text. > > > > > > > > > > Not sure how others see that, but I found PlantUML easy to use and > > > > helping > > > > > a lot dealing with complexity of documenting more advanced > concepts. > > I > > > > can > > > > > prepare JIRA ticket and PR if there is interest in that. > > > > > > > > > > Thanks > > > > > Kamil > > > > > > > > > > On Wed, Apr 10, 2019 at 8:04 AM Kamil Breguła < > > > [email protected] > > > > > > > > > > wrote: > > > > > > > > > > > - The structure of the documentation requires rethinking. > > > > > > A lot of information is contained on one subpage - "concepts". > Some > > > > > > information should not be on this subpage e.x. .airflowignore > > > > > > Other information should be extended, but then they will occupy > too > > > > much > > > > > > space on this page e. x. creating relation including via chain > > method > > > > > > https://github.com/apache/airflow/pull/4779 > > > > > > > > > > > > - Difference between Kubernetes Executor and Operator > > > > > > The difference is not obvious for beginners. > > > > > > > > > > > > - How to setup Kubernetes Executor. > > > > > > It causes problems. > > > > > > > > > https://apache-airflow.slack.com/archives/CCV3FV9KL/p1554319091033300 > > > > > > > > > https://apache-airflow.slack.com/archives/CCV3FV9KL/p1553708569192000 > > > > > > > > > > > > - How Airflow distributes tasks. > > > > > > Inspiration: > > > > > > > > > https://blog.sicara.com/using-airflow-with-celery-workers-54cb5212d405 > > > > > > > > > > > > - Description of Airflow architecture and description of > components > > > > > > Scheduler/webserver/worker/metadata db > > > > > > I would like the part of this document to be a diagram similar to > > > this: > > > > > > https://imgur.com/a/YGpg5Wa > > > > > > > > > > > > - Setup Prometheus and Grafana - simple introduction > > > > > > > > > > > > - Recommendations on how to create new operators. > > > > > > When reviewing new integration code(hook/operators) very often > the > > > > same > > > > > > mistakes are made. I would like to have one document to which I > > could > > > > > > refer. > > > > > > > > > > > > - Xcom and macros are a lot of problems for new users. > > > > > > The documentation is not well described. I am sending other > people > > > > > > references to describe the fields in the class when I want to > > explain > > > > > this > > > > > > concept. > > > > > > > > > https://apache-airflow.slack.com/archives/CCR6P6JRL/p1551967217291400 > > > > > > > > > > > > I think it's worth reviewing the list of articles that the > > community > > > > > > wrote. This can be the source of inspiration for new pages. > > > > > > https://github.com/jghoman/awesome-apache-airflow > > > > > > > > > > > > I will try to write down the problems that appear in the > community. > > > In > > > > > the > > > > > > next step, we can check whether the solution is in the > > documentation > > > > and > > > > > is > > > > > > clearly explained. > > > > > > > > > > > > I think PMCs should look more closely at the documentation when > > > > > accepting a > > > > > > new piece of code. I look at a large amount of PR and very often > > > > suggest > > > > > to > > > > > > write new documentation. I feel that I am alone in this matter. > Any > > > new > > > > > > functionality that has only docstring can be accepted. > > > > > > For example: > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/airflow/commit/b78193240ad4266a7a22dced3832f51a9dce1897 > > > > > > A new function has been added, but the description in doc is > > missing. > > > > > > https://airflow.readthedocs.io/en/latest/concepts.html#variables > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Apr 10, 2019, 3:01 AM Gerardo Curiel <[email protected]> > > > wrote: > > > > > > > > > > > > > - Deployment in {AWS, Azure, GCP} > > > > > > > - Monitoring, health checks, etc > > > > > > > - Best practices for {dags, subdags, dag scheduling} > > > > > > > > > > > > > > Gerard Toonstra has some amazing docs here for some of these > > topics > > > > as > > > > > > > well: https://gtoonstra.github.io/etl-with-airflow/ > > > > > > > > > > > > > > <https://gtoonstra.github.io/etl-with-airflow/> > > > > > > > > > > > > > > On 10 April 2019 at 5:56:52 am, Ash Berlin-Taylor ( > > [email protected]) > > > > > > wrote: > > > > > > > > > > > > > > - how to configure different types of connections > > > > > > > - how to rerun a failed task. > > > > > > > - tips for designing dags > > > > > > > - writing custom operators (or tweaking the behaviour of a > > built-in > > > > > one) > > > > > > > without needing plug in. > > > > > > > > > > > > > > Ash > > > > > > > > > > > > > > On 9 April 2019 20:52:02 BST, Aizhamal Nurmamat kyzy > > > > > > > <[email protected]> wrote: > > > > > > > >Hello all, > > > > > > > > > > > > > > > >Let's put together ideas for the Season of Docs. Can you > please > > > > share > > > > > > > >any > > > > > > > >documentation pain points that you are aware of as a user or > as > > a > > > > > > > >contributor? Anything where you think the Airflow docs could > be > > > > > > > >improved > > > > > > > >significantly? > > > > > > > > > > > > > > > >Some ideas that members of the community have shared with me > > are: > > > > > > > > > > > > > > > >- Document how to unit-test DAGs > > > > > > > >- Document how to set up a local testing environment > > > > > > > >- Go through JIRA documentation requests. > > > > > > > > > > > > > > > >Please share any other ideas that you may have. There are no > bad > > > > ideas > > > > > > > >here. We want to generate as many as possible, and prioritize > > the > > > > most > > > > > > > >important ones later. > > > > > > > > > > > > > > > >Thank you, > > > > > > > >Aizhamal > > > > > > > > > > > > > > -- > > > > > > > Gerardo Curiel // https://gerar.do > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > E: [email protected] >
