This is indeed great. There is a WEALTH of information in it - a lot to digest and work on .. Some of those actions to work on are pretty vague, some of them very concrete, but certainly there is A LOT to improve.
Without going into "what exactly to improve", I see a few general types of improvement we can do: a) actually improve something in airflow code to make debugging better and easier (example: review code and improve vague messages) b) add documentation on how to do things that are non obvious (or improve discoverability of existing documentation) c) increase awareness - maybe via blog posts, recorded talks, asking influencers (Mark Lamberti comes to my mind immediately) to explain how debugging is already possible/easy but people do not know about it (example: I clearly see for example that our recent open-telemetry integration with traces is used sporadically, while it solves a lot of problems with tracking what's going on in a distributed environment that is mentioned as a big pain) d) develop new tooling hat will make it easier for people to debug (for example IDE integration) But then there is an important question of what to focus on first. Because we won't be able to address everything quickly. In order to do something actionable and really "progress" on improving ehre, I think those who are interested in improving debuggability should read the report in detail and possibly (my proposal) we should all discuss here what should be the most impactful and important things to work on. And maybe people seeing it here should propose (say) 3 top things that they think are most important and impactful, and then we could see if we have clear contenders. Maybe even we can eventually vote on it, if there won't be a clear consensus. I have my 3 picks already, but I do not want to skew other's choices :). WDYT? On Sat, Oct 19, 2024 at 2:05 AM Vikram Koka <vik...@astronomer.io.invalid> wrote: > +1 > Excellent work, Omkar, lliya, and Amogh. > > Really appreciate the details in the report. It was fascinating reading! > Looking forward to seeing the changes in response. > > Vikram > > > On Fri, Oct 18, 2024 at 4:15 PM Kaxil Naik <kaxiln...@gmail.com> wrote: > > > Great work here Omkar, Iliya & Amogh for all your hard work on this 👏🏻 > > > > On Fri, 18 Oct 2024 at 23:25, Omkar P <droiddev5...@gmail.com> wrote: > > > > > Hey all, > > > > > > Hope you are doing great! > > > > > > So recently the Airflow debugging survey made the rounds and we've > > > received a total of 69 responses. > > > > > > We've analyzed those responses and created a report which we can refer > to > > > for creation of GH issues to improve Airflow. > > > > > > There's a new GH project created today for these issues, in case any of > > you > > > want to contribute: > https://github.com/orgs/apache/projects/421/views/1 > > > > > > Currently just a few high-level issues are created, which we can > discuss > > > and drill down one level further to get the implementation issues. > > > > > > For your reference, the survey report can be viewed here: > > > > > > > > > https://html-preview.github.io/?url=https://github.com/omkar-foss/airflow-debugging-survey-2024/blob/main/report/html/survey_report.html > > > > > > And all our debugging survey related resources (form pdf, responses > data, > > > notebook, etc.) are currently here: > > > https://github.com/omkar-foss/airflow-debugging-survey-2024 > > > > > > We'll eventually move these from my GH repo to the Airflow site or > blog. > > > Meanwhile, look forward to any suggestions or inputs from you on the GH > > > issues. Cheers. > > > > > > Regards, > > > Omkar > > > > > >
