Some other (crazy) stuff we were thinking about: Make Airflow data center aware (yarn) And related: make airflow data location aware Progress reporting for tasks
So if we fixed all other stuff and skip Alex' tasks we might get to this ;-) Bolke Verstuurd vanaf mijn iPad > Op 5 mrt. 2017 om 05:40 heeft Alex Van Boxel <a...@vanboxel.be> het volgende > geschreven: > > Hey Gurer, > > can I ask *not* to work on this, because I'm currently working in a major > rework for logging, because I'm working on consolidated logging and > integration with Google Cloud. > > - Streaming support for S3 logs in the webserver. > - Realtime logs for the webserver. > > This will be done through a plugin that should easiliy then be applied to > S3 or whatever logging system you have. It's quite some work to keep it > backward compatible so if you work on this it could conflict with the > plugin and I then need to refactor again. > > That said, I'm in SF for Google Cloud NEXT, so maybe I'll jump in to say hi > :-) > > > > On Fri, Mar 3, 2017 at 8:29 PM Gurer Kiratli > <gurer.kira...@airbnb.com.invalid> wrote: > >> Hi folks, >> >> As a reminder we are going to have an Airflow Hackathon next >> Thursday(03/09) and Friday(03/10) at the Airbnb HQ at San Francisco on 888 >> Brannan St. >> >> *Logistics* >> Couple folks have already reached out to me and one common question was if >> this is going to be a day time thing. Yes it will be during regular hours. >> We are going to start at 9:15am with optional breakfast we will commence >> the event at 10am. We can decide how long we want to go when we are here, >> if people wanna work through late night on Thursday, we can do that as >> well. We will end on Friday at around 5pm. >> >> We will provide the meals, snacks, and the beverages. >> >> *Possible Projects* >> As the Airbnb team we met internally and here are some things that we think >> might be interesting to work on: >> >> - Performance Improvements >> - Profiling support for the webserver. >> - Homepage, task instance page loading times performance >> improvements. >> - Improve Airflow DB queries’ performance. >> - Making the scheduler schedule tasks faster. >> - Make a tool for restarting tasks after system failures/outages. >> - Improving test coverage for core components. >> - Creating REST APIs for Airflow operations. >> - Improve UI to enable operations like cancel, mark success, clear for >> all the tasks of the DAG from the UI. >> - Streaming support for S3 logs in the webserver. >> - Realtime logs for the webserver. >> >> Also you can check Airflow 2017 roadmap document >> <https://cwiki.apache.org/confluence/display/AIRFLOW/2017+Roadmap+Items> >> for >> other interesting things to work on and you can see the community survey >> results <https://www.surveymonkey.com/results/SM-99G9YHS3/> on the roadmap >> items' prioritization. >> >> *How to participate? Questions?* >> Please reach out to me at gurer.kira...@airbnb.com by EOD Tuesday(03/07) >> if you would like to participate. Also let me know if you have any >> questions. >> >> Cheers, >> >> Gurer >> > -- > _/ > _/ Alex Van Boxel
