IF YOU WOULD LIKE TO SEE THE FORMATTED VERSION OF THE NEWSLETTER, PLEASE GO HERE <https://www.astronomer.io/community-newsletter/december-2023/>
Hey Team, Happy New Year! This community accomplished so much in the month of December, and I’m excited to be sharing it all with you now! From a new release to record breaking survey results, we closed out 2023 with the same vivacity we entered it with. Highlights from the month of December include: - The release of Airflow 2.8 <https://airflow.apache.org/blog/airflow-2.8.0/ > - The launch of the 2023 Survey Results <https://airflow.apache.org/survey/> - Announcing Airflow Summit in San Francisco Sept. 10-12th, 2024! < https://airflowsummit.org/> As always, please reach out with items for the next issue to: < [email protected]> Thanks for reading! AIRFLOW SUMMIT 2024 SEPTEMBER 10TH-12TH, SAN FRANCISCO It’s official folks! We’re hosting Airflow Summit 2024 on September 10-12th in San Francisco, CA! Sign up for updates on the conference, including access to early bird tickets here: <https://airflowsummit.org/> THE AIRFLOW 2023 USER SURVEY RESULTS ARE IN! The results are IN for the Airflow 2023 User Survey! This year, we had a record breaking number of respondents, and are thankful to each and every person that took the time to fill out the survey. Please view the full Infographic here! <https://airflow.apache.org/survey/> AIRFLOW 2.8.0 Airflow 2.8 was released on December 18th and includes a host of significant enhancements and new features that will greatly benefit our community. - PyPI <https://pypi.org/project/apache-airflow/> - Release Notes <https://pypi.org/project/apache-airflow/> - Docs <https://airflow.apache.org/blog/airflow-2.8.0/> - Sources < https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-sources.html > - Docker <https://hub.docker.com/r/apache/airflow/tags/?page=1&name=2.8.0> - Constraints <https://github.com/apache/airflow/tree/constraints-2.8.0> PROVIDERS New versions of 59 Airflow Providers packages were released. For all source releases, PyPI packages and docs, see: - Sources < https://airflow.apache.org/docs/apache-airflow-providers/installing-from-sources > - PyPI < https://airflow.apache.org/docs/apache-airflow-providers/installing-from-pypi > - Docs <https://airflow.apache.org/docs/> Ecosystem Initial releases: Astronomer Cosmos 1.3.0 <https://pypi.org/project/astronomer-cosmos/> OpenLineage 1.7.0 < https://github.com/OpenLineage/OpenLineage/releases/tag/1.7.0> UPCOMING EVENTS 1/11 Webinar: Airflow Town Hall < https://astronomer.zoom.us/meeting/register/tZUpdOusrTItG90cw69WtvO2tOKfZbxxbQqH#/registration > 1/11 Webinar: What’s new in Airflow 2.8 < https://www.astronomer.io/events/webinars/whats-new-in-airflow-2-8/> 1/18 Webinar: How to manage connections in Airflow < https://www.astronomer.io/events/webinars/how-to-manage-connections-in-airflow/ > 1/31/24 Webinar: Run MLOps production pipelines with Astronomer and Snowpark < https://www.snowflake.com/webinar/virtual-hands-on-labs/snowpark-day-where-python-developers-can-learn-the-latest-innovations-for-ai-ml-workflows-in-snowflake-2024-01-31/?utm_source=astronomer&utm_medium=partner&utm_campaign=na--en-customers&utm_content=-wb-snowpark-day-2024-01-31 > 2/3-4 Conference in Brussels: Data workflows: translating dbt to Apache Airflow < https://fosdem.org/2024/schedule/event/fosdem-2024-1651-data-workflows-translating-dbt-to-apache-airflow/> & `New` Workflow Orchestrator in town: "Apache Airflow 2.x" < https://fosdem.org/2024/schedule/event/fosdem-2024-1652--new-workflow-orchestrator-in-town-apache-airflow-2-x-/ > 2/9 Meetup in São Paulo: First São Paulo Airflow Meetup at iFood! < https://www.meetup.com/sao-paulo-apache-airflow-meetup-group/events/298283822/ > 2/13 Meetup in Bangalore: Bangalore Airflow Meetup @ Cloudera < https://www.meetup.com/bangalore-apache-airflow-meetup/events/298355258/> RECENT EVENTS 12/7 Webinar: What’s New in the Astro Platform Release, Q4 2023 < https://www.astronomer.io/events/webinars/whats-new-in-the-astro-platform-release-q4-2023/ > 12/7 Meetup in Prague: First Airflow Prague Meetup with Kiwi.com & Astronomer < https://www.meetup.com/prague-airflow-meetup-group/events/296837456/> 12/12 Meetup in Hyderabad: Hyderabad Airflow Meetup @ Microsoft HQ < https://www.meetup.com/hyderabad-apache-airflow-meetup-group/events/297490529/ > 12/14 Webinar: Orchestrate next-generation AI tools: Explore six new Airflow providers < https://www.astronomer.io/events/webinars/orchestrate-next-generation-ai-tools-explore-six-new-airflow-providers/ > PR OF THE MONTH PR #35719 Add XCom tab to Grid <https://github.com/apache/airflow/pull/35719> By @huyduong <https://github.com/hduong-mwam> The December 2023 PR of the Month comes in amidst a medley of relevant PR’s for the Airflow community. However, in the words of Committer Amogh Desai, “Having a nice UI fix come in, that too from a first time contributor, makes it a winner for me!” COMMUNICATIONS DIGEST Airflow Updates and Releases Apache Airflow 2.8.0 is here by Ephraim Anierobi on the Airflow Blog < https://airflow.apache.org/blog/airflow-2.8.0/> Configuring the KubernetesExecutor to Hum at Etsy by Victor Chiapaikeo on Airflow Medium < https://medium.com/apache-airflow/configuring-the-kubernetesexecutor-to-hum-at-etsy-94c2924e9526 > Airflow 2.8 Release: Meet the Contributors on Youtube < https://youtu.be/Ubh-4i1pxX0?feature=shared> Unraveling the Code: Navigating a CI/Release Security Vulnerability in Apache Airflow by Jarek Potiuk on Airflow Medium < https://medium.com/apache-airflow/unraveling-the-code-navigating-a-ci-release-security-vulnerability-in-apache-airflow-620214a96297 > Introducing Airflow 2.8 by Kenten Danas on the Astronomer Blog < https://www.astronomer.io/blog/introducing-airflow-2-8/> Airflow in Practice and Use Cases Apache Airflow at Adyen: Our journey and challenges to achieve reliability at scale by Jorrick Sleijster and Natasha Shroff on Data Engineering Weekly < https://www.dataengineeringweekly.com/i/139860695/adyen-apache-airflow-at-adyen-our-journey-and-challenges-to-achieve-reliability-at-scale > How to share code across Airflow projects on Astronomer Learn < https://docs.astronomer.io/learn/sharing-code-multiple-projects> Airflow in action: providing environmental forecast services by Eloi Codina-Torras on Airflow Medium < https://medium.com/apache-airflow/airflow-in-action-providing-environmental-forecast-services-1a5a86110864 > Use Airflow object storage to interact with cloud storage in an ML pipeline on Astronomer Learn < https://docs.astronomer.io/learn/airflow-object-storage-tutorial> Leveraging Customization in Airflow with Databricks by Hanish Bansal on Airflow Medium < https://medium.com/apache-airflow/leveraging-customization-in-airflow-with-databricks-d263fde9a24d > LIVE with Astronomer: How Chartboost Uses Airflow Plugins with George Yates on Youtube <https://www.youtube.com/watch?v=10-XCFKnmM4> Deploy Rollbacks: Upgrade Airflow and Deploy DAGs with Confidence by David Koenitzer on the Astronomer Blog < https://www.astronomer.io/blog/deploy-rollbacks-upgrade-airflow-and-deploy-dags-with-confidence/ > Use a listener to send a Slack notification when a dataset is updated on Astronomer Learn <https://docs.astronomer.io/learn/airflow-listeners> LIVE with Astronomer: DAG Serialization Deep Dive with Utkarsh! by George Yates on Youtube <https://www.youtube.com/watch?v=yiGclH8RMzs> Airflow Innovations Introducing Apache Airflow™ on Astro – an Azure Native ISV Service by Conor Beverland on the Astronomer Blog < https://www.astronomer.io/blog/introducing-apache-airflow-on-astro-an-azure-native-isv-service/ > Accelerating ML Application Development: Production-Ready Airflow Integrations with Critical AI Tools by Steven Hillion on the Astronomer Blog < https://www.astronomer.io/blog/accelerating-ml-app-development-with-ai-airflow-integrations/ > Enhanced Authentication Security to your Data Services on Azure with Astro by Piotr Chomiak on the Astronomer Blog < https://www.astronomer.io/blog/enhanced-authentication-security-to-your-data-services-on-azure-with-astro/ > Exploring Airflow’s XComs: Unveiling the Power of Shared State by Amitabh Roy Choudhary on Airflow Medium < https://medium.com/apache-airflow/exploring-airflows-xcoms-unveiling-the-power-of-shared-state-a71270d824ba > Introducing the Astro Platform Release, Q4 2023 by Juliana O’Donohue on the Astronomer Blog < https://www.astronomer.io/blog/introducing-the-astro-platform-release-q4-2023/ > DEVLIST DIGEST Votes: Airflow Providers prepared on December 08, 2023 < https://lists.apache.org/thread/coy1t0dhtmpq1xyr7zr06ccmdfr70pkk> [RESULT][VOTE] Airflow Providers - release of December 08, 2023 < https://lists.apache.org/thread/cpbmng8z9wbrxgxgnyyr669xsosrtcsh> Remove 'apache.scoop' provider < https://lists.apache.org/thread/wnov3n2m2v0blnxl3q413mj8vphdrz4b> Remove 'plexus' provider < https://lists.apache.org/thread/6so1qh8kf088spd2y4nl0o5tpbyn0x3b> Release Airflow 2.8.0 from 2.8.0rc1 < https://lists.apache.org/thread/x5wt2z3ddvxpo1lkzotfbo94phncbs4n> Release Airflow 2.8.0 from 2.8.0rc2 < https://lists.apache.org/thread/3c8st8kyq2wbtbmsrh38sv1b34sy3w4y> Release Airflow 2.8.0 from 2.8.0rc3 < https://lists.apache.org/thread/7mxlh0m3v7xq717yobog0023djtpvbk2> Airflow Providers prepared on December 12, 2023 < https://lists.apache.org/thread/1r9w8mxw7165x29d118brgvpz3161zqw> [RESULT][VOTE] Airflow Providers - release of 12 December 2023 < https://lists.apache.org/thread/ll0pf7lmr0sfk4zgoj9y47ot5gy35hcm> Release Airflow 2.8.0 from 2.8.0rc4 < https://lists.apache.org/thread/x8tzt1k3rk63wdb71dqnbxvnkrhoc3nw> [RESULT][VOTE] Release Airflow 2.8.0 from 2.8.0rc4 < https://lists.apache.org/thread/ll0pf7lmr0sfk4zgoj9y47ot5gy35hcm> Switch 2.8.0 images to use MariaDB client < https://lists.apache.org/thread/rxbyxg11jg7y35k8om0f8wgb2l9h459l> Making Listeners API non-experimental < https://lists.apache.org/thread/8hr9bnfscgtf0ng3g44z13tthgk33oz4> Release Airflow Python Client 2.8.0 from 2.8.0rc1 < https://lists.apache.org/thread/pg9xod30s3prvk0ctp3gd220hmtl4r1o> [RESULT][VOTE] Release Airflow Python Client 2.8.0 from 2.8.0rc1 < https://lists.apache.org/thread/bv69z5qpjf2j3dz7fj4cdj2c3bfb04sl> Airflow Providers prepared on December 23, 2023 < https://lists.apache.org/thread/mrkqrsl7d4l482dr935wlozbgcfw56s1> [RESULT][VOTE] Airflow Providers - release of 23rd of December 2023 < https://lists.apache.org/thread/pctjv209k2gpro53dthytsh9xjqxbqmw> Airflow Providers prepared on December 28, 2023 < https://lists.apache.org/thread/cs6mcvpn2lk9w2p4oz43t20z3fg5nl7l> [RESULT][VOTE] Airflow Providers - release of 28th December 2023 < https://lists.apache.org/thread/brvbl5ndjcmfzfzcv6llrx0dvn186qo5> Airflow Providers prepared on December 31, 2023 < https://lists.apache.org/thread/gzfy6ccwr5hj2sp4bnzm03xqto9gk4y4> Discussed: Serialization in Apache Airflow: Clarifications and Future Directions < https://lists.apache.org/thread/28r9x52cc51tcfxc5c6n3vh9frgp287x> Helm Chart Version 1.9.0 Unavailable < https://lists.apache.org/thread/kgp308ws0vo324o8070mtlc9m7o9qsq0> Common.sql interface / API < https://lists.apache.org/thread/pxt1tq6cmb69sncz1q4dp4z2o9zms7of> Allowlist in serialization [was: Serialization in Apache Airflow] < https://lists.apache.org/thread/1s6xpqcbdb3wrgtjv8op5zzkbsrsoc9f> Capturing Architectural decisions (ADRS?) < https://lists.apache.org/thread/kzz6w8gy9b1mxrqrbw1r5fkhk39qqbgc> Using serde for PyODBC case/other cases [was Serialization in Apache Airflow] <https://lists.apache.org/thread/z4sh6v9bnxhpgggxos8d0rc705tpz1k5> Add BigQuery partition sensor for sequential (or just list of) partition < https://lists.apache.org/thread/x3y79b2jq5g1tgm60wkf89r9zdly315l> MappedArgument Object is not Subscriptable < https://lists.apache.org/thread/g1jf7h04r31pphx3z48r5c6rz7s2mmz0> Do the resources on the Community landing page need reordering/reprioritizing? < https://lists.apache.org/thread/22vnlrc039hk7bmflv7ym665y5os5x5w> New Airflow Community Provider: Teradata < https://lists.apache.org/thread/xj5v9gkmlr3gr20wtozh34l85n5g92fc> "Require conversation resolution" in our PRs before merge? < https://lists.apache.org/thread/hxrp5th37mn6g4q78grc1vrg1lx1srtb> Looking for co-mentor in Major League Hacking Spring 2024 Fellows < https://lists.apache.org/thread/0d0xt9fo9dt1m3rhfn1gm4oh3cfjq5ky> Interested in joining/leaving the triage team? < https://lists.apache.org/thread/37b2r9vjhs1ssrb4g2hm6d009c8jzp5r> Enabling `pre-commit.ci` application for Airflow < https://lists.apache.org/thread/fgoqj23hpl9mlo1n6gfkgvzjxvhfchzm> Airflow vs Temporal < https://lists.apache.org/thread/gq547w4ff5h1r8o0bq7cbfrf13116z7t> Announced: New committer: Utkarsh Sharma < https://lists.apache.org/thread/p4b0jwsxofr2xsrnp81c45hcyp3wfg2t> Apache Airflow Providers prepared on December 08, 2023 are released < https://lists.apache.org/thread/lcooxhk1rhyysg1j9svx5hn9fb03xthw> Apache Airflow Providers prepared on 12 December 2023 are released < https://lists.apache.org/thread/41g5po1y6y7m6f9g7y1wsqdtlbofhw9d> Apache Airflow 2.8.0 Released < https://lists.apache.org/thread/pj6nolktscoltwd8wxn496wgc61opxyh> Airflow Virtual Town Hall- Jan. 11th < https://lists.apache.org/thread/c5vjk6ry9qqkvj60q089oc304vsjm6w5> Save the date for Airflow Summit 2024! < https://lists.apache.org/thread/jddytk5xvqlo9x5185rmtyswtpyfwzvn> Airflow Survey is Closed! < https://lists.apache.org/thread/33nsdboj9oxoyq6jb4ofzblq25v1rhot> Meet our keynote speakers and register to Community Over Code EU! < https://lists.apache.org/thread/zjktxfdtb97j48wypd8xtlm5x5rm5jjv> Apache Airflow Python Client 2.8.0 Released < https://lists.apache.org/thread/2s9glcwgsbn00ggmckdn5ynjkvrhh1jd> Apache Airflow Providers prepared on 23rd December 2023 are released < https://lists.apache.org/thread/b372nwq6q2z2bxjbsjjrry43ghcvc9zo> Starting experimenting with "Require conversation resolution" setting < https://lists.apache.org/thread/bzg00p8yolboqgwlwfnlkdfvnn996601> Apache Airflow Providers prepared on 28th December 2023 are released < https://lists.apache.org/thread/y3xgtwjq0g8qckchf9kotnt1jfs6kco9> Proposed: Security team rotation introduction to our process < https://lists.apache.org/thread/mo52q5cv0shzqdcwzobxg8ffosldpzsg> NEED URGENT FEEDBACK! MySQL Badly signed repo breaks Airflow images < https://lists.apache.org/thread/dy3fgwhoc5m2f3tpqkwqcytrs6nhjlc4> Making Listeners API non-experimental < https://lists.apache.org/thread/cm89z2btvmw292dwlpw42f59zhgm2pqd> IN-DEVELOPMENT AIP-44: Airflow Internal API < https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-44+Airflow+Internal+API > AIP-49: Open Telemetry Support < https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-49+OpenTelemetry+Support+for+Apache+Airflow > AIP-51 Removing Executor Coupling from Core Airflow < https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-51+Removing+Executor+Coupling+from+Core+Airflow > AIP-56: Extensible user management < https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management > AIP-57: Refactor SLA feature < https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-57+Refactor+SLA+Feature > AIP-58 Airflow ObjectStore (AS) < https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263430565> AIP-59: Performance test framework < https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-59+Performance+tests+framework > AIP-60: Standard URI representation < https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-60+Standard+URI+representation+for+Airflow+Datasets > See the Wiki for more of the project’s approved AIPs. < https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals > AIRFLOW FACT OF THE MONTH Did you know that dolphins give names to each other? Research has revealed that the marine mammals use a unique whistle to identify each other. A team from the University of St Andrews in Scotland found that when the animals hear their own call played back to them, they respond. A humble reminder that humans are just another mammal ;). Source: BBC News <https://www.bbc.com/news/science-environment-23410137> STAY TUNED Please subscribe to this newsletter to ensure you always receive the latest issue: <http://eepurl.com/hXUA3r> Know of an upcoming event or publication the community should know about? Let me know at [email protected].
