Will it still be possible to import and use configuration directly? For
example:
```
from airflow.configuration import conf
from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import
KubernetesPodOperator
namespace = conf.get("kubernetes", "NAMESPACE")
KubernetesPodOperator(
nam
In my experience, when you ask those with Airflow experience what a dag is,
they'll start talking about workflow attributes - stuff like dags being a
series of steps or tasks with owners. The structure doesn't come up.
Echo-ing others, at this point, my vote is to embrace the name and
de-emphasize
That was a lot to read through, and to be honest, it's hard for me to tell
whether or not Jarek's proposal solves David's problem. However, if the
debate is whether it's worthwhile or not to provide a first-class way for
DAG authors to use Operators as part of TaskFlow Tasks, it is.
Operators are
The main reason for pools is to control task execution parallelism,
especially when tasks interact with systems like APIs or databases, to
avoid overwhelming them. For deferrable operators, if a trigger is just
'sleeping' or waiting, it’s fine to exclude the task from the pool while in
a deferred s
I'm partial to everything that we expect users to use to be importable from
`airflow`, but would love to hear other people's thoughts.
On Fri, Aug 30, 2024 at 5:48 AM Ash Berlin-Taylor wrote:
> Hi everyone,
>
> It’s time to have a another discussion about everyone's favourite
> discussion - nami
Seems valid for default behaviour, but if I backfill for a year and realize
there was something wrong with the code, I don't want to manually fail each
dag run that is running. How about a force kill option?
On Wed, Jul 10, 2024 at 9:28 AM Daniel Standish
wrote:
> Yup that's true @Tzu-ping Chung
I love it and 100% agree. Thinking "Dag Groups", where you can group dags
(static & dynamic) into a subfolder. Tags are great for filtering, but they
aren't a replacement for dirs especially at a large scale. We have some
deployments with 20k dags and as designed today, it's not navigable at that
s
I love the idea. If we were to store it in the DB, would we keep a history,
or only the latest stats from the most recent dag parsing loop? DAG parsing
by default is every 30s right?
On Fri, Jun 14, 2024 at 6:53 AM Jarek Potiuk wrote:
> > I think we still need to enable the ability for DAGs at p
nt. We will follow up with a set of formal AIPs.
Constance
--
Constance Martineau
Senior Product Manager
Email: consta...@astronomer.io
Time zone: US Eastern (EST UTC-5 / EDT UTC-4)
Hello again,
Given all the enthusiasm - assuming Nielsen is ok with this - what if
someone recorded the meeting so that it could be shared with those that are
interested?
Constance
On Mon, Jun 10, 2024 at 1:48 PM Constance Martineau
wrote:
> Hi Jarek,
>
> Same :)
>
> Than
Hi Jarek,
Same :)
Thanks,
Constance
On Mon, Jun 10, 2024 at 9:57 AM Amogh Desai
wrote:
> Hello Jarek,
>
> Please add me to the invite as well.
>
> Thanks & Regards,
> Amogh Desai
>
>
> On Mon, Jun 10, 2024 at 11:22 AM Abhishek Bhakat
> wrote:
>
> > Hi Jarek,
> >
> > I would also like to join
While I know that #39336 was a lot of work and big from a dev perspective
(thanks @Daniel Standish !), my vote goes to
#39650 as task-level CPU and memory metrics are a long-standing feature
request.
On Tue, May 28, 2024 at 1:42 PM Jarek Potiuk wrote:
> #39336 hands down
>
> On Tue, May 28, 2024
Agreed. When Jed and team wrote the AIP, we intentionally limited the
scope to DAGs since the AIPs were already really large, but the intention
is to extend the concept to datasets.
Funny that you bring up point #2. A few of us met last week to talk about
DAG Versioning, and that use-case came up
+1 non-binding
On Thu, May 9, 2024 at 7:17 AM Tomasz Urbaszek wrote:
> +1 binding
>
> On Thu, 9 May 2024 at 12:40, Andrey Anshin
> wrote:
>
> > +1 binding
> >
> >
> >
> >
> >
> > On Thu, 9 May 2024 at 13:25, Wei Lee wrote:
> >
> > > Got it. Thanks Jarek for pointing out!
> > >
> > > Best,
> >
nks to the cuts). But I also here
> would mostly revert to Astronomer, Google. AWS team to define collectively
> what is the absolute minimum set of features that would get the "target"
> part of their customers happy. And ONLY do that.
>
> So in short - I think the big part of
Hi Michal,
Thanks for your thoughts on the Airflow 3 proposal. I appreciate your
concerns about the migration overhead for our users with a major new
version and see the appeal in your suggestion to integrate many of the
proposed changes into Airflow 2 through separate AIPs. It’s a valid point
and
> Maybe we restrict who can post in development for a
period of time with a message directing folks to the right places?
As long as we don't make it committer only. If you're contributing
something and want some help/feedback, it's not welcoming to find out that
you're to be restricted from the de
+1 for #contributing and leaving #troubleshooting. Shorter names in slack
are nice where possible.
No strong opinion on the actual names. Agree that #development needs to be
renamed to something more obvious though.
On Thu, Feb 8, 2024 at 9:30 AM Vincent Beck wrote:
> I am +1 in renaming these
cover here IMHO.
> Did I miss anything? Please let me know.
>
>
> Thanks again! Looking forward to more questions/comments!
>
>
> XD
>
>
> > On Feb 2, 2024, at 13:29, Constance Martineau
> >
> wrote:
> >
> > Naive question: Instead of running the co
Naive question: Instead of running the code on the scheduler - could the
condition check be delegated to the triggerer?
On Fri, Feb 2, 2024 at 2:33 PM Pierre Jeambrun
wrote:
> But maybe it’s time to reconsider that :), curious to see what others
> think.
>
> On Fri 2 Feb 2024 at 20:30, Pierre Je
> > I'm not saying it's impossible for these interfaces to coexist if you
> > isolate them from one another, especially when multiple dag-processors
> > already do something similar for dags even now (isolating sets of objects
> > from one another using proces
t; we can ensure folks can anonymously submit a post as "breaking the
> guidelines" if they cannot DM an individual.
>
> On Fri, Jan 26, 2024 at 10:16 AM Constance Martineau
> wrote:
>
> > Wow Briana! This is fantastic, what a great idea! I added a few comments.
> &
Wow Briana! This is fantastic, what a great idea! I added a few comments.
I also had a similar question as Jarek that I think merits a discussion:
Should we have a committee or group to handle reported guideline
violations? If we single out one person to report violations to, we'll have
to continu
ating Airflow) my actual
> experience with Datasets is limited, I've been mainly observing what was
> going on, so I would love to hear what those who created (and continue to
> think about future of) the datasets :).
>
> J,
>
> On Wed, Jan 24, 2024 at 7:27 PM Constance Ma
it to be a task output as part of a
dag. The only valid reason to now allow it IMHO is because they were
designed to be defined within a dag file, similar to a dag, and we don't
want to deal with the impediment I laid out.
On Wed, Jan 24, 2024 at 12:45 PM Jarek Potiuk wrote:
> On Wed, Jan
t; >> > > >> these tenants, we currently have a complex setup:
> >> > > >>
> >> > > >>1. Containers run on a schedule to export metadata to CosmosDB
> >> > (these
> >> > > >>will be replaced by the listener).
> >> > > >>2. Additional scheduled containers pull data from CosmosDB and
> >> > write
> >> > > >>it to a shared file system, enabling generated DAGS to read it
> >> and
> >> > > mirror a
> >> > > >>dataset across tenants.
> >> > > >>
> >> > > >>
> >> > > >> Proposed Workflow
> >> > > >> Here's a breakdown of our proposed workflow:
> >> > > >>
> >> > > >>1. Cross-Tenant Dataset Interaction: We have Dags in Tenant 1
> >> > > >>producing Dataset A. We need a mechanism to trigger all Dags
> >> > > consuming
> >> > > >>Dataset A in Tenant 2. This interaction is crucial for our
> data
> >> > > pipeline's
> >> > > >>efficiency and synchronicity.
> >> > > >>2. Dataset Listener Implementation: Our approach involves
> >> > > >>implementing a Dataset listener that programmatically creates
> >> > > Dataset A in
> >> > > >>all tenants where it's not present (like Tenant 2) and export
> >> > Dataset
> >> > > >>updates when they happen. This would trigger an update on all
> >> Dags
> >> > > >>consuming from that Dataset.
> >> > > >>3. Standardized Dataset Names: We plan to use standardized
> >> dataset
> >> > > >>names, which makes sense since a URI is its identifier and
> >> > > uniqueness is a
> >> > > >>logical requirement.
> >> > > >>
> >> > > >> [image: image.png]
> >> > > >>
> >> > > >> Why This Matters:
> >> > > >>
> >> > > >>- It offers a streamlined, automated way to manage datasets
> >> across
> >> > > >>different Airflow instances.
> >> > > >>- It aligns with a need for efficient, interconnected
> workflows
> >> in
> >> > a
> >> > > >>multi-tenant environment.
> >> > > >>
> >> > > >>
> >> > > >> I invite the community to discuss:
> >> > > >>
> >> > > >>- Are there alternative methods within Airflow's current
> >> framework
> >> > > >>that could achieve similar goals?
> >> > > >>- Any insights or experiences that could inform our approach?
> >> > > >>
> >> > > >> Your feedback and suggestions are invaluable, and I look forward
> >> to a
> >> > > >> collaborative discussion.
> >> > > >>
> >> > > >> Best Regards,
> >> > > >> Eduardo Nicastro
> >> > > >>
> >> > > >
> >> > >
> >> >
> >>
> >
>
--
Constance Martineau
Senior Product Manager
Email: consta...@astronomer.io
Time zone: US Eastern (EST UTC-5 / EDT UTC-4)
<https://www.astronomer.io/>
or a future issue of the newsletter, please drop me a line at <
> briana.oky...@astronomer.io>.
>
> --
> Briana Okyere
> Community Manager
> *Astronomer*
>
--
Constance Martineau
Senior Product Manager
Email: consta...@astronomer.io
Time zone: US Eastern (EST UTC-5 / EDT UTC-4)
<https://www.astronomer.io/>
tests/providers/teradata
> > > >
> > > > System Tests:
> > > >
> > >
> https://github.com/Teradata/airflow/tree/td_develop/tests/system/providers/teradata
> > > > System Tests Dashboard: https://teradata.github.io/airflow/
> > > >
> > >
> >
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
> For additional commands, e-mail: dev-h...@airflow.apache.org
>
>
--
Constance Martineau
Senior Product Manager
Email: consta...@astronomer.io
Time zone: US Eastern (EST UTC-5 / EDT UTC-4)
<https://www.astronomer.io/>
Tue, Oct 31, 2023 at 6:00 PM Jed Cunningham
> wrote:
>
> > The new OpenSearch provider gets my vote - 34705.
> >
>
--
Constance Martineau
Senior Product Manager
Email: consta...@astronomer.io
Time zone: US Eastern (EST UTC-5 / EDT UTC-4)
<https://www.astronomer.io/>
at 9 am PST. The winner(s) will be
> featured in the next issue of the Airflow newsletter.
>
> Also, if there’s an article or event that you think should be included in
> this or a future issue, please drop me a line at <
> briana.oky...@astronomer.io>.
>
> --
> Briana Ok
gt; Geschäftsführung: Dr. Stefan Hartung, Dr. Christian Fischer, Dr. Markus
> Forschner, Stefan Grosch, Dr. Markus Heyn, Dr. Tanja Rückert
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
> For additional commands, e-mail: dev-h...@airflow.apache.org
>
--
Constance Martineau
Senior Product Manager
Email: consta...@astronomer.io
Time zone: US Eastern (EST UTC-5 / EDT UTC-4)
<https://www.astronomer.io/>
gistergericht: Amtsgericht Stuttgart, HRB 14000;
> Aufsichtsratsvorsitzender: Prof. Dr. Stefan Asenkerschbaumer;
> Geschäftsführung: Dr. Stefan Hartung,
> Dr. Christian Fischer, Filiz Albrecht, Dr. Markus Forschner, Dr. Markus
> Heyn, Rolf Najork
>
>
--
Constance Martineau
Product Manager
Email: consta...@astronomer.io
Time zone: US Eastern (EST UTC-5 / EDT UTC-4)
<https://www.astronomer.io/>
ttps://github.com/apache/airflow/pull/27540
>> >>
>> >> [27597] Add max_wait for exponential_backoff in BaseSensor
>> >> https://github.com/apache/airflow/pull/27597
>> >>
>> >> [27506] Fix mini scheduler expansion of mapped task
>> >> https://github.com/apache/airflow/pull/27506
>> >>
>> >> [27526] Clean backcompat code kpo
>> >> https://github.com/apache/airflow/pull/27526
>> >>
>> >> Regards,
>> >> John
>> >
>> >
>>
>
--
Constance Martineau
Product Manager
Email: consta...@astronomer.io
Time zone: US Eastern (EST UTC-5 / EDT UTC-4)
<https://www.astronomer.io/>
, Daniel Standish <
> daniel.stand...@astronomer.io.INVALID> wrote:
>
> One vote for https://github.com/apache/airflow/pull/26400 (improved test
> command)
>
> On Tue, Sep 27, 2022 at 10:50 AM Jed Cunningham
> wrote:
>
>> My write-in is ExternalPythonOperator:
r the local Quick Start in docs
>>>> https://github.com/apache/airflow/pull/25888
>>>>
>>>> [25788] Properly check the existence of missing mapped TIs
>>>> https://github.com/apache/airflow/pull/25788
>>>>
>>>> [25610] Grid logs for mapped instances
>>>> https://github.com/apache/airflow/pull/25610
>>>>
>>>> [25857] Add `RedshiftCreateClusterSnapshotOperator`
>>>> https://github.com/apache/airflow/pull/25857
>>>>
>>>> --
>
> Collin McNulty
> Lead Airflow Engineer
>
> Email: col...@astronomer.io
> Time zone: US Central (CST UTC-6 / CDT UTC-5)
>
>
> <https://www.astronomer.io/>
>
--
Constance Martineau
Product Manager
Email: consta...@astronomer.io
Time zone: US Eastern (EST UTC-5 / EDT UTC-4)
<https://www.astronomer.io/>
n modules where strong authentication such as
>>> Kerberos can be used.
>>> 2. Execute DAG code as the API identity, i.e. A DAG created through the
>>> API service will have run_as_user set to be the API identity.
>>> 3. To enforce data access control on DAGs, the API identity should also
>>> be used to access the data warehouse.
>>>
>>> We shared a demo based on a prototype implementation in the summit and
>>> some details are described in this ppt
>>> <https://drive.google.com/file/d/1luDGvWRA-hwn2NjPoobis2SL4_UNYfcM/view>,
>>> and would love to get feedback and comments from the community about this
>>> initiative.
>>>
>>> thanks
>>> Mocheng
>>>
>>
--
Constance Martineau
Product Manager
Email: consta...@astronomer.io
Time zone: US Eastern (EST UTC-5 / EDT UTC-4)
<https://www.astronomer.io/>
Am intrigued. Curious about dynamic dag pattern, where you create the DAG
object in a a create_dag function and adding the DAG to globals. Would this new
way prevent someone from modifying the dag object within the function, or
returning it?
> On Apr 27, 2022, at 11:20 AM, Ferruzzi, Dennis
>
the next day. The oddness is amplified when you consider a monthly
>> dag, where if you deploy now, start date is now, first schedulable run is
>> next month, therefore first run _more_ than a month away. To fix this I
>> think we need to add support in our timetables for running
2022 at 3:00 PM Philippe Lanoe
>>>> wrote:
>>>> >>>>
>>>> >>>> Hello Daniel,
>>>> >>>>
>>>> >>>> Thank you for your answer. In your example, as I experienced, the
>>>> first run would n
Hi Neeku,
I used to work in an environment that heavily relied on MSSQL Server and
SSIS. Among other things, we used Airflow to orchestrate the SSIS jobs when
moving to Airflow. While there is no specific "SSIS" package, assuming you
are using odbc drivers, there is an odbc provider (
https://airf
+1 (non-binding)
On Wed, Mar 3, 2021 at 10:31 AM Ryan Hamilton
wrote:
> Team,
>
> This email calls for a vote on the project proposed in AIP-38:
>
>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-38+Modern+Web+Application
>
> Discussion thread:
>
>
> https://lists.apache.org/thread.ht
41 matches
Mail list logo
