Re: [VOTE] AIP-45 Remove double dag parsing in airflow run

+1(non-binding) On Mon, Mar 7, 2022 at 12:29 PM Jarek Potiuk wrote: > +! (binding) > > On Mon, Mar 7, 2022 at 8:47 PM Kevin Yang wrote: > >> +1 (binding) >> >> On Mon, Mar 7, 2022 at 9:39 AM Ping Zhang wrote: >> >>> Hi friends, >>> >>> I would like to call for a vote on AIP-45 Remove double da

Re: [VOTE] AIP-17: Consolidate and de-duplicate sensor tasks in airflow Smart Sensor

> https://airflow.readthedocs.io/en/latest/dag-serialization.html) to get > > the > > "task level fields" without re-parsing the DAGs or storing it in the new > > table. > > > > And can we use some of the operators like the BigQueryOperator, > > Spa

Re: [VOTE] AIP-17: Consolidate and de-duplicate sensor tasks in airflow Smart Sensor

ration? > > Also, a couple of clarifying questions: > 1. Do you know if this is more suitable to certain kinds of sensors vs. > others? > 2. What do you think about leveraging this to enable "async" operations > using Airflow i.e. submit a task and then use a "

[VOTE] AIP-17: Consolidate and de-duplicate sensor tasks in airflow Smart Sensor

Hello everyone! This email calls for a vote to add the airflow smart sensor at https://github.com/apache/airflow/pull/5499 AIP: https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17%3A+Consolidate+and+de-duplicate+sensor+tasks+in+airflow+Smart+Sensor Change summary: - Add a new mode cal

[DISCUSS] AIP-17 Airflow smart sensor for consolidating sensor tasks

Hi everyone, There is a early proposal for optimizing airflow sensor tasks in AIP-17 https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17+Airflow+sensor+optimization The smart sensor mentioned in this document was deployed internally in our company and have largely reduced sensor workers. A

Re: [Discussion] In Prep for AIP: Stateful XComs and Poke Rescheduling in Operators

The updated AIP with smart sensor design and some implementation is in https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17+Airflow+sensor+optimization Open source PR https://github.com/apache/airflow/pull/5499 On Fri, Jan 10, 2020 at 1:44 PM Alex Guziel wrote: > I feel like for this, we

Re: [ANNOUNCE] Please welcome new Airflow committer Kevin Yang

Congrats Kevin! On Tue, Apr 30, 2019 at 6:43 PM Deng Xiaodong wrote: > Congrats Kevin! > > > XD > > On Wed, May 1, 2019 at 1:09 AM Daniel Imberman < > dimberman.opensou...@gmail.com> wrote: > > > Congrats Kevin! > > > > On Tue, Apr 30, 2019 at 9:09 AM Aizhamal Nurmamat kyzy > > wrote: > > > > >

Re: [Discuss] Airflow sensor optimization

gt; >> does not reduce the number of worker processes for sensor. The batch > sensor > >> idea is proposed for this purpose and should work well with reschedule > >> mode. > >> > >> On Wed, Mar 6, 2019 at 11:30 AM Yingbo Wang wrote: > >> > &

Re: [Discuss] Airflow sensor optimization

. On Wed, Mar 6, 2019 at 11:30 AM Yingbo Wang wrote: > Wow, Great work from Seelmann! Thanks Fokko for letting us know it. We are > super happy to have this feature. > > On Wed, Mar 6, 2019 at 11:24 AM Driesprong, Fokko > wrote: > >> Thanks for bringing this up. I'v

Re: [Discuss] Airflow sensor optimization

l the time. The details are in the PR > https://github.com/apache/airflow/pull/3596 > > I would propose to make this the default behavior in Airflow 2.0. > > Cheers, Fokko > > Op wo 6 mrt. 2019 om 15:32 schreef Yingbo Wang : > > > hi, > > > > I would

[Discuss] Airflow sensor optimization

hi, I would like to open an AIP for Airflow sensor optimization. *Motivation*: Low efficiency in Airflow Sensor Implementation Sensors are a special kind of operator that will keep running until a certain criterion is met. Examples include a specific file landing in HDFS or S3, a partition app

AIP access

Hi, My username on confluence is ybwang. I'd like to get access to post a new AIP. Could anyone help to give me the access right? Thanks. Yingbo