Django provides a really good model for allowing users to customize the
behavior of Class Based Views. It's in line w/ what Daniel/Kaxil and co are
saying about a consistent backend class. It uses a standard base class as
well as a default concrete implementation. Customization then only requires
s
I like (2) but without adding a new 'apache-airflow-providers-prestodb' for
Facebook's prestodb -- especially since I don't currently know if the
python client library will continue to work with it unless someone tests
and adds it.
On Wed, Jan 20, 2021 at 11:43 PM Philippe Gagnon
wrote:
> Hell
Kaxil you beat me to it. I actually have a dag where I achieve an irregular
schedule by overriding DAG.next_dagrun_info(). If that method were swapped out
for an object it may be a semi-easy way to make the schedule “plugable”.
James Coder
> On Jan 20, 2021, at 6:37 PM, Kaxil Naik wrote:
>
>
Hello Airflow-dev,
About a couple of years ago the Presto community was fractured when the
three founders of the project, Dain, David and Martin, left Facebook due to
irreconcilable differences with the company concerning the stewardship of
the open-source project. In order to continue working on
"CronBackend" / "ScheduleIntervalBackend" :D similar to Xcom and Secrets
Backend
Would be definitely good to have Custom Schedule intervals using
functions/class that is Serializable too.
On Wed, Jan 20, 2021 at 11:02 PM QP Hou wrote:
> On Wed, Jan 20, 2021 at 10:22 AM Daniel Imberman
> wrot
On Wed, Jan 20, 2021 at 10:22 AM Daniel Imberman
wrote:
>
> I love the idea of allowing users to create their own scheduling
> objects/scheduling python functions. They could either live in the scheduler
> or as a seperate process that trips some value in the DB when it is “true”.
> Would be gr
This all just to say: I think this one specific use case is already achievable.
On 20 January 2021 19:36:54 GMT, Ash Berlin-Taylor wrote:
>I'll have to test, bit my gut says that isn't how it works, instead next
>schedule = previous schedule + interval.
>
>On 20 January 2021 19:25:50 GMT, Elad
I'll have to test, bit my gut says that isn't how it works, instead next
schedule = previous schedule + interval.
On 20 January 2021 19:25:50 GMT, Elad Kalif wrote:
>Sorry ignore my explanation. It's wrong.
>
> https://github.com/apache/airflow/issues/8649 explains why
>`schedule_interval=timed
Sorry ignore my explanation. It's wrong.
https://github.com/apache/airflow/issues/8649 explains why
`schedule_interval=timedelta(days=14), start_date=...`. is not the exactly
the same as bi-weekly for an exact specified time.
It also provide an example for daily.
On Wed, Jan 20, 2021 at 9:19 PM
> `schedule_interval=timedelta(days=14), start_date=...`.
ash from what I've seen It doesn't give the same exact functionality
This is what I observed:
cron = schedule exactly on regardless of what happened with previous run
(mostly)
timedelta() = schedule after delta passed. However it's from end
> `schedule_interval=timedelta(days=14), start_date=...`.
That won't support every second Thursday for example
On Wed, Jan 20, 2021 at 6:54 PM Ash Berlin-Taylor wrote:
> Running exactly every two weeks can be done by setting
> `schedule_interval=timedelta(days=14), start_date=...`.
>
> Does th
Running exactly every two weeks can be done by setting
`schedule_interval=timedelta(days=14), start_date=...`.
Does this do what you need Elad?
On 20 January 2021 18:12:36 GMT, Elad Kalif wrote:
>>> In the example of a twice-a-month dag (not sure if it you have this use
>case too?) what do you
Re: Providers.
At the call I'd love to get the feedback on the way the new automated parts
of "providers 2.0" page. Example for google provider here:
http://gabby-cough.surge.sh/
Some discussion started by Kamil here:
https://github.com/apache/airflow/pull/13767#discussion_r560629925 and it
touch
With respect to the Providers release, I would like to discuss the notion
of a standard every 2 or 3 week (whatever the frequency) release of
Providers.
Jarek suggested this in the Issue Triage call last week and I had
incorporated that into the notes, but wanted to make sure to bring this up
agai
On Wed, Jan 20, 2021 at 7:28 PM Daniel Imberman
wrote:
> @Jarek the problem with just cron is I don’t think cron can handle “every
> third Thursday” or “the next open market day after the 15th.” I think we
> need something more flexible than just cron (though agree that cron syntax
> can get a fa
@Jarek the problem with just cron is I don’t think cron can handle “every third
Thursday” or “the next open market day after the 15th.” I think we need
something more flexible than just cron (though agree that cron syntax can get a
fair bit of mileage)
On Wed, Jan 20, 2021 at 10:24 AM, Jarek Po
My thoughts (no final solution in mind, just wild thoughts):
1) I think we should add support for regular CRON behaviour. Simply "cron"
schedule for dag execution, without the "data interval" rhetoric.
There are a number of good cases where Airflow can be used as just a
scheduler to run the jobs.
I love the idea of allowing users to create their own scheduling
objects/scheduling python functions. They could either live in the scheduler or
as a seperate process that trips some value in the DB when it is “true”. Would
be great from a “marketplace” standpoint as well as users could post the
>> In the example of a twice-a-month dag (not sure if it you have this use
case too?) what do you expect the "data interval" (i.e. execution_date) to
be?
Yes we have this use case too. The execution date does matter because I
want it to be bi-weekly for starting specific day and time
so with the cu
A quick thought (*maybe not making sense*): if *schedule_interval* accepts
a list of values, we may support much higher complexity.
For example, I may want to schedule my jobs at every days' 04:05 AND 02:31
, which cannot be expressed by single Cron pattern. Then I may want to
have *schedule_inter
Yes, we quite possibly could do this -- I'm trying to work out what the
needs are here.
In the example of a twice-a-month dag (not sure if it you have this use
case too?) what do you expect the "data interval" (i.e. execution_date)
to be?
Or for this case does it not matter?
-ash
On Wed,
Another case that is mentioned in one of the issues is the ability to
schedule a bi-weekly job (equivalent of bi-weekly meeting that you can set
in a calendar) which is very much needed.
Maybe this is unrealistic but I think the game changer is if it would be
possible to let the users define their
Here is again a use case for dynamic scheduling:
from the origin link: https://github.com/apache/airflow/issues/10449
Description
There are several question (issues) on stack overflow, asking for the
need of a dynamic schedule interval. This means, the ability to change
the schedule interval af
Hi everyone,
I'd like to (re)start the discussion about a new feature I'd like to
add for Airflow 2.1, that I am loosely calling "improving
schedule_interval" (catchy name I know!)
I have two main high-level goals in mind here:
1. To reduce the confusion around execution_date (specifically t
Hi all,
I would like to remind whoever is interested to join the Airflow Community
Dev call that we will have our fortnightly dev call later today.
*Airflow Dev Call Calendar Link*:
https://calendar.google.com/calendar/embed?src=c_dhdh3bdjc42c4ngtnpg7ovcs9g%40group.calendar.google.com&ctz=Europe%
dependabot[bot] closed pull request #9:
URL:
https://github.com/apache/airflow-JamesIves-github-pages-deploy-action/pull/9
This is an automated message from the Apache Git Service.
To respond to the message, please log on t
dependabot[bot] opened a new pull request #11:
URL:
https://github.com/apache/airflow-JamesIves-github-pages-deploy-action/pull/11
Bumps
[@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node)
from 14.14.14 to 14.14.22.
Commits
See full diff in
dependabot[bot] commented on pull request #9:
URL:
https://github.com/apache/airflow-JamesIves-github-pages-deploy-action/pull/9#issuecomment-763492770
Superseded by #11.
This is an automated message from the Apache Git Serv
28 matches
Mail list logo
