It sounds like we can get the best of both worlds with the original
proposals to have minimal requirements in setup.py and "guaranteed to work"
complete requirements in a separate file. That way we have flexibility for
teams that run airflow and tasks in the same environment and guidance on a
Anthony:
Could you just have the "success" path be declared with "all_success" (the
default), and the "failure" side branches be declared with "all_failed"
depending on the previous task? This will have the same branching structure
you want but with less intermediary operators.
-James M.
On
Hi all,
The slides are now up:
https://docs.google.com/presentation/d/1gbfR79u4Ws8ctASZiGuQaslX3bHgnW5L9GFuyfQWRFM/edit#slide=id.p
Recordings are on O'Reilly's Safari platform, and you can of course reach
out to me if you have questions.
-James M.
On Thu, May 17, 2018 at 10:49 AM Vasyl
This would be a very desirable feature. It's not just an issue of
differing environments, but also changing requirements. A dag may be
used for a time, and then removed from the environment. But if it's
still in the airflow db, it will get a row on the UI with a black "I'
indicating that the
I like James solution better, but the initial thought I had was to deploy
airflowignore files to the environments to filter out files that should not
be processed when filling the DagBag.
Chris
On Mon, Oct 8, 2018 at 10:22 AM James Meickle
wrote:
> As long as the Airflow process can't find the
Although I think I come down on the side against pinning, my reasons are
different.
For the two (or more) people who have expressed concern about it would pip's
"Constraint Files" help:
https://pip.pypa.io/en/stable/user_guide/#constraints-files
For example, you could add
As long as the Airflow process can't find the DAG as a top-level object in
the module, it won't be registered. For example, we have a function that
returns DAGs; the function returns nothing if it's not in the right
environment.
On Sun, Oct 7, 2018 at 2:31 PM Shah Altaf wrote:
> Hi all,
>
>
Hi,
I plan to work on this again but got busy between work and personal life.
I'll see if I can revisit it this month.
-James M.
On Fri, Oct 5, 2018 at 7:30 AM Colin Nattrass
wrote:
> Hello all,
>
> Is there any update on the status of this PR?
>
> I discovered this following a request for
I hope others also will chime in. it's great to see different points of
view :).
Just one comment - something that Ash mentioned already before - the
problem is not at all limited to CI.
Much bigger issue is that currently released 1.10.0 airflow package has
dependencies that make it impossible
As a member of a team that will also have really big problems if
Airflow pins all requirements (for reasons similar to those already
stated), I would like to add a very strong -1 to the idea of pinning
them for all installations.
In a number of situation on our end, to avoid similar problems with
10 matches
Mail list logo
