Any discoveries from this thread should probably be added to
https://airflow.apache.org/concepts.html#core-ideas
FWIW, I'm not sure if it's still this way on the master branch, but at one
point a couple useful features (e.g. was it accessing task logs from the UI
and maybe running from the UI?) we
Hi Brian,
Really appreciate your quick reply. Just to be clear, I did not intend to
run them in particular order. as a matter of fact, these are expensive db
queries that I cant afford to run in parallel.
I think I have setup the tasks correctly to use pool but may be missing the
priority_weight s
I should probably mention that our implementation will be running multiple
executors/schedulers/workers for the sake of resilience and recovery. We
are more concerned about task state correctness than performance if that
makes sense, i.e. the same task should not run twice by accident etc.
On Sat,
Hi
We have been running airflow for several years and decided to create a
executor plugin.
After studying the code quite a bit (both code interacting with the
executor and some of the implementations), I still find it hard to distil
concrete requirements for an BaseExecutor implementation.
Speci
To be clear, you’re hoping that setting the slots to 1 will cause the tasks
across district dags to run in order based on the assumption that they’ll queue
up and then execute off the pool?
I don’t think it will quite work that way - there’s no guarantee the scheduler
will execute your tasks ac
Hi
We are using Airflow to build up a DWH from many source systems but I have a
couple of edge cases that I’m not quite sure how to proceed with.
Each table from a source system is supplied as a CSV. With each set of files
delivered (daily) we receive a manifest/description file containing a l
Hi All,
Greetings. I am writing unit tests for some Elasticsearch related changes and
want to use this library: https://github.com/vrcmarcos/elasticmock for mocking.
Unfortunately it is not working with current version of elasticsearch and
elasticsearch_dsl libs so I had to copy/paste the code a
Hi
Hopefully this makes enough sense to people reading.
We are using Airflow to build up a DWH from many source systems but I have
a couple of edge cases that I’m not quite sure how to proceed with.
Each table from a source system is supplied as a CSV. With each set of
files delivered (daily) w
Hi Airflow devs,
I have a use case to limit the # of calls to a certain database. I am using
the pool along with priority weight to schedule the tasks to the slot pool.
I have around 5 operators that I need to execute in serial order across
different dags.
Slot pool is created with "1" slot to en
Hi All,
Fokko and I are picking up the release management role for the 1.10 release.
The aim of this release is to make sure we are, from the perspective of a
release, graduation ready. So license compliancy is paramount and proper
publishing of security vulnerabilities. It should also be our l
10 matches
Mail list logo
