2. There is a feature 'XCom' which allows you to use the Airflow database
as a key-value store. If you wish DAG and task instances to see particular
data items, you can store them via 'XCom'. I have not done this.
On Fri, Aug 5, 2016 at 7:25 PM, Andrew Phillips wrote:
> I'd like to ask you all
I'd like to ask you all a question in turn: what do you
know now that you wish you knew when you first deployed
Airflow?
Mainly questions around suggestions/best practice for addressing
reasonably common (?) integration challenges:
1. Talking to Airflow from remote/external systems (e.g. on a
Note that on master (but not in the latest release), you can already
specify resource constraints for tasks. They are not consumed anywhere in
airflow itself yet, but you can use them in an operator if it fits your use
case:
https://github.com/apache/incubator-airflow/pull/1669
On Fri, Aug 5, 2016
We're adding cgroups semantics to Airflow's BaseOperator, along with
integration to activate those settings when firing up the task instance as
a subprocess. Paul Yang (@Airbnb) is working on a prototype / design doc
and should be able to share more shortly.
It lines up nicely with the recent work
Linux cgroups gives Docker the ability to control memory use inside a
container.
https://www.cloudsigma.com/manage-docker-resources-with-cgroups/
Here is an example that allocates the CPU:
http://blog.viktorpetersson.com/post/115562026784/using-cgroups-with-docker-on-ubuntu-1404
(I have not work
I think you can define a new operator which take an existing operator base
class and keep monitoring the memory usage (maybe base on airflow
heartbeat?).
On Thu, Aug 4, 2016 at 8:23 PM, Adinata wrote:
> Is there any way for task instances to limit the memory task execution?
>
> I run the worker
I think you can define a new operator which take an existing operator base
class and keep monitoring the memory usage (maybe base on airflow
heartbeat?).
This is the easiest, I think
On Thu, Aug 4, 2016 at 8:23 PM, Adinata wrote:
> Is there any way for task instances to limit the memory task ex
