Re: [DISCUSS] Utility DAGs

2020-04-14 Thread Jarek Potiuk
> > Personally, I think the best solution is to create a new command line > sub-command responsible for log and/or database cleaning. Users could then > come up with their own mechanism on how to run it (trigger it when disk or > storage percentage reaches high value or simply periodically using

Re: [DISCUSS] Utility DAGs

2020-04-10 Thread Kamil Olszewski
Hello all, I would like to resume discussion on this topic. I am also uncertain about using the DAG as a solution in this case. There are a couple of issues regarding the DAG if you are also using CeleryExecutor: - there is no dynamic way of telling how many nodes are used on given