Curious to know why you want to run k8spodoperator while you have k8s
executor in prod? Could it solve for a different use case that cannot solve
by k8s executor?
On Thu, Dec 12, 2019 at 11:13 PM Maulik Soneji
wrote:
> Hi Maxime,
>
> We have been using KubernetesExecutor for our ETL use cases.
Friend don't let friends use LocalExecutor in production.
LocalExecutor is essentially a subprocess pool running in-process. When I
wrote it originally I never thought it would ever be used in production.
Celery / CeleryExecutor is more reasonable as Celery is a proper
process/thread pool that's
Hello,
I realize that the scheduler is waiting for the tasks to be completed
before shutting down.
The problem is that the scheduler stops sending heartbeat and just waits
for the task queue to be joined.
Is there a way where we can horizontally scale the number of instances for
scheduler, so
Hello all,
*TLDR*: We are using local executor with KubernetesPodOperator for our
airflow dags.
>From stack trace of scheduler we see that it is waiting on queue to join.
File:
"/usr/local/lib/python3.7/site-packages/airflow/executors/local_executor.py",
line 212, in end
self.queue.join()
Big yes to asserts in pytest!
On Tue, Dec 10, 2019 at 12:50 AM Michał Słowikowski <
michal.slowikow...@polidea.com> wrote:
> I like this idea, and such assertion looks very promising.
>
> assert "maximum recursion" in str(excinfo.value)
>
> Have a nice day!
>
>
> On Mon, Dec 9, 2019 at 6:16 PM
