Hi Tao,
We have an internal tool called Optimus that we use for ETL
transformations. Optimus takes in the Bigquery transformation requests and
creates dags and leverages ExternalTaskSensor to manage the dependencies
between dags. We have used KubernetesOperator for this tool, as it has to
run an
Curious to know why you want to run k8spodoperator while you have k8s
executor in prod? Could it solve for a different use case that cannot solve
by k8s executor?
On Thu, Dec 12, 2019 at 11:13 PM Maulik Soneji
wrote:
> Hi Maxime,
>
> We have been using KubernetesExecutor for our ETL use cases.
Friend don't let friends use LocalExecutor in production.
LocalExecutor is essentially a subprocess pool running in-process. When I
wrote it originally I never thought it would ever be used in production.
Celery / CeleryExecutor is more reasonable as Celery is a proper
process/thread pool that's
Hello,
I realize that the scheduler is waiting for the tasks to be completed
before shutting down.
The problem is that the scheduler stops sending heartbeat and just waits
for the task queue to be joined.
Is there a way where we can horizontally scale the number of instances for
scheduler, so
Hello all,
*TLDR*: We are using local executor with KubernetesPodOperator for our
airflow dags.
>From stack trace of scheduler we see that it is waiting on queue to join.
File:
"/usr/local/lib/python3.7/site-packages/airflow/executors/local_executor.py",
line 212, in end
self.queue.join()