@Kyle didn't see your middle message there:
You could certainly have k8s scale a Dask Cluster (I think k8s can
autoscale based on CPU and memory usage). In that case, yeah I'd say making
a DaskOperator would probably be the most straightforward way to go. You
can use almost every operator in the k
@Kylen so what I'm trying to understand is why you would want to run a
static DASK cluster when you can launch Dask containers/pods using the
executor?
Seems like there are a few possible options:
1. add the Dask pip modules to the airflow docker image and call on that
image in the executor_conf
Hi Fokko,
So its always been my intention to use the KubernetesExecutor. What I'm
trying to figure out is how to pair the KubernetesExecutor with a
Dask cluster, since Dask clusters have many optimizations for ML type tasks.
On Sat, Apr 28, 2018 at 2:29 PM Driesprong, Fokko
wrote:
> Also one of
Also one of the main benefits of the Kubernetes Executor is having a Docker
image that contains all the dependencies that you need for your job.
Personally I would switch to Kubernetes when it leaves the experimental
stage.
Cheers, Fokko
2018-04-28 16:27 GMT+02:00 Kyle Hamlin :
> I don't have a
I don't have a Dask cluster yet, but I'm interested in taking advantage of
it for ML tasks. My use case would be bursting a lot of ML jobs into a
Dask cluster all at once.
>From what I understand, Dask clusters utilize caching to help speed up jobs
so I don't know if it makes sense to launch a Dask
Hi Kyle,
So you have a static Dask cluster running your k8s cluster? Is there any
reason you wouldn't just launch the Dask cluster for the job you're running
and then tear it down? I feel like with k8s the elasticity is one of the
main benefits.
On Fri, Apr 27, 2018 at 12:32 PM Kyle Hamlin wrote
