+1 Thanks for taking on this. That was my feedback on one of the long comment thread as well, I think we should have one docker image instead of 3 (also pending in the fork are python and R variant, we should consider having one that we official release instead of 9, for example)
________________________________ From: 蒋星博 <jiangxb1...@gmail.com> Sent: Friday, January 5, 2018 10:57:53 PM To: Marcelo Vanzin Cc: dev Subject: Re: Kubernetes backend and docker images Agree it should be nice to have this simplification, and users can still create their custom images by copy/modifying the default one. Thanks for bring this out Marcelo! 2018-01-05 17:06 GMT-08:00 Marcelo Vanzin <van...@cloudera.com<mailto:van...@cloudera.com>>: Hey all, especially those working on the k8s stuff. Currently we have 3 docker images that need to be built and provided by the user when starting a Spark app: driver, executor, and init container. When the initial review went by, I asked why do we need 3, and I was told that's because they have different entry points. That never really convinced me, but well, everybody wanted to get things in to get the ball rolling. But I still think that's not the best way to go. I did some pretty simple hacking and got things to work with a single image: https://github.com/vanzin/spark/commit/k8s-img Is there a reason why that approach would not work? You could still create separate images for driver and executor if wanted, but there's no reason I can see why we should need 3 images for the simple case. Note that the code there can be cleaned up still, and I don't love the idea of using env variables to propagate arguments to the container, but that works for now. -- Marcelo --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org<mailto:dev-unsubscr...@spark.apache.org>
