A crucial point here is considering whether we want to have a separate scheduler backend code path for client mode versus cluster mode. If we need such a separation in the code paths, it would be difficult to make it possible to run spark-submit in client mode from the driver container.
We discussed this already when we started to think about client mode. See https://github.com/apache-spark-on-k8s/spark/pull/456. In our initial designs for a client mode, we considered that there are some concepts that would only apply to cluster mode and not to client mode – see https://github.com/apache-spark-on-k8s/spark/pull/456#issuecomment-343007093. But we haven’t worked out all of the details yet. The situation may work out such that client mode is similar enough to cluster mode that we can consider the cluster mode as being a spark-submit in client mode from a container. I’d imagine this is a reason why YARN hasn’t went with using spark-submit from the application master, because there are separate code paths for a YarnClientSchedulerBackend versus a YarnClusterSchedulerBackend, and the deploy mode serves as the switch between the two implementations. Though I am curious as to why Spark standalone isn’t using spark-submit – the DriverWrapper is manually fetching the user’s jars and putting them on a classloader before invoking the user’s main class with that classloader. But there’s only one scheduler backend for both client and cluster mode for standalone’s case. The main idea here is that we need to understand if we need different code paths for a client mode scheduler backend versus a cluster mode scheduler backend, before we can know if we can use spark-submit in client mode from the driver container. But using init-containers makes it such that we don’t need to use spark-submit at all, meaning that the differences can more or less be ignored at least in this particular context. -Matt Cheah On 1/10/18, 8:40 AM, "Marcelo Vanzin" <van...@cloudera.com> wrote: On a side note, while it's great that you guys have meetings to discuss things related to the project, it's general Apache practice to discuss these things in the mailing list - or at the very list send detailed info about what discussed in these meetings to the mailing list. Not everybody can attend these meetings, and I'm not just talking about people being busy, but there are people who live in different time zones. Now that this code is moving into Spark I'd recommend getting people more involved with the Spark project to move things forward. On Tue, Jan 9, 2018 at 8:23 PM, Anirudh Ramanathan <ramanath...@google.com> wrote: > Marcelo, I can see that we might be misunderstanding what this change > implies for performance and some of the deeper implementation details here. > We have a community meeting tomorrow (at 10am PT), and we'll be sure to > explore this idea in detail, and understand the implications and then get > back to you. > > Thanks for the detailed responses here, and for spending time with the idea. > (Also, you're more than welcome to attend the meeting - there's a link here > if you're around.) > > Cheers, > Anirudh > > > On Jan 9, 2018 8:05 PM, "Marcelo Vanzin" <van...@cloudera.com> wrote: > > One thing I forgot in my previous e-mail is that if a resource is > remote I'm pretty sure (but haven't double checked the code) that > executors will download it directly from the remote server, and not > from the driver. So there, distributed download without an init > container. > > On Tue, Jan 9, 2018 at 7:15 PM, Yinan Li <liyinan...@gmail.com> wrote: >> The init-container is required for use with the resource staging server >> >> (https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache-2Dspark-2Don-2Dk8s_userdocs_blob_master_src_jekyll_running-2Don-2Dkubernetes.md-23resource-2Dstaging-2Dserver&d=DwIFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=hzwIMNQ9E99EMYGuqHI0kXhVbvX3nU3OSDadUnJxjAs&m=rQzoyVLMucfZdPLZAwNlE-PZ90ViBJzTQ49K1dzjr3c&s=HcCtT_KLkPi_05ojHei1nbXUpwoJomou8bitD-WkYmI&e=). > > If the staging server *requires* an init container you have already a > design problem right there. > >> Additionally, the init-container is a Kubernetes >> native way of making sure that the dependencies are localized > > Sorry, but the init container does not do anything by itself. You had > to add a whole bunch of code to execute the existing Spark code in an > init container, when not doing it would have achieved the exact same > goal much more easily, in a way that is consistent with how Spark > already does things. > > Matt: >> the executors wouldn’t receive the jars on their class loader until after >> the executor starts > > I actually consider that a benefit. It means spark-on-k8s application > will behave more like all the other backends, where that is true also > (application jars live in a separate class loader). > >> traditionally meant to prepare the environment for the application that is >> to be run > > You guys are forcing this argument when it all depends on where you > draw the line. Spark can be launched without downloading any of those > dependencies, because Spark will download them for you. Forcing the > "kubernetes way" just means you're writing a lot more code, and > breaking the Spark app initialization into multiple container > invocations, to achieve the same thing. > >> would make the SparkSubmit code inadvertently allow running client mode >> Kubernetes applications as well > > Not necessarily. I have that in my patch; it doesn't allow client mode > unless a property that only the cluster mode submission code sets is > present. If some user wants to hack their way around that, more power > to them; users can also compile their own Spark without the checks if > they want to try out client mode in some way. > > Anirudh: >> Telling users that they must rebuild images ... every time seems less >> than convincing to me. > > Sure, I'm not proposing people use the docker image approach all the > time. It would be a hassle while developing an app, as it is kind of a > hassle today where the code doesn't upload local files to the k8s > cluster. > > But it's perfectly reasonable for people to optimize a production app > by bundling the app into a pre-built docker image to avoid > re-downloading resources every time. Like they'd probably place the > jar + dependencies on HDFS today with YARN, to get the benefits of the > YARN cache. > > -- > Marcelo > > --------------------------------------------------------------------- > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > > -- Marcelo
smime.p7s
Description: S/MIME cryptographic signature
