Can a '.gitlab-ci.yml' be considered code, in the same way that the k8s related dockerfiles are code? In other words, something like: "here is a piece of code you might choose to use for building your own binaries, that is not specifically endorsed by Apache Spark"? So it would not be involved in the creation of nightly binaries by Apache Spark per se, but could be used by individuals for that purpose.
On Thu, Jan 23, 2020 at 3:52 PM Jim Kleckner <j...@cloudphysics.com> wrote: > I understand that "non-dev" persons could become confused and that some > sort of signposting/warning makes sense. > > Certainly I consider my personal registry on gitlab.com as ephemeral and > not intended to publish. > We have our own private instance of gitlab where I put artifacts that are > derived and this was needed to work with GKE as mentioned since 2.4.4 does > not out of the box work with service accounts the way we use them.. > > I can keep this file as a branch of my own that I manually merge when > needed if others don't find this useful or the risk of confusion is greater > than the value. > > Simply close as not desirable the JIRA at: > https://issues.apache.org/jira/browse/SPARK-30275 > > And now there are discussions both in email and JIRA... > > Jim > > > > > On Thu, Jan 23, 2020 at 11:15 AM Sean Owen <sro...@gmail.com> wrote: > >> Yeah the color on this is that 'snapshot' or 'nightly' builds are not >> quite _discouraged_ by the ASF, but need to be something only devs are >> likely to find and clearly signposted, because they aren't official >> blessed releases. It gets into a gray area if the project is >> 'officially' hosting a way to get snapshot builds. It is not at all >> impossible, just something that's come up and generated some angst in >> the past, so we dropped it. >> >> On Thu, Jan 23, 2020 at 1:09 PM Dongjoon Hyun <dongjoon.h...@gmail.com> >> wrote: >> > >> > Hi, Jim. >> > >> > Thank you for the proposal. I understand the request. >> > However, the following key benefit sounds like unofficial snapshot >> binary releases. >> > >> > > For example, this was used to build a version of spark that included >> SPARK-28938 which has yet to be released and was necessary for >> spark-operator to work properly with GKE service accounts >> > >> > Historically, we removed the existing snapshot binaries in some >> personal repositories and there is no plan to add it back. >> > Also, for snapshot dev jars, we use only the official Apache Maven >> snapshot repository. >> > >> > For official releases, we aim to release Apache Spark source code (and >> its artifacts) according to the pre-defined release cadence in an official >> manner. >> > >> > BTW, SPARK-28938 doesn't mean that we need to publish a docker image. >> Even in the official release, as you know, we only provide a reference >> Dockerfile. That's the reason why we don't publish docker image via GitHub >> Action (as of Today). >> > >> > To achieve the following custom requirement, I'd like to recommend you >> to have your own Dockerfile. >> > That is the best way for you to have the flexibility. >> > >> > > One value of this is the ability to create versions of dependent >> packages such as spark-on-k8s-operator >> > >> > Thanks, >> > Dongjoon. >> > >> > >> > On Thu, Jan 23, 2020 at 9:32 AM Jim Kleckner <j...@cloudphysics.com> >> wrote: >> >> >> >> This story [1] proposes adding a .gitlab-ci.yml file to make it easy >> to create artifacts and images for spark. >> >> >> >> Using this mechanism, people can submit any subsequent version of >> spark for building and image hosting with gitlab.com. >> >> >> >> There is a companion WIP branch [2] with a candidate and example for >> doing this. >> >> The exact steps for building are in the yml file [3]. >> >> The images get published into the namespace of the user as here [4] >> >> >> >> One value of this is the ability to create versions of dependent >> packages such as spark-on-k8s-operator that might use upgraded packages or >> modifications for testing. For example, this was used to build a version >> of spark that included SPARK-28938 which has yet to be released and was >> necessary for spark-operator to work properly with GKE service accounts [5]. >> >> >> >> Comments about desirability? >> >> >> >> [1] https://issues.apache.org/jira/browse/SPARK-30275 >> >> [2] https://gitlab.com/jkleckner/spark/tree/add-gitlab-ci-yml >> >> [3] >> https://gitlab.com/jkleckner/spark/blob/add-gitlab-ci-yml/.gitlab-ci.yml >> >> [4] https://gitlab.com/jkleckner/spark/container_registry >> >> [5] >> https://gitlab.com/jkleckner/spark-on-k8s-operator/container_registry >> >
