This is purely about the YuniKorn application CRD. Not about the Spark operator integration, we do not want to touch that one The Spark link was only to show that a CRD provides limited functionality.
The POC that Frank did was also just the YuniKorn application CRD Wilfred On Sat, 29 Jul 2023 at 05:52, Weiwei Yang <w...@apache.org> wrote: > > Remove YuniKorn application CRD should be fine, I don't think we ever > reached the GA for this feature, and I don't think anyone is using it. > The operator plugin, however, I don't think we should remove, for example > the Spark CRD plugin > <https://github.com/apache/yunikorn-k8shim/blob/master/pkg/appmgmt/sparkoperator/spark.go> > provides > the hook to YK in order to update the Spark job status more consistently. > > Weiwei > > On Fri, Jul 28, 2023 at 12:06 PM Craig Condit <ccon...@apache.org> wrote: > > > I’m definitely in favor of removing the CRD, and sooner rather than later. > > It negatively impacts some of the ongoing refactoring tasks as it > > influences the recovery pipeline. I think given the alpha state of the CRD, > > if there are no objections we can remove this in 1.4.0. > > > > Craig > > > > > > > On Jul 27, 2023, at 9:41 PM, Wilfred Spiegelenburg <wilfr...@apache.org> > > wrote: > > > > > > We added the YuniKorn application CRD via YUNIKORN-170 [1]. The idea > > > was back then that we used the CRD to implement gang scheduling. > > > During the later design of gang scheduling we completely stepped away > > > from using the CRD as the basis for gang scheduling. Some of the other > > > advantages that we were expecting from the CRD were never observed > > > (finished state for an application, one point to manage pods). The > > > Spark CRD integration was mostly reversed [2] in favour of normal pod > > > handling due to issues. > > > The second phase for the application CRD YUNIKORN-599 [3] was never > > > started due to the limited advantages we expected to get. > > > > > > There have been no changes in the code or jiras logged against the CRD > > > for two years besides making the build work on later K8s versions [4] > > > The current usage of the application CRD is limited to the > > > TaskGrooupDefinition being used for gang scheduling. > > > > > > Based on all this I would like to start the discussion on removing the > > > application CRD from YuniKorn. Frank Yang has looked at the changes > > > needed to remove the CRD and the impact on the code for the K8shim. A > > > commit with all the changes can be seen in his repo [5] to reference. > > > The change will remove almost 3,000 lines of code just from the K8shim > > > repository. There will be some further changes needed to clean up the > > > build (K8shim) and helmchart (release). Those changes will be removal > > > of scripts and code only. > > > > > > Before we progress with this further we would like to know: > > > * If the application CRD is used by anyone. > > > * If it is used, what part(s) of the CRD are used and what is it used > > for? > > > * Is removing the CRD in YuniKorn 1.4.0 OK or do we need to push to a > > > later release. > > > > > > Objections, comments please let us know. > > > > > > Thank you, > > > Wilfred > > > > > > [1] https://issues.apache.org/jira/browse/YUNIKORN-170 > > > [2] https://issues.apache.org/jira/browse/YUNIKORN-643 > > > [3] https://issues.apache.org/jira/browse/YUNIKORN-599 > > > [4] > > https://github.com/apache/yunikorn-k8shim/commits/master/pkg/apis/yunikorn.apache.org > > > [5] > > https://github.com/FrankYang0529/yunikorn-k8shim/commit/354248b3c24accd679ff2d5a557c599123a58408 > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org > > > For additional commands, e-mail: dev-h...@yunikorn.apache.org > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org > > For additional commands, e-mail: dev-h...@yunikorn.apache.org > > > > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org For additional commands, e-mail: dev-h...@yunikorn.apache.org
