Hi Aravind, Thanks for your detailed response and insight into the CLI implementation for Celeborn. I appreciate the explanation regarding the choice to use Java/Scala for the CLI. I agree that aligning with the prevalent use of Java/Scala within the Celeborn community is the right approach.
Your proposed use cases for the CLI functionalities are comprehensive and interesting. I appreciate your proactive approach in considering various cluster management systems and the need for an abstraction layer to interface uniformly. +1 for the current proposal. Looking forward to the CLI evolution. regards, Ethan Aravind Patnam <akpatna...@gmail.com> 于2024年6月13日周四 05:35写道： > > Hi Ethan, > > Thanks for your comments! > > Regarding using Java/Scala for the CLI, I am fine with this. I had believed > that using Python would be an easier/simpler implementation given that many > CLI's are implemented in Python, but the points you make are fair. Most of > the Celeborn community uses Java/Scala, so this would be more beneficial > for the development and evolution of the CLI. > > Yes, I think the CLI should contain capabilities beyond the HTTP endpoints > Celeborn exposes. The Celeborn HTTP endpoints work great for application > specific use cases, such as finding the applications or shuffles on a > particular worker, however it would not work for situations in which we > would need information on the cluster itself. For example, we use K8s and > these are use cases internally I can foresee that require communication > with an external cluster manager: > > - Retrieve all pods running masters/workers and their statuses > - Manually evict an Celeborn unhealthy pod > - SSH into various different Celeborn pods > - Manage ACLs of the cluster > - Manually restart pods > - Wipe Ratis storage if state is messed up > - Wipe shuffle directories if state is messed up > - Adding/removing new nodes into our node pool > - Perform any other manual arbitrary function on a Celeborn pod > > > These are just a few of the use cases I can think of, but I am sure more > will arise as more users adopt Celeborn :) > > Given that users will have various different cluster managers, I think as I > mentioned before there should be an abstraction layer present that exposes > different operations. Based on the cluster manager the user is using, the > user can implement their specific logic. We can have a few default ones > included (e.g. Kubernetes). > > Hope this answers your questions, let me know if you have any more > questions! > > Thanks, > Aravind > > On Tue, Jun 11, 2024 at 11:57 PM Ethan Feng <ethanf...@apache.org> wrote: > > > Hi Aravind, > > > > I hope this message finds you well. I wanted to express my > > appreciation for the energy and creativity you've invested in the > > Celeborn project; the proposal you submitted is intriguing. > > > > I apologize for the delayed feedback on your proposal — it took me a > > bit longer to get to it than anticipated. After reviewing it, I have a > > couple of inquiries that I'd like to discuss in order to gain a > > clearer understanding: > > > > I observed that you're planning to implement the CLI in Python. Could > > you elaborate on the choice behind not leveraging the Java stack for > > this purpose? The Java ecosystem already includes mature tools such as > > "commons-cli" or "Scala CLI," which are capable of facilitating CLI > > tool development. Given the prevalent familiarity with the Java stack > > within our community, I believe leveraging it could accelerate the > > CLI's development and evolution through wider collaboration. > > > > From email discussions, you've indicated an interest in offering a > > generic interface API for Celeborn, which is certainly exciting. > > However, I'm concerned that basing a CLI on HTTP API might not fully > > align with this vision. Could you provide additional insights into how > > you envision the CLI advancing beyond the capabilities of the current > > HTTP REST API? > > > > Based on previous exchanges, the CLI is expected to communicate with > > an external cluster manager. Is there an abstraction layer in place to > > interface uniformly with various external cluster managers, or is this > > something under consideration? > > > > I'm looking forward to learning more about your perspectives and the > > pathway you foresee for the CLI's development. > > > > regards, > > Ethan > > > > Mridul Muralidharan <mri...@gmail.com> 于2024年6月12日周三 14:36写道： > > > > > > +1 > > > > > > Regards, > > > Mridul > > > > > > > > > On Wed, Jun 12, 2024 at 1:08 AM Shaoyun Chen <c...@apache.org> wrote: > > > > > > > +1 > > > > > > > > Keyong Zhou <zho...@apache.org> 于2024年6月12日周三 13:47写道： > > > > > > > > > > +1 > > > > > > > > > > Thanks for the proposal! > > > > > > > > > > Regards, > > > > > Keyong Zhou > > > > > > > > > > Nicholas Jiang <nicholasji...@apache.org> 于2024年6月12日周三 13:02写道： > > > > > > > > > > > +1. Looking forward to Celeborn CLI. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > Nicholas Jiang > > > > > > > > > > > > > > > > > > At 2024-06-12 12:26:34, "Aravind Patnam" <akpatna...@gmail.com> > > wrote: > > > > > > >Hi all, > > > > > > > > > > > > > >Sorry, this is the correct link to the Celeborn CLI CIP > > > > > > >< > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/CELEBORN/CIP+7+-+Celeborn+CLI> > > > > > > >. > > > > > > > > > > > > > >Thanks, > > > > > > >Aravind > > > > > > > > > > > > > >On Tue, Jun 11, 2024 at 9:24 PM Aravind Patnam < > > akpatna...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > >> Hi all, > > > > > > >> > > > > > > >> This is a call to vote to contribute the Celeborn CLI CIP > > > > > > >> < > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/CELEBORN/Celeborn+Improvement+Proposals > > > > > > > > > > > to > > > > > > >> Apache Celeborn. > > > > > > >> > > > > > > >> Please do vote accordingly: > > > > > > >> [ ] +1 approve > > > > > > >> [ ] +0 no opinion > > > > > > >> [ ] -1 disapprove (and the reason) > > > > > > >> > > > > > > >> Thanks once again!! > > > > > > >> > > > > > > >> Aravind > > > > > > >> > > > > > > > > > > > > > > > > > > > > >-- > > > > > > >Aravind K. Patnam > > > > > > > > > > > > > > > -- > Aravind K. Patnam
