Re: [DISCUSS] Celeborn CLI Proposal

[Aravind Patnam]

Sounds good!! I will start a vote thread.

Thanks,
Aravind

Aravind K. Patnam


On Tue, Jun 11, 2024 at 1:01 AM rexxiong  wrote:

> Hi Aravind,
>
> Thank you for the proposal. It looks good to me, agree with Yu that there
> is no blocker. Please go ahead and initiate a formal vote for this CIP, See
> the process section in the CIP guidelines
> <
> https://cwiki.apache.org/confluence/display/CELEBORN/Celeborn+Improvement+Proposals
> >
> .
>
>
> Thanks,
> Jiashu Xiong
>
> Yu Li  于2024年6月11日周二 15:26写道：
>
> > IIUIC, we need a formal vote on this CIP and make sure there are (at
> > least one) committer(s) to help review the coming up PRs.
> >
> > From the existing discussions, I believe there's no blocker but just a
> > normal process for this CIP to move on (smile).
> >
> > And thanks for the proposal, Aravind!
> >
> > Best Regards,
> > Yu
> >
> > On Tue, 11 Jun 2024 at 13:10, Aravind Patnam 
> wrote:
> > >
> > > Hi all,
> > >
> > > Thanks for the reviews everyone!!
> > >
> > > Is there any other process required (such as a vote thread), or can I
> > start
> > > contributing the CLI now?
> > >
> > > Thanks,
> > > Aravind
> > >
> > > Aravind K. Patnam
> > >
> > >
> > > On Mon, Jun 10, 2024 at 11:29 AM Mridul Muralidharan  >
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > >   Looks good to me as well, I had reviewed this proposal internally
> > already
> > > > :-)
> > > >
> > > > Regards,
> > > > Mridul
> > > >
> > > >
> > > > On Fri, Jun 7, 2024 at 11:32 PM Keyong Zhou 
> wrote:
> > > >
> > > > > Hi Aravind,
> > > > >
> > > > > Thanks for the proposal! The proposal LGTM, I think it's very
> > valuable.
> > > > >
> > > > > Regards,
> > > > > Keyong Zhou
> > > > >
> > > > > Aravind Patnam  于2024年6月7日周五 12:47写道：
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > Thanks Nicholas for the comments!
> > > > > >
> > > > > > I now got access to put the proposal in Confluence in the form of
> > CIP,
> > > > > here
> > > > > > <
> > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/CELEBORN/CIP+7+-+Celeborn+CLI
> > > > > > >
> > > > > > it is.
> > > > > >
> > > > > > Regarding your questions:
> > > > > >
> > > > > > > 1. From a user's perspective, the CLI is more used for some
> > > > maintenance
> > > > > > operations such as online and offline of server, rescaling of
> > cluster
> > > > > etc,
> > > > > > not only based on the REST API. What CLI interfaces are there
> that
> > the
> > > > > REST
> > > > > > API doesn’t have for maintenance?
> > > > > > This is highly dependent on what the user is leveraging to manage
> > their
> > > > > > cluster. For example, in k8s, you would be using k8s APIs to
> > achieve
> > > > > this.
> > > > > > We can probably add a generic interface API for it that provides
> > basic
> > > > > > operations that users can implement themselves for their cluster
> > > > > management
> > > > > > logic based on what cluster managers they are using. Although, I
> > think
> > > > > this
> > > > > > will likely be a later evolution of the CLI, once basic REST API
> > > > > operations
> > > > > > are implemented in the CLI. WDYT?
> > > > > >
> > > > > > > 2. There are same sub-commands between MASTER and WORKER. Why
> not
> > > > these
> > > > > > sub-commands belong to BOTH?
> > > > > > Agreed - this was a formatting mistake. I fixed it now, thanks
> for
> > > > > pointing
> > > > > > that out.
> > > > > >
> > > > > > > 3. Does the implementation of CLI invoke the REST API? IMO, the
> > CLI
> > > > > works
> > > > > > well no matter the server is alive.
> > > > > > Yes, I agree. I think for this we would have to talk to the
> cluster
> > > > > > manager, similar to my response to #1. We would have to query the
> > > > > specific
> > > > > > cluster manager to get details if the Celeborn servers are dead,
> > since
> > > > > the
> > > > > > Celeborn REST API would not work then. We can add a generic API
> > that
> > > > > users
> > > > > > can implement based on their own environment.
> > > > > >
> > > > > > Thanks,
> > > > > > Aravind
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Wed, Jun 5, 2024 at 10:43 PM Nicholas Jiang <
> > > > nicholasji...@apache.org
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Aravind,
> > > > > > >
> > > > > > > Thanks for driving this CIP about Celeborn CLI. I have some
> > comments
> > > > on
> > > > > > > this CIP:
> > > > > > >
> > > > > > > 1. From a user's perspective, the CLI is more used for some
> > > > maintenance
> > > > > > > operations such as online and offline of server, rescaling of
> > cluster
> > > > > > etc,
> > > > > > > not only based on the REST API. What CLI interfaces are there
> > that
> > > > the
> > > > > > REST
> > > > > > > API doesn’t have for maintenance?
> > > > > > >
> > > > > > > 2. There are same sub-commands between MASTER and WORKER. Why
> not
> > > > these
> > > > > > > sub-commands belong to BOTH?
> > > > > > >
> > > > > > > 3. Does the implementation of CLI invoke the REST API? IMO, the
> > CLI
> > > > > 

Re: [DISCUSS] Celeborn CLI Proposal

[rexxiong]

Hi Aravind,

Thank you for the proposal. It looks good to me, agree with Yu that there
is no blocker. Please go ahead and initiate a formal vote for this CIP, See
the process section in the CIP guidelines

.


Thanks,
Jiashu Xiong

Yu Li  于2024年6月11日周二 15:26写道：

> IIUIC, we need a formal vote on this CIP and make sure there are (at
> least one) committer(s) to help review the coming up PRs.
>
> From the existing discussions, I believe there's no blocker but just a
> normal process for this CIP to move on (smile).
>
> And thanks for the proposal, Aravind!
>
> Best Regards,
> Yu
>
> On Tue, 11 Jun 2024 at 13:10, Aravind Patnam  wrote:
> >
> > Hi all,
> >
> > Thanks for the reviews everyone!!
> >
> > Is there any other process required (such as a vote thread), or can I
> start
> > contributing the CLI now?
> >
> > Thanks,
> > Aravind
> >
> > Aravind K. Patnam
> >
> >
> > On Mon, Jun 10, 2024 at 11:29 AM Mridul Muralidharan 
> > wrote:
> >
> > > Hi,
> > >
> > >   Looks good to me as well, I had reviewed this proposal internally
> already
> > > :-)
> > >
> > > Regards,
> > > Mridul
> > >
> > >
> > > On Fri, Jun 7, 2024 at 11:32 PM Keyong Zhou  wrote:
> > >
> > > > Hi Aravind,
> > > >
> > > > Thanks for the proposal! The proposal LGTM, I think it's very
> valuable.
> > > >
> > > > Regards,
> > > > Keyong Zhou
> > > >
> > > > Aravind Patnam  于2024年6月7日周五 12:47写道：
> > > >
> > > > > Hi,
> > > > >
> > > > > Thanks Nicholas for the comments!
> > > > >
> > > > > I now got access to put the proposal in Confluence in the form of
> CIP,
> > > > here
> > > > > <
> > > >
> > >
> https://cwiki.apache.org/confluence/display/CELEBORN/CIP+7+-+Celeborn+CLI
> > > > > >
> > > > > it is.
> > > > >
> > > > > Regarding your questions:
> > > > >
> > > > > > 1. From a user's perspective, the CLI is more used for some
> > > maintenance
> > > > > operations such as online and offline of server, rescaling of
> cluster
> > > > etc,
> > > > > not only based on the REST API. What CLI interfaces are there that
> the
> > > > REST
> > > > > API doesn’t have for maintenance?
> > > > > This is highly dependent on what the user is leveraging to manage
> their
> > > > > cluster. For example, in k8s, you would be using k8s APIs to
> achieve
> > > > this.
> > > > > We can probably add a generic interface API for it that provides
> basic
> > > > > operations that users can implement themselves for their cluster
> > > > management
> > > > > logic based on what cluster managers they are using. Although, I
> think
> > > > this
> > > > > will likely be a later evolution of the CLI, once basic REST API
> > > > operations
> > > > > are implemented in the CLI. WDYT?
> > > > >
> > > > > > 2. There are same sub-commands between MASTER and WORKER. Why not
> > > these
> > > > > sub-commands belong to BOTH?
> > > > > Agreed - this was a formatting mistake. I fixed it now, thanks for
> > > > pointing
> > > > > that out.
> > > > >
> > > > > > 3. Does the implementation of CLI invoke the REST API? IMO, the
> CLI
> > > > works
> > > > > well no matter the server is alive.
> > > > > Yes, I agree. I think for this we would have to talk to the cluster
> > > > > manager, similar to my response to #1. We would have to query the
> > > > specific
> > > > > cluster manager to get details if the Celeborn servers are dead,
> since
> > > > the
> > > > > Celeborn REST API would not work then. We can add a generic API
> that
> > > > users
> > > > > can implement based on their own environment.
> > > > >
> > > > > Thanks,
> > > > > Aravind
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Jun 5, 2024 at 10:43 PM Nicholas Jiang <
> > > nicholasji...@apache.org
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Aravind,
> > > > > >
> > > > > > Thanks for driving this CIP about Celeborn CLI. I have some
> comments
> > > on
> > > > > > this CIP:
> > > > > >
> > > > > > 1. From a user's perspective, the CLI is more used for some
> > > maintenance
> > > > > > operations such as online and offline of server, rescaling of
> cluster
> > > > > etc,
> > > > > > not only based on the REST API. What CLI interfaces are there
> that
> > > the
> > > > > REST
> > > > > > API doesn’t have for maintenance?
> > > > > >
> > > > > > 2. There are same sub-commands between MASTER and WORKER. Why not
> > > these
> > > > > > sub-commands belong to BOTH?
> > > > > >
> > > > > > 3. Does the implementation of CLI invoke the REST API? IMO, the
> CLI
> > > > works
> > > > > > well no matter the server is alive.
> > > > > >
> > > > > > BTW, could this design doc of proposal follow the template of
> CIP[1]?
> > > > > >
> > > > > > [1]
> > > > > >
> > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/CELEBORN/Celeborn+Improvement+Proposals
> > > > > >
> > > > > > Regards,
> > > > > > Nicholas Jiang
> > > > > >
> > > > > > On 2024/06/05 23:33:02 Aravind Patnam wrote:
> > > > > > > Hi all,
> > > > > > >
> > > > > > 

Re: [DISCUSS] Celeborn CLI Proposal

[Yu Li]

IIUIC, we need a formal vote on this CIP and make sure there are (at
least one) committer(s) to help review the coming up PRs.

>From the existing discussions, I believe there's no blocker but just a
normal process for this CIP to move on (smile).

And thanks for the proposal, Aravind!

Best Regards,
Yu

On Tue, 11 Jun 2024 at 13:10, Aravind Patnam  wrote:
>
> Hi all,
>
> Thanks for the reviews everyone!!
>
> Is there any other process required (such as a vote thread), or can I start
> contributing the CLI now?
>
> Thanks,
> Aravind
>
> Aravind K. Patnam
>
>
> On Mon, Jun 10, 2024 at 11:29 AM Mridul Muralidharan 
> wrote:
>
> > Hi,
> >
> >   Looks good to me as well, I had reviewed this proposal internally already
> > :-)
> >
> > Regards,
> > Mridul
> >
> >
> > On Fri, Jun 7, 2024 at 11:32 PM Keyong Zhou  wrote:
> >
> > > Hi Aravind,
> > >
> > > Thanks for the proposal! The proposal LGTM, I think it's very valuable.
> > >
> > > Regards,
> > > Keyong Zhou
> > >
> > > Aravind Patnam  于2024年6月7日周五 12:47写道：
> > >
> > > > Hi,
> > > >
> > > > Thanks Nicholas for the comments!
> > > >
> > > > I now got access to put the proposal in Confluence in the form of CIP,
> > > here
> > > > <
> > >
> > https://cwiki.apache.org/confluence/display/CELEBORN/CIP+7+-+Celeborn+CLI
> > > > >
> > > > it is.
> > > >
> > > > Regarding your questions:
> > > >
> > > > > 1. From a user's perspective, the CLI is more used for some
> > maintenance
> > > > operations such as online and offline of server, rescaling of cluster
> > > etc,
> > > > not only based on the REST API. What CLI interfaces are there that the
> > > REST
> > > > API doesn’t have for maintenance?
> > > > This is highly dependent on what the user is leveraging to manage their
> > > > cluster. For example, in k8s, you would be using k8s APIs to achieve
> > > this.
> > > > We can probably add a generic interface API for it that provides basic
> > > > operations that users can implement themselves for their cluster
> > > management
> > > > logic based on what cluster managers they are using. Although, I think
> > > this
> > > > will likely be a later evolution of the CLI, once basic REST API
> > > operations
> > > > are implemented in the CLI. WDYT?
> > > >
> > > > > 2. There are same sub-commands between MASTER and WORKER. Why not
> > these
> > > > sub-commands belong to BOTH?
> > > > Agreed - this was a formatting mistake. I fixed it now, thanks for
> > > pointing
> > > > that out.
> > > >
> > > > > 3. Does the implementation of CLI invoke the REST API? IMO, the CLI
> > > works
> > > > well no matter the server is alive.
> > > > Yes, I agree. I think for this we would have to talk to the cluster
> > > > manager, similar to my response to #1. We would have to query the
> > > specific
> > > > cluster manager to get details if the Celeborn servers are dead, since
> > > the
> > > > Celeborn REST API would not work then. We can add a generic API that
> > > users
> > > > can implement based on their own environment.
> > > >
> > > > Thanks,
> > > > Aravind
> > > >
> > > >
> > > >
> > > > On Wed, Jun 5, 2024 at 10:43 PM Nicholas Jiang <
> > nicholasji...@apache.org
> > > >
> > > > wrote:
> > > >
> > > > > Hi Aravind,
> > > > >
> > > > > Thanks for driving this CIP about Celeborn CLI. I have some comments
> > on
> > > > > this CIP:
> > > > >
> > > > > 1. From a user's perspective, the CLI is more used for some
> > maintenance
> > > > > operations such as online and offline of server, rescaling of cluster
> > > > etc,
> > > > > not only based on the REST API. What CLI interfaces are there that
> > the
> > > > REST
> > > > > API doesn’t have for maintenance?
> > > > >
> > > > > 2. There are same sub-commands between MASTER and WORKER. Why not
> > these
> > > > > sub-commands belong to BOTH?
> > > > >
> > > > > 3. Does the implementation of CLI invoke the REST API? IMO, the CLI
> > > works
> > > > > well no matter the server is alive.
> > > > >
> > > > > BTW, could this design doc of proposal follow the template of CIP[1]?
> > > > >
> > > > > [1]
> > > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/CELEBORN/Celeborn+Improvement+Proposals
> > > > >
> > > > > Regards,
> > > > > Nicholas Jiang
> > > > >
> > > > > On 2024/06/05 23:33:02 Aravind Patnam wrote:
> > > > > > Hi all,
> > > > > >
> > > > > > I have written up a proposal about introducing a CLI for Celeborn.
> > > You
> > > > > can
> > > > > > find the proposal
> > > > > > <
> > > > >
> > > >
> > >
> > https://docs.google.com/document/d/1j9wKFSR_ychYDF0NU5YN67WCCtNAgYTbN5CN8V3SOnk/edit?usp=sharing
> > > > > >
> > > > > > here.
> > > > > > Please let me know if you have any comments or questions.
> > > > > >
> > > > > > TLDR by introducing a CLI, it would complement the existing
> > dashboard
> > > > and
> > > > > > would benefit us internally. We rely on CLI tools internally a lot
> > > for
> > > > > > automation and other operations.
> > > > > >
> > > > > > FYI, I was not able to access the cwiki page to put 

Re: [DISCUSS] Celeborn CLI Proposal

[Aravind Patnam]

Hi all,

Thanks for the reviews everyone!!

Is there any other process required (such as a vote thread), or can I start
contributing the CLI now?

Thanks,
Aravind

Aravind K. Patnam


On Mon, Jun 10, 2024 at 11:29 AM Mridul Muralidharan 
wrote:

> Hi,
>
>   Looks good to me as well, I had reviewed this proposal internally already
> :-)
>
> Regards,
> Mridul
>
>
> On Fri, Jun 7, 2024 at 11:32 PM Keyong Zhou  wrote:
>
> > Hi Aravind,
> >
> > Thanks for the proposal! The proposal LGTM, I think it's very valuable.
> >
> > Regards,
> > Keyong Zhou
> >
> > Aravind Patnam  于2024年6月7日周五 12:47写道：
> >
> > > Hi,
> > >
> > > Thanks Nicholas for the comments!
> > >
> > > I now got access to put the proposal in Confluence in the form of CIP,
> > here
> > > <
> >
> https://cwiki.apache.org/confluence/display/CELEBORN/CIP+7+-+Celeborn+CLI
> > > >
> > > it is.
> > >
> > > Regarding your questions:
> > >
> > > > 1. From a user's perspective, the CLI is more used for some
> maintenance
> > > operations such as online and offline of server, rescaling of cluster
> > etc,
> > > not only based on the REST API. What CLI interfaces are there that the
> > REST
> > > API doesn’t have for maintenance?
> > > This is highly dependent on what the user is leveraging to manage their
> > > cluster. For example, in k8s, you would be using k8s APIs to achieve
> > this.
> > > We can probably add a generic interface API for it that provides basic
> > > operations that users can implement themselves for their cluster
> > management
> > > logic based on what cluster managers they are using. Although, I think
> > this
> > > will likely be a later evolution of the CLI, once basic REST API
> > operations
> > > are implemented in the CLI. WDYT?
> > >
> > > > 2. There are same sub-commands between MASTER and WORKER. Why not
> these
> > > sub-commands belong to BOTH?
> > > Agreed - this was a formatting mistake. I fixed it now, thanks for
> > pointing
> > > that out.
> > >
> > > > 3. Does the implementation of CLI invoke the REST API? IMO, the CLI
> > works
> > > well no matter the server is alive.
> > > Yes, I agree. I think for this we would have to talk to the cluster
> > > manager, similar to my response to #1. We would have to query the
> > specific
> > > cluster manager to get details if the Celeborn servers are dead, since
> > the
> > > Celeborn REST API would not work then. We can add a generic API that
> > users
> > > can implement based on their own environment.
> > >
> > > Thanks,
> > > Aravind
> > >
> > >
> > >
> > > On Wed, Jun 5, 2024 at 10:43 PM Nicholas Jiang <
> nicholasji...@apache.org
> > >
> > > wrote:
> > >
> > > > Hi Aravind,
> > > >
> > > > Thanks for driving this CIP about Celeborn CLI. I have some comments
> on
> > > > this CIP:
> > > >
> > > > 1. From a user's perspective, the CLI is more used for some
> maintenance
> > > > operations such as online and offline of server, rescaling of cluster
> > > etc,
> > > > not only based on the REST API. What CLI interfaces are there that
> the
> > > REST
> > > > API doesn’t have for maintenance?
> > > >
> > > > 2. There are same sub-commands between MASTER and WORKER. Why not
> these
> > > > sub-commands belong to BOTH?
> > > >
> > > > 3. Does the implementation of CLI invoke the REST API? IMO, the CLI
> > works
> > > > well no matter the server is alive.
> > > >
> > > > BTW, could this design doc of proposal follow the template of CIP[1]?
> > > >
> > > > [1]
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/CELEBORN/Celeborn+Improvement+Proposals
> > > >
> > > > Regards,
> > > > Nicholas Jiang
> > > >
> > > > On 2024/06/05 23:33:02 Aravind Patnam wrote:
> > > > > Hi all,
> > > > >
> > > > > I have written up a proposal about introducing a CLI for Celeborn.
> > You
> > > > can
> > > > > find the proposal
> > > > > <
> > > >
> > >
> >
> https://docs.google.com/document/d/1j9wKFSR_ychYDF0NU5YN67WCCtNAgYTbN5CN8V3SOnk/edit?usp=sharing
> > > > >
> > > > > here.
> > > > > Please let me know if you have any comments or questions.
> > > > >
> > > > > TLDR by introducing a CLI, it would complement the existing
> dashboard
> > > and
> > > > > would benefit us internally. We rely on CLI tools internally a lot
> > for
> > > > > automation and other operations.
> > > > >
> > > > > FYI, I was not able to access the cwiki page to put this proposal
> > > there,
> > > > > there seems to be some permissions issue. Hope it is okay to just
> > share
> > > > as
> > > > > a google doc here for now.
> > > > >
> > > > > --
> > > > > Aravind K. Patnam
> > > > >
> > > > >  Apache Celeborn CLI Proposal
> > > > > <
> > > >
> > >
> >
> https://docs.google.com/document/d/1j9wKFSR_ychYDF0NU5YN67WCCtNAgYTbN5CN8V3SOnk/edit?usp=drive_web
> > > > >
> > > > >
> > > >
> > >
> > >
> > > --
> > > Aravind K. Patnam
> > >
> >
>

Re: [DISCUSS] Celeborn CLI Proposal

[Mridul Muralidharan]

Hi,

  Looks good to me as well, I had reviewed this proposal internally already
:-)

Regards,
Mridul


On Fri, Jun 7, 2024 at 11:32 PM Keyong Zhou  wrote:

> Hi Aravind,
>
> Thanks for the proposal! The proposal LGTM, I think it's very valuable.
>
> Regards,
> Keyong Zhou
>
> Aravind Patnam  于2024年6月7日周五 12:47写道：
>
> > Hi,
> >
> > Thanks Nicholas for the comments!
> >
> > I now got access to put the proposal in Confluence in the form of CIP,
> here
> > <
> https://cwiki.apache.org/confluence/display/CELEBORN/CIP+7+-+Celeborn+CLI
> > >
> > it is.
> >
> > Regarding your questions:
> >
> > > 1. From a user's perspective, the CLI is more used for some maintenance
> > operations such as online and offline of server, rescaling of cluster
> etc,
> > not only based on the REST API. What CLI interfaces are there that the
> REST
> > API doesn’t have for maintenance?
> > This is highly dependent on what the user is leveraging to manage their
> > cluster. For example, in k8s, you would be using k8s APIs to achieve
> this.
> > We can probably add a generic interface API for it that provides basic
> > operations that users can implement themselves for their cluster
> management
> > logic based on what cluster managers they are using. Although, I think
> this
> > will likely be a later evolution of the CLI, once basic REST API
> operations
> > are implemented in the CLI. WDYT?
> >
> > > 2. There are same sub-commands between MASTER and WORKER. Why not these
> > sub-commands belong to BOTH?
> > Agreed - this was a formatting mistake. I fixed it now, thanks for
> pointing
> > that out.
> >
> > > 3. Does the implementation of CLI invoke the REST API? IMO, the CLI
> works
> > well no matter the server is alive.
> > Yes, I agree. I think for this we would have to talk to the cluster
> > manager, similar to my response to #1. We would have to query the
> specific
> > cluster manager to get details if the Celeborn servers are dead, since
> the
> > Celeborn REST API would not work then. We can add a generic API that
> users
> > can implement based on their own environment.
> >
> > Thanks,
> > Aravind
> >
> >
> >
> > On Wed, Jun 5, 2024 at 10:43 PM Nicholas Jiang  >
> > wrote:
> >
> > > Hi Aravind,
> > >
> > > Thanks for driving this CIP about Celeborn CLI. I have some comments on
> > > this CIP:
> > >
> > > 1. From a user's perspective, the CLI is more used for some maintenance
> > > operations such as online and offline of server, rescaling of cluster
> > etc,
> > > not only based on the REST API. What CLI interfaces are there that the
> > REST
> > > API doesn’t have for maintenance?
> > >
> > > 2. There are same sub-commands between MASTER and WORKER. Why not these
> > > sub-commands belong to BOTH?
> > >
> > > 3. Does the implementation of CLI invoke the REST API? IMO, the CLI
> works
> > > well no matter the server is alive.
> > >
> > > BTW, could this design doc of proposal follow the template of CIP[1]?
> > >
> > > [1]
> > >
> >
> https://cwiki.apache.org/confluence/display/CELEBORN/Celeborn+Improvement+Proposals
> > >
> > > Regards,
> > > Nicholas Jiang
> > >
> > > On 2024/06/05 23:33:02 Aravind Patnam wrote:
> > > > Hi all,
> > > >
> > > > I have written up a proposal about introducing a CLI for Celeborn.
> You
> > > can
> > > > find the proposal
> > > > <
> > >
> >
> https://docs.google.com/document/d/1j9wKFSR_ychYDF0NU5YN67WCCtNAgYTbN5CN8V3SOnk/edit?usp=sharing
> > > >
> > > > here.
> > > > Please let me know if you have any comments or questions.
> > > >
> > > > TLDR by introducing a CLI, it would complement the existing dashboard
> > and
> > > > would benefit us internally. We rely on CLI tools internally a lot
> for
> > > > automation and other operations.
> > > >
> > > > FYI, I was not able to access the cwiki page to put this proposal
> > there,
> > > > there seems to be some permissions issue. Hope it is okay to just
> share
> > > as
> > > > a google doc here for now.
> > > >
> > > > --
> > > > Aravind K. Patnam
> > > >
> > > >  Apache Celeborn CLI Proposal
> > > > <
> > >
> >
> https://docs.google.com/document/d/1j9wKFSR_ychYDF0NU5YN67WCCtNAgYTbN5CN8V3SOnk/edit?usp=drive_web
> > > >
> > > >
> > >
> >
> >
> > --
> > Aravind K. Patnam
> >
>

Re: [DISCUSS] Celeborn CLI Proposal

[Keyong Zhou]

Hi Aravind,

Thanks for the proposal! The proposal LGTM, I think it's very valuable.

Regards,
Keyong Zhou

Aravind Patnam  于2024年6月7日周五 12:47写道：

> Hi,
>
> Thanks Nicholas for the comments!
>
> I now got access to put the proposal in Confluence in the form of CIP, here
>  >
> it is.
>
> Regarding your questions:
>
> > 1. From a user's perspective, the CLI is more used for some maintenance
> operations such as online and offline of server, rescaling of cluster etc,
> not only based on the REST API. What CLI interfaces are there that the REST
> API doesn’t have for maintenance?
> This is highly dependent on what the user is leveraging to manage their
> cluster. For example, in k8s, you would be using k8s APIs to achieve this.
> We can probably add a generic interface API for it that provides basic
> operations that users can implement themselves for their cluster management
> logic based on what cluster managers they are using. Although, I think this
> will likely be a later evolution of the CLI, once basic REST API operations
> are implemented in the CLI. WDYT?
>
> > 2. There are same sub-commands between MASTER and WORKER. Why not these
> sub-commands belong to BOTH?
> Agreed - this was a formatting mistake. I fixed it now, thanks for pointing
> that out.
>
> > 3. Does the implementation of CLI invoke the REST API? IMO, the CLI works
> well no matter the server is alive.
> Yes, I agree. I think for this we would have to talk to the cluster
> manager, similar to my response to #1. We would have to query the specific
> cluster manager to get details if the Celeborn servers are dead, since the
> Celeborn REST API would not work then. We can add a generic API that users
> can implement based on their own environment.
>
> Thanks,
> Aravind
>
>
>
> On Wed, Jun 5, 2024 at 10:43 PM Nicholas Jiang 
> wrote:
>
> > Hi Aravind,
> >
> > Thanks for driving this CIP about Celeborn CLI. I have some comments on
> > this CIP:
> >
> > 1. From a user's perspective, the CLI is more used for some maintenance
> > operations such as online and offline of server, rescaling of cluster
> etc,
> > not only based on the REST API. What CLI interfaces are there that the
> REST
> > API doesn’t have for maintenance?
> >
> > 2. There are same sub-commands between MASTER and WORKER. Why not these
> > sub-commands belong to BOTH?
> >
> > 3. Does the implementation of CLI invoke the REST API? IMO, the CLI works
> > well no matter the server is alive.
> >
> > BTW, could this design doc of proposal follow the template of CIP[1]?
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/CELEBORN/Celeborn+Improvement+Proposals
> >
> > Regards,
> > Nicholas Jiang
> >
> > On 2024/06/05 23:33:02 Aravind Patnam wrote:
> > > Hi all,
> > >
> > > I have written up a proposal about introducing a CLI for Celeborn. You
> > can
> > > find the proposal
> > > <
> >
> https://docs.google.com/document/d/1j9wKFSR_ychYDF0NU5YN67WCCtNAgYTbN5CN8V3SOnk/edit?usp=sharing
> > >
> > > here.
> > > Please let me know if you have any comments or questions.
> > >
> > > TLDR by introducing a CLI, it would complement the existing dashboard
> and
> > > would benefit us internally. We rely on CLI tools internally a lot for
> > > automation and other operations.
> > >
> > > FYI, I was not able to access the cwiki page to put this proposal
> there,
> > > there seems to be some permissions issue. Hope it is okay to just share
> > as
> > > a google doc here for now.
> > >
> > > --
> > > Aravind K. Patnam
> > >
> > >  Apache Celeborn CLI Proposal
> > > <
> >
> https://docs.google.com/document/d/1j9wKFSR_ychYDF0NU5YN67WCCtNAgYTbN5CN8V3SOnk/edit?usp=drive_web
> > >
> > >
> >
>
>
> --
> Aravind K. Patnam
>

Re: [DISCUSS] Celeborn CLI Proposal

[Aravind Patnam]

Hi,

Thanks Nicholas for the comments!

I now got access to put the proposal in Confluence in the form of CIP, here

it is.

Regarding your questions:

> 1. From a user's perspective, the CLI is more used for some maintenance
operations such as online and offline of server, rescaling of cluster etc,
not only based on the REST API. What CLI interfaces are there that the REST
API doesn’t have for maintenance?
This is highly dependent on what the user is leveraging to manage their
cluster. For example, in k8s, you would be using k8s APIs to achieve this.
We can probably add a generic interface API for it that provides basic
operations that users can implement themselves for their cluster management
logic based on what cluster managers they are using. Although, I think this
will likely be a later evolution of the CLI, once basic REST API operations
are implemented in the CLI. WDYT?

> 2. There are same sub-commands between MASTER and WORKER. Why not these
sub-commands belong to BOTH?
Agreed - this was a formatting mistake. I fixed it now, thanks for pointing
that out.

> 3. Does the implementation of CLI invoke the REST API? IMO, the CLI works
well no matter the server is alive.
Yes, I agree. I think for this we would have to talk to the cluster
manager, similar to my response to #1. We would have to query the specific
cluster manager to get details if the Celeborn servers are dead, since the
Celeborn REST API would not work then. We can add a generic API that users
can implement based on their own environment.

Thanks,
Aravind



On Wed, Jun 5, 2024 at 10:43 PM Nicholas Jiang 
wrote:

> Hi Aravind,
>
> Thanks for driving this CIP about Celeborn CLI. I have some comments on
> this CIP:
>
> 1. From a user's perspective, the CLI is more used for some maintenance
> operations such as online and offline of server, rescaling of cluster etc,
> not only based on the REST API. What CLI interfaces are there that the REST
> API doesn’t have for maintenance?
>
> 2. There are same sub-commands between MASTER and WORKER. Why not these
> sub-commands belong to BOTH?
>
> 3. Does the implementation of CLI invoke the REST API? IMO, the CLI works
> well no matter the server is alive.
>
> BTW, could this design doc of proposal follow the template of CIP[1]?
>
> [1]
> https://cwiki.apache.org/confluence/display/CELEBORN/Celeborn+Improvement+Proposals
>
> Regards,
> Nicholas Jiang
>
> On 2024/06/05 23:33:02 Aravind Patnam wrote:
> > Hi all,
> >
> > I have written up a proposal about introducing a CLI for Celeborn. You
> can
> > find the proposal
> > <
> https://docs.google.com/document/d/1j9wKFSR_ychYDF0NU5YN67WCCtNAgYTbN5CN8V3SOnk/edit?usp=sharing
> >
> > here.
> > Please let me know if you have any comments or questions.
> >
> > TLDR by introducing a CLI, it would complement the existing dashboard and
> > would benefit us internally. We rely on CLI tools internally a lot for
> > automation and other operations.
> >
> > FYI, I was not able to access the cwiki page to put this proposal there,
> > there seems to be some permissions issue. Hope it is okay to just share
> as
> > a google doc here for now.
> >
> > --
> > Aravind K. Patnam
> >
> >  Apache Celeborn CLI Proposal
> > <
> https://docs.google.com/document/d/1j9wKFSR_ychYDF0NU5YN67WCCtNAgYTbN5CN8V3SOnk/edit?usp=drive_web
> >
> >
>


-- 
Aravind K. Patnam

Re: [DISCUSS] Celeborn CLI Proposal

[Nicholas Jiang]

Hi Aravind,

Thanks for driving this CIP about Celeborn CLI. I have some comments on this 
CIP:

1. From a user's perspective, the CLI is more used for some maintenance 
operations such as online and offline of server, rescaling of cluster etc, not 
only based on the REST API. What CLI interfaces are there that the REST API 
doesn’t have for maintenance?

2. There are same sub-commands between MASTER and WORKER. Why not these 
sub-commands belong to BOTH?

3. Does the implementation of CLI invoke the REST API? IMO, the CLI works well 
no matter the server is alive.

BTW, could this design doc of proposal follow the template of CIP[1]?

[1] 
https://cwiki.apache.org/confluence/display/CELEBORN/Celeborn+Improvement+Proposals

Regards,
Nicholas Jiang

On 2024/06/05 23:33:02 Aravind Patnam wrote:
> Hi all,
> 
> I have written up a proposal about introducing a CLI for Celeborn. You can
> find the proposal
> 
> here.
> Please let me know if you have any comments or questions.
> 
> TLDR by introducing a CLI, it would complement the existing dashboard and
> would benefit us internally. We rely on CLI tools internally a lot for
> automation and other operations.
> 
> FYI, I was not able to access the cwiki page to put this proposal there,
> there seems to be some permissions issue. Hope it is okay to just share as
> a google doc here for now.
> 
> -- 
> Aravind K. Patnam
> 
>  Apache Celeborn CLI Proposal
> 
> 

[DISCUSS] Celeborn CLI Proposal

[Aravind Patnam]

Hi all,

I have written up a proposal about introducing a CLI for Celeborn. You can
find the proposal

here.
Please let me know if you have any comments or questions.

TLDR by introducing a CLI, it would complement the existing dashboard and
would benefit us internally. We rely on CLI tools internally a lot for
automation and other operations.

FYI, I was not able to access the cwiki page to put this proposal there,
there seems to be some permissions issue. Hope it is okay to just share as
a google doc here for now.

-- 
Aravind K. Patnam

 Apache Celeborn CLI Proposal