[PR] add description about how to update license and notice [celeborn-website]
FMX opened a new pull request, #67: URL: https://github.com/apache/celeborn-website/pull/67 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@celeborn.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] add description about how to update license and notice [celeborn-website]
pan3793 commented on code in PR #67: URL: https://github.com/apache/celeborn-website/pull/67#discussion_r1636041487 ## docs/community/release_guide.md: ## @@ -175,6 +175,45 @@ After cutting release branch, don't forget bump version in `main` branch. export RELEASE_RC_NO= ``` +- Check License and Notice + +Build a release package and check license and notice with the binary package's jars directory instead of the pom.xml. +```shell +./buile/make-distribution.sh --release --sbt-enabled +``` +According to the Apache License section 4, the principal of License and Notice is as following: Review Comment: can we add the hyperlink for "Apache License section 4", and use quote syntax for the reference sections -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@celeborn.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [VOTE] Release Apache Celeborn 0.5.0-rc2
+1 (non-binding) I have checked: - Download links are valid. - git commit hash is correct - Checksums and signatures are valid. - No binary files in the source release - Successfully built binary from source via ./build/make-distribution.sh -Pspark-3.5/flink-1.19/mr Meanwhile, I have upgraded internal version to 0.5.0 in our production environment. Regards, Nicholas Jiang On 2024/06/12 05:56:07 Ethan Feng wrote: > Hello, Celeborn community, > > This is a call for a vote to release Apache Celeborn > 0.5.0-rc2 > > The git tag to be voted upon: > https://github.com/apache/celeborn/releases/tag/v0.5.0-rc2 > > Source and binary artifacts can be found at: > https://dist.apache.org/repos/dist/dev/celeborn/v0.5.0-rc2 > > The git commit hash: > 68c503eb0023e274f8ae09bf4c2687f6a0c01a25 > > The staging repo: > https://repository.apache.org/content/repositories/orgapacheceleborn-1075/ > > The fingerprint of the PGP key release artifacts is signed with: > FCF20BB29C7BEFDF58F998F76392F71F37356FA0 > > My public key to verify signatures can be found in: > https://dist.apache.org/repos/dist/release/celeborn/KEYS > > The vote will be open for at least 72 hours or until the necessary > number of votes are reached. > > Please vote accordingly: > > [ ] +1 approve > [ ] +0 no opinion > [ ] -1 disapprove (and the reason) > > Steps to validate the release: > https://www.apache.org/info/verification.html > > * Download links, checksums, and PGP signatures are valid. > * Source code distributions have correct names matching the current release. > * LICENSE and NOTICE files are correct. > * All files have license headers if necessary. > * No unlicensed compiled archives bundled in the source archive. > * The source tarball matches the git tag. > * Build from source is successful. > > There are additional tests: > * Performance test no regression > 1 TB TPC-DS, 0.5.0 VS 0.4.1 : 2042(s) VS 2050(s) > 1.1 TB pure shuffle, 0.5.0 VS 0.4.1 : 11.8min vs 11.8min > > * Result correctness test passed > 1TB TPC-DS runs concurrently, the results are identical. > > * Usability test passed > Rolling upgrade from version 0.4.1 to 0.5.0 succeed. > The metrics system works as expected. > > * Stability test passed > Random worker failures, Celeborn works as expected. > Random master failures, Celeborn works as expected. > Master meta corrupted, Celeborn works as expected. > > * Compatibility test passed > The Celeborn server version of 0.5.0 works fine with the Celeborn client > 0.4.1. > > > Regards, > Ethan Feng >
Re: Re: [VOTE] Contrinute Apache Celeborn CLI
Hi Ethan, Thanks for your comments! Regarding using Java/Scala for the CLI, I am fine with this. I had believed that using Python would be an easier/simpler implementation given that many CLI's are implemented in Python, but the points you make are fair. Most of the Celeborn community uses Java/Scala, so this would be more beneficial for the development and evolution of the CLI. Yes, I think the CLI should contain capabilities beyond the HTTP endpoints Celeborn exposes. The Celeborn HTTP endpoints work great for application specific use cases, such as finding the applications or shuffles on a particular worker, however it would not work for situations in which we would need information on the cluster itself. For example, we use K8s and these are use cases internally I can foresee that require communication with an external cluster manager: - Retrieve all pods running masters/workers and their statuses - Manually evict an Celeborn unhealthy pod - SSH into various different Celeborn pods - Manage ACLs of the cluster - Manually restart pods - Wipe Ratis storage if state is messed up - Wipe shuffle directories if state is messed up - Adding/removing new nodes into our node pool - Perform any other manual arbitrary function on a Celeborn pod These are just a few of the use cases I can think of, but I am sure more will arise as more users adopt Celeborn :) Given that users will have various different cluster managers, I think as I mentioned before there should be an abstraction layer present that exposes different operations. Based on the cluster manager the user is using, the user can implement their specific logic. We can have a few default ones included (e.g. Kubernetes). Hope this answers your questions, let me know if you have any more questions! Thanks, Aravind On Tue, Jun 11, 2024 at 11:57 PM Ethan Feng wrote: > Hi Aravind, > > I hope this message finds you well. I wanted to express my > appreciation for the energy and creativity you've invested in the > Celeborn project; the proposal you submitted is intriguing. > > I apologize for the delayed feedback on your proposal — it took me a > bit longer to get to it than anticipated. After reviewing it, I have a > couple of inquiries that I'd like to discuss in order to gain a > clearer understanding: > > I observed that you're planning to implement the CLI in Python. Could > you elaborate on the choice behind not leveraging the Java stack for > this purpose? The Java ecosystem already includes mature tools such as > "commons-cli" or "Scala CLI," which are capable of facilitating CLI > tool development. Given the prevalent familiarity with the Java stack > within our community, I believe leveraging it could accelerate the > CLI's development and evolution through wider collaboration. > > From email discussions, you've indicated an interest in offering a > generic interface API for Celeborn, which is certainly exciting. > However, I'm concerned that basing a CLI on HTTP API might not fully > align with this vision. Could you provide additional insights into how > you envision the CLI advancing beyond the capabilities of the current > HTTP REST API? > > Based on previous exchanges, the CLI is expected to communicate with > an external cluster manager. Is there an abstraction layer in place to > interface uniformly with various external cluster managers, or is this > something under consideration? > > I'm looking forward to learning more about your perspectives and the > pathway you foresee for the CLI's development. > > regards, > Ethan > > Mridul Muralidharan 于2024年6月12日周三 14:36写道： > > > > +1 > > > > Regards, > > Mridul > > > > > > On Wed, Jun 12, 2024 at 1:08 AM Shaoyun Chen wrote: > > > > > +1 > > > > > > Keyong Zhou 于2024年6月12日周三 13:47写道： > > > > > > > > +1 > > > > > > > > Thanks for the proposal! > > > > > > > > Regards, > > > > Keyong Zhou > > > > > > > > Nicholas Jiang 于2024年6月12日周三 13:02写道： > > > > > > > > > +1. Looking forward to Celeborn CLI. > > > > > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > Nicholas Jiang > > > > > > > > > > > > > > > At 2024-06-12 12:26:34, "Aravind Patnam" > wrote: > > > > > >Hi all, > > > > > > > > > > > >Sorry, this is the correct link to the Celeborn CLI CIP > > > > > >< > > > > > > > > > https://cwiki.apache.org/confluence/display/CELEBORN/CIP+7+-+Celeborn+CLI> > > > > > >. > > > > > > > > > > > >Thanks, > > > > > >Aravind > > > > > > > > > > > >On Tue, Jun 11, 2024 at 9:24 PM Aravind Patnam < > akpatna...@gmail.com> > > > > > wrote: > > > > > > > > > > > >> Hi all, > > > > > >> > > > > > >> This is a call to vote to contribute the Celeborn CLI CIP > > > > > >> < > > > > > > > > > https://cwiki.apache.org/confluence/display/CELEBORN/Celeborn+Improvement+Proposals > > > > > > > > > to > > > > > >> Apache Celeborn. > > > > > >> > > > > > >> Please do vote accordingly: > > > > > >> [ ] +1 approve > > > > > >> [ ] +0 no opinion > > > > > >> [ ] -1 disapprove (an
Re: Re: [VOTE] Contrinute Apache Celeborn CLI
Hi Aravind, Thanks for your detailed response and insight into the CLI implementation for Celeborn. I appreciate the explanation regarding the choice to use Java/Scala for the CLI. I agree that aligning with the prevalent use of Java/Scala within the Celeborn community is the right approach. Your proposed use cases for the CLI functionalities are comprehensive and interesting. I appreciate your proactive approach in considering various cluster management systems and the need for an abstraction layer to interface uniformly. +1 for the current proposal. Looking forward to the CLI evolution. regards, Ethan Aravind Patnam 于2024年6月13日周四 05:35写道： > > Hi Ethan, > > Thanks for your comments! > > Regarding using Java/Scala for the CLI, I am fine with this. I had believed > that using Python would be an easier/simpler implementation given that many > CLI's are implemented in Python, but the points you make are fair. Most of > the Celeborn community uses Java/Scala, so this would be more beneficial > for the development and evolution of the CLI. > > Yes, I think the CLI should contain capabilities beyond the HTTP endpoints > Celeborn exposes. The Celeborn HTTP endpoints work great for application > specific use cases, such as finding the applications or shuffles on a > particular worker, however it would not work for situations in which we > would need information on the cluster itself. For example, we use K8s and > these are use cases internally I can foresee that require communication > with an external cluster manager: > >- Retrieve all pods running masters/workers and their statuses >- Manually evict an Celeborn unhealthy pod >- SSH into various different Celeborn pods >- Manage ACLs of the cluster >- Manually restart pods >- Wipe Ratis storage if state is messed up >- Wipe shuffle directories if state is messed up >- Adding/removing new nodes into our node pool >- Perform any other manual arbitrary function on a Celeborn pod > > > These are just a few of the use cases I can think of, but I am sure more > will arise as more users adopt Celeborn :) > > Given that users will have various different cluster managers, I think as I > mentioned before there should be an abstraction layer present that exposes > different operations. Based on the cluster manager the user is using, the > user can implement their specific logic. We can have a few default ones > included (e.g. Kubernetes). > > Hope this answers your questions, let me know if you have any more > questions! > > Thanks, > Aravind > > On Tue, Jun 11, 2024 at 11:57 PM Ethan Feng wrote: > > > Hi Aravind, > > > > I hope this message finds you well. I wanted to express my > > appreciation for the energy and creativity you've invested in the > > Celeborn project; the proposal you submitted is intriguing. > > > > I apologize for the delayed feedback on your proposal — it took me a > > bit longer to get to it than anticipated. After reviewing it, I have a > > couple of inquiries that I'd like to discuss in order to gain a > > clearer understanding: > > > > I observed that you're planning to implement the CLI in Python. Could > > you elaborate on the choice behind not leveraging the Java stack for > > this purpose? The Java ecosystem already includes mature tools such as > > "commons-cli" or "Scala CLI," which are capable of facilitating CLI > > tool development. Given the prevalent familiarity with the Java stack > > within our community, I believe leveraging it could accelerate the > > CLI's development and evolution through wider collaboration. > > > > From email discussions, you've indicated an interest in offering a > > generic interface API for Celeborn, which is certainly exciting. > > However, I'm concerned that basing a CLI on HTTP API might not fully > > align with this vision. Could you provide additional insights into how > > you envision the CLI advancing beyond the capabilities of the current > > HTTP REST API? > > > > Based on previous exchanges, the CLI is expected to communicate with > > an external cluster manager. Is there an abstraction layer in place to > > interface uniformly with various external cluster managers, or is this > > something under consideration? > > > > I'm looking forward to learning more about your perspectives and the > > pathway you foresee for the CLI's development. > > > > regards, > > Ethan > > > > Mridul Muralidharan 于2024年6月12日周三 14:36写道： > > > > > > +1 > > > > > > Regards, > > > Mridul > > > > > > > > > On Wed, Jun 12, 2024 at 1:08 AM Shaoyun Chen wrote: > > > > > > > +1 > > > > > > > > Keyong Zhou 于2024年6月12日周三 13:47写道： > > > > > > > > > > +1 > > > > > > > > > > Thanks for the proposal! > > > > > > > > > > Regards, > > > > > Keyong Zhou > > > > > > > > > > Nicholas Jiang 于2024年6月12日周三 13:02写道： > > > > > > > > > > > +1. Looking forward to Celeborn CLI. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > Nicholas Jiang > > > > > > > > >
Re: Re: [VOTE] Contrinute Apache Celeborn CLI
+1 Thanks for the proposal! Thanks, Fu Chen Ethan Feng 于2024年6月13日周四 10:59写道： > > Hi Aravind, > > Thanks for your detailed response and insight into the CLI > implementation for Celeborn. I appreciate the explanation regarding > the choice to use Java/Scala for the CLI. I agree that aligning with > the prevalent use of Java/Scala within the Celeborn community is the > right approach. > > Your proposed use cases for the CLI functionalities are comprehensive > and interesting. I appreciate your proactive approach in considering > various cluster management systems and the need for an abstraction > layer to interface uniformly. > > +1 for the current proposal. > > Looking forward to the CLI evolution. > > regards, > Ethan > > Aravind Patnam 于2024年6月13日周四 05:35写道： > > > > Hi Ethan, > > > > Thanks for your comments! > > > > Regarding using Java/Scala for the CLI, I am fine with this. I had believed > > that using Python would be an easier/simpler implementation given that many > > CLI's are implemented in Python, but the points you make are fair. Most of > > the Celeborn community uses Java/Scala, so this would be more beneficial > > for the development and evolution of the CLI. > > > > Yes, I think the CLI should contain capabilities beyond the HTTP endpoints > > Celeborn exposes. The Celeborn HTTP endpoints work great for application > > specific use cases, such as finding the applications or shuffles on a > > particular worker, however it would not work for situations in which we > > would need information on the cluster itself. For example, we use K8s and > > these are use cases internally I can foresee that require communication > > with an external cluster manager: > > > >- Retrieve all pods running masters/workers and their statuses > >- Manually evict an Celeborn unhealthy pod > >- SSH into various different Celeborn pods > >- Manage ACLs of the cluster > >- Manually restart pods > >- Wipe Ratis storage if state is messed up > >- Wipe shuffle directories if state is messed up > >- Adding/removing new nodes into our node pool > >- Perform any other manual arbitrary function on a Celeborn pod > > > > > > These are just a few of the use cases I can think of, but I am sure more > > will arise as more users adopt Celeborn :) > > > > Given that users will have various different cluster managers, I think as I > > mentioned before there should be an abstraction layer present that exposes > > different operations. Based on the cluster manager the user is using, the > > user can implement their specific logic. We can have a few default ones > > included (e.g. Kubernetes). > > > > Hope this answers your questions, let me know if you have any more > > questions! > > > > Thanks, > > Aravind > > > > On Tue, Jun 11, 2024 at 11:57 PM Ethan Feng wrote: > > > > > Hi Aravind, > > > > > > I hope this message finds you well. I wanted to express my > > > appreciation for the energy and creativity you've invested in the > > > Celeborn project; the proposal you submitted is intriguing. > > > > > > I apologize for the delayed feedback on your proposal — it took me a > > > bit longer to get to it than anticipated. After reviewing it, I have a > > > couple of inquiries that I'd like to discuss in order to gain a > > > clearer understanding: > > > > > > I observed that you're planning to implement the CLI in Python. Could > > > you elaborate on the choice behind not leveraging the Java stack for > > > this purpose? The Java ecosystem already includes mature tools such as > > > "commons-cli" or "Scala CLI," which are capable of facilitating CLI > > > tool development. Given the prevalent familiarity with the Java stack > > > within our community, I believe leveraging it could accelerate the > > > CLI's development and evolution through wider collaboration. > > > > > > From email discussions, you've indicated an interest in offering a > > > generic interface API for Celeborn, which is certainly exciting. > > > However, I'm concerned that basing a CLI on HTTP API might not fully > > > align with this vision. Could you provide additional insights into how > > > you envision the CLI advancing beyond the capabilities of the current > > > HTTP REST API? > > > > > > Based on previous exchanges, the CLI is expected to communicate with > > > an external cluster manager. Is there an abstraction layer in place to > > > interface uniformly with various external cluster managers, or is this > > > something under consideration? > > > > > > I'm looking forward to learning more about your perspectives and the > > > pathway you foresee for the CLI's development. > > > > > > regards, > > > Ethan > > > > > > Mridul Muralidharan 于2024年6月12日周三 14:36写道： > > > > > > > > +1 > > > > > > > > Regards, > > > > Mridul > > > > > > > > > > > > On Wed, Jun 12, 2024 at 1:08 AM Shaoyun Chen wrote: > > > > > > > > > +1 > > > > > > > > > > Keyong Zhou 于2024年6月12日周三 13:47写道： > > > > > > > > > > > > +1 > > > > > > > > > > >
Re: [PR] add description about how to update license and notice [celeborn-website]
FMX commented on code in PR #67: URL: https://github.com/apache/celeborn-website/pull/67#discussion_r1637599694 ## docs/community/release_guide.md: ## @@ -175,6 +175,45 @@ After cutting release branch, don't forget bump version in `main` branch. export RELEASE_RC_NO= ``` +- Check License and Notice + +Build a release package and check license and notice with the binary package's jars directory instead of the pom.xml. +```shell +./buile/make-distribution.sh --release --sbt-enabled +``` +According to the Apache License section 4, the principal of License and Notice is as following: Review Comment: Added the hyperlink. The following text is not a reference, it's a conclusion. Ping @pan3793 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@celeborn.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: Re: [VOTE] Contrinute Apache Celeborn CLI
+1 Best Regards, Yu On Thu, 13 Jun 2024 at 13:54, Fu Chen wrote: > > +1 > > Thanks for the proposal! > > Thanks, > Fu Chen > > Ethan Feng 于2024年6月13日周四 10:59写道： > > > > Hi Aravind, > > > > Thanks for your detailed response and insight into the CLI > > implementation for Celeborn. I appreciate the explanation regarding > > the choice to use Java/Scala for the CLI. I agree that aligning with > > the prevalent use of Java/Scala within the Celeborn community is the > > right approach. > > > > Your proposed use cases for the CLI functionalities are comprehensive > > and interesting. I appreciate your proactive approach in considering > > various cluster management systems and the need for an abstraction > > layer to interface uniformly. > > > > +1 for the current proposal. > > > > Looking forward to the CLI evolution. > > > > regards, > > Ethan > > > > Aravind Patnam 于2024年6月13日周四 05:35写道： > > > > > > Hi Ethan, > > > > > > Thanks for your comments! > > > > > > Regarding using Java/Scala for the CLI, I am fine with this. I had > > > believed > > > that using Python would be an easier/simpler implementation given that > > > many > > > CLI's are implemented in Python, but the points you make are fair. Most of > > > the Celeborn community uses Java/Scala, so this would be more beneficial > > > for the development and evolution of the CLI. > > > > > > Yes, I think the CLI should contain capabilities beyond the HTTP endpoints > > > Celeborn exposes. The Celeborn HTTP endpoints work great for application > > > specific use cases, such as finding the applications or shuffles on a > > > particular worker, however it would not work for situations in which we > > > would need information on the cluster itself. For example, we use K8s and > > > these are use cases internally I can foresee that require communication > > > with an external cluster manager: > > > > > >- Retrieve all pods running masters/workers and their statuses > > >- Manually evict an Celeborn unhealthy pod > > >- SSH into various different Celeborn pods > > >- Manage ACLs of the cluster > > >- Manually restart pods > > >- Wipe Ratis storage if state is messed up > > >- Wipe shuffle directories if state is messed up > > >- Adding/removing new nodes into our node pool > > >- Perform any other manual arbitrary function on a Celeborn pod > > > > > > > > > These are just a few of the use cases I can think of, but I am sure more > > > will arise as more users adopt Celeborn :) > > > > > > Given that users will have various different cluster managers, I think as > > > I > > > mentioned before there should be an abstraction layer present that exposes > > > different operations. Based on the cluster manager the user is using, the > > > user can implement their specific logic. We can have a few default ones > > > included (e.g. Kubernetes). > > > > > > Hope this answers your questions, let me know if you have any more > > > questions! > > > > > > Thanks, > > > Aravind > > > > > > On Tue, Jun 11, 2024 at 11:57 PM Ethan Feng wrote: > > > > > > > Hi Aravind, > > > > > > > > I hope this message finds you well. I wanted to express my > > > > appreciation for the energy and creativity you've invested in the > > > > Celeborn project; the proposal you submitted is intriguing. > > > > > > > > I apologize for the delayed feedback on your proposal — it took me a > > > > bit longer to get to it than anticipated. After reviewing it, I have a > > > > couple of inquiries that I'd like to discuss in order to gain a > > > > clearer understanding: > > > > > > > > I observed that you're planning to implement the CLI in Python. Could > > > > you elaborate on the choice behind not leveraging the Java stack for > > > > this purpose? The Java ecosystem already includes mature tools such as > > > > "commons-cli" or "Scala CLI," which are capable of facilitating CLI > > > > tool development. Given the prevalent familiarity with the Java stack > > > > within our community, I believe leveraging it could accelerate the > > > > CLI's development and evolution through wider collaboration. > > > > > > > > From email discussions, you've indicated an interest in offering a > > > > generic interface API for Celeborn, which is certainly exciting. > > > > However, I'm concerned that basing a CLI on HTTP API might not fully > > > > align with this vision. Could you provide additional insights into how > > > > you envision the CLI advancing beyond the capabilities of the current > > > > HTTP REST API? > > > > > > > > Based on previous exchanges, the CLI is expected to communicate with > > > > an external cluster manager. Is there an abstraction layer in place to > > > > interface uniformly with various external cluster managers, or is this > > > > something under consideration? > > > > > > > > I'm looking forward to learning more about your perspectives and the > > > > pathway you foresee for the CLI's development. > > > > > > > > regards, > > > > Ethan > >
