Re: [DISCUSS] Time for 0.5.2
+1. I believe that these changes are worthy of a new release. Thanks, Ethan Feng Nicholas Jiang 于2024年9月23日周一 17:15写道： > > Hey, Celeborn community, > > > It has been a while since the 0.5.1 release, and there are some critical > fixes land branch-0.5, for example: > > > [CELEBORN-1496] Differentiate map results with only different > stageAttemptId[1] > [CELEBORN-1506][FOLLOWUP] InFlightRequestTracker should not reset > totalInflightReqs for cleaning up to avoid negative totalInflightReqs for > limitZeroInFlight[2] > > > From my perspective, it’s time to prepare for releasing 0.5.2. > > WDYT? And I’m volunteering to be the release manager if no one has applied. > > [1] > https://github.com/apache/celeborn/commit/b9382a045b0e1c0d04d470d53237b8dee680e2be > [2] > https://github.com/apache/celeborn/commit/1578f6701105fcc61977b13a4c8d866c093f045f > > Regards, > Nicholas Jiang
Re: [VOTE] CIP-11: Supporting Tags in Celeborn
+1(binding) Thanks, Ethan Feng Sanskar Modi 于2024年9月6日周五 17:20写道： > > Hi All, > > Thanks for reviewing the design and providing feedback about the > CIP-11: Supporting Tags in Celeborn [1]. The discussion thread is here [2]. > > I'd like to start a vote for the CIP-11 [3]. The vote will be open for at > least 72 hours unless there is an objection or insufficient votes. > > [1] - > https://docs.google.com/document/d/1aCnxFSKqK687aJS01U0Z_EyHTKhvq1PEk-2I7pI3Mr0/edit#heading=h.989t4btat0zg > [2] - https://lists.apache.org/thread/911d40mqm2vwm1np0z305omwnb2dnsjx > [3] - > https://cwiki.apache.org/confluence/display/CELEBORN/CIP-11+Supporting+Tags+in+Celeborn > > Regards > Sanskar Modi
Re: [DISCUSS] Support Tags In Celeborn Proposal
Hi Sanskar, Thanks for your enthusiasm. Before starting the development, there can be a vote to ensure that the majority of the Celeborn community has reached a consensus. Do you need help to move the design docs to the confluence like other CIPs[0]? Regards, Ethan Feng [0] https://cwiki.apache.org/confluence/display/CELEBORN/Celeborn+Improvement+Proposals Sanskar Modi 于2024年9月2日周一 20:46写道： > > Hi all, > > Thanks for reviewing the doc. I've addressed most of the comments, please > let me know if there are any final comments. If not, I'll start working on > the development. > > Regards, > Sanskar Modi > > On Tue, Aug 20, 2024 at 11:08 PM Sanskar Modi > wrote: > > > Hi all, > > > > I've written up a proposal for supporting tags in Celeborn. You can find > > the proposal here > > <https://docs.google.com/document/d/1aCnxFSKqK687aJS01U0Z_EyHTKhvq1PEk-2I7pI3Mr0/edit#heading=h.989t4btat0zg>. > > Please let me know what you think about this and if we can improve the > > design. > > > > Supporting tags can enable Celeborn to create isolated environments with > > different spec / configs / releases and manage the worker fleet with ease. > > > > Regards, > > Sanskar Modi > >
Re: [Discussion] Disk spills in Celeborn
Hello Aidar, Thank you for sharing your insights on minimizing disk spills attributed to external sorting. I've grasped the concept and am keen to delve deeper into this proposal. Both our MapReduce plugin and TEZ plugin stand to benefit significantly from such an enhancement. I've identified a component called "PartitionFilesSorter" that has the potential to facilitate the sorting of shuffle files directly on disk, bypassing the conventional K-way merge process. This approach holds promise in optimizing our operations. To ensure compatibility across multiple engines, implementing an interface for sorting serialized values becomes imperative. A possible signature for this interface might resemble: int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2) This would allow for flexible and engine-agnostic sorting capabilities. I'm looking forward to seeing the design. Thanks, Ethan Feng Binjie Yang 于2024年8月30日周五 17:27写道： > > Hi Aidar, > This feature sounds great and can solve disk problems in many scenarios. > > I have some small questions about the details. If it's a spill in a > non-shuffle scenario, will celeborn cluster also be responsible for > the corresponding spill data? > > Overall, sort the reduce partition files on the celeborn worker looks > good, looking forward to seeing the overall design and the CIP. > > Thanks, > Binjie Yang > > Aidar Bariev 于2024年8月29日周四 05:39写道： > > > > Hey Celeborn community! > > > > We (Stripe) are discovering the possibility of reducing the amount of disk > > spills due to external sorting on the reducer side and trying to find a > > partial solution to https://issues.apache.org/jira/browse/CELEBORN-1435 > > issue. > > > > Currently, the idea that we have is to sort the ReducePartition file on the > > Celeborn worker at the write time and use K-way merge to further sort the > > data on the reducers. It would still require spilling some data back to > > Celeborn in case the number of ReducePartition files is high and we need > > multiple passes of external merge/sort. > > > > This description might seem pretty generic, but I'm curious to hear > > opinions before starting to draft the CIP! It would be nice to see whether > > there have been previous thoughts and interest in reducing disk usage on > > executors other than the jira ticket above.
Re: [VOTE] CIP-10: Introduce Celeborn Chaos Testing Framework
+1 This will improve our test efficiency. Thanks, Ethan Feng Mridul Muralidharan 于2024年8月3日周六 09:37写道： > > +1 > This will be very useful for hardening Celeborn as we evolve it ! > > Regards, > Mridul > > > > On Fri, Aug 2, 2024 at 9:37 AM Nicholas Jiang > wrote: > > > Hi all, > > > > Thanks for all the feedback about the CIP-10: Introduce Celeborn Chaos > > Testing Framework[1]. The discussion thread is here [2]. > > > > I'd like to start a vote for it. The vote will be open for at least 72 > > hours unless there is an objection or insufficient votes. > > > > Please vote accordingly: > > > > [ ] +1 approve > > [ ] +0 no opinion > > [ ] -1 disapprove (and the reason) > > > > > > [1] > > https://cwiki.apache.org/confluence/display/CELEBORN/CIP-10+Introduce+Celeborn+Chaos+Testing+Framework > > [2] https://lists.apache.org/thread/670qw80wwfflgv3djqg4304xqy9y8l19 > > > > Regards, > > Nicholas Jiang
[ANNOUNCE] Apache Celeborn 0.5.1 available
Hello all, Apache Celeborn community is glad to announce the new release of Apache Celeborn 0.5.1. Celeborn is dedicated to improving the efficiency and elasticity of different map-reduce engines and provides an elastic, highly efficient service for intermediate data including shuffle data, spilled data, result data, etc. Download Link: https://celeborn.apache.org/download/ GitHub Release Tag: - https://github.com/apache/celeborn/releases/tag/v0.5.1 Release Notes: - https://celeborn.apache.org/community/release_notes/release_note_0.5.1 Home Page: https://celeborn.apache.org/ Celeborn Resources: - Issue Management: https://issues.apache.org/jira/projects/CELEBORN - Mailing List: dev@celeborn.apache.org Regards, Ethan Feng On behalf of the Apache Celeborn community
[RESULT][VOTE] Release Apache Celeborn 0.5.1-rc0
Hello, Celeborn community, The vote closes now as 72 hours have passed. The vote PASSES with: (* = binding) +1: Ethan Feng* Nicholas Jiang* Cheng Pan* Jiashu Xiong* There are no 0 or -1 votes. The voting thread: https://lists.apache.org/thread/bo39dq4yp4j2kb6ykgl21vkoxxv1t80t Thanks, Ethan Feng
Re: [VOTE] Release Apache Celeborn 0.5.1-rc0
+1(binding) Thanks for your feedback, I will close this vote thread and announce the results soon since 72 hours have passed. Thanks, Ethan Feng rexxiong 于2024年7月29日周一 10:30写道： > > +1 (binding) > I checked > - Download links are valid. > - git commit hash is correct > - Checksums and signatures are valid. > - Successfully built the binary from the source on MacOs (Java 8) with > Command: build/make-distribution.sh --sbt-enabled --release > > Thanks, > Jiashu Xiong > > Cheng Pan 于2024年7月25日周四 16:40写道： > > > +1 > > > > - links are valid > > - signatured and checksums are good > > - can build from source code > > - Ubuntu 22.04, x86, Java 17 > > - build/make-distribution.sh --sbt-enabled --release > > > > Thanks, > > Cheng Pan > > > > On Thu, Jul 25, 2024 at 4:38 PM Nicholas Jiang > > wrote: > > > > > > +1 (binding) > > > > > > I have checked > > > - Download links are valid. > > > - Checksums and signatures are valid. > > > - Git commit hash is correct > > > - No binary files in the source release > > > - Built binary from source code with command successfully: > > ./build/make-distribution.sh -Pspark-3.5 > > > > > > I have upgraded version to 0.5.1-snapshot in internal test environment a > > week ago. 0.5.1 works well in test environment. > > > > > > Regards, > > > Nicholas Jiang > > > > > > On 2024/07/23 03:38:57 Ethan Feng wrote: > > > > Hello, Celeborn community, > > > > > > > > This is a call for a vote to release Apache Celeborn > > > > 0.5.1-rc0 > > > > > > > > The git tag to be voted upon: > > > > https://github.com/apache/celeborn/releases/tag/v0.5.1-rc0 > > > > > > > > Source and binary artifacts can be found at: > > > > https://dist.apache.org/repos/dist/dev/celeborn/v0.5.1-rc0 > > > > > > > > The git commit hash: > > > > 85297ca64c0973f19b15831031859c3104f0db5b > > > > > > > > The staging repo: > > > > > > https://repository.apache.org/content/repositories/orgapacheceleborn-1081 > > > > > > > > The fingerprint of the PGP key release artifacts is signed with: > > > > FCF20BB29C7BEFDF58F998F76392F71F37356FA0 > > > > > > > > My public key to verify signatures can be found in: > > > > https://dist.apache.org/repos/dist/release/celeborn/KEYS > > > > > > > > The vote will be open for at least 72 hours or until the necessary > > > > number of votes are reached. > > > > > > > > Please vote accordingly: > > > > > > > > [ ] +1 approve > > > > [ ] +0 no opinion > > > > [ ] -1 disapprove (and the reason) > > > > > > > > Steps to validate the release: > > > > https://www.apache.org/info/verification.html > > > > > > > > * Download links, checksums, and PGP signatures are valid. > > > > * Source code distributions have correct names matching the current > > release. > > > > * LICENSE and NOTICE files are correct. > > > > * All files have license headers if necessary. > > > > * No unlicensed compiled archives bundled in the source archive. > > > > * The source tarball matches the git tag. > > > > * Build from source is successful. > > > > > > > > Regards, > > > > Ethan Feng > > > > > >
[VOTE] Release Apache Celeborn 0.5.1-rc0
Hello, Celeborn community, This is a call for a vote to release Apache Celeborn 0.5.1-rc0 The git tag to be voted upon: https://github.com/apache/celeborn/releases/tag/v0.5.1-rc0 Source and binary artifacts can be found at: https://dist.apache.org/repos/dist/dev/celeborn/v0.5.1-rc0 The git commit hash: 85297ca64c0973f19b15831031859c3104f0db5b The staging repo: https://repository.apache.org/content/repositories/orgapacheceleborn-1081 The fingerprint of the PGP key release artifacts is signed with: FCF20BB29C7BEFDF58F998F76392F71F37356FA0 My public key to verify signatures can be found in: https://dist.apache.org/repos/dist/release/celeborn/KEYS The vote will be open for at least 72 hours or until the necessary number of votes are reached. Please vote accordingly: [ ] +1 approve [ ] +0 no opinion [ ] -1 disapprove (and the reason) Steps to validate the release: https://www.apache.org/info/verification.html * Download links, checksums, and PGP signatures are valid. * Source code distributions have correct names matching the current release. * LICENSE and NOTICE files are correct. * All files have license headers if necessary. * No unlicensed compiled archives bundled in the source archive. * The source tarball matches the git tag. * Build from source is successful. Regards, Ethan Feng
Re: [DISCUSS] CIP-10: Introduce Celeborn Chaos Testing Framework
Hi Nicholas, Thanks for reaching out. More and more Celeborn users are moving to cloud-native environments. Is there any plan to support testing Celeborn cluster on the K8s environment? Best, Ethan Feng Nicholas Jiang 于2024年7月3日周三 05:21写道： > > Hi all, > > I would like to start a discussion on CIP-10: Introduce Celeborn Chaos > Testing Framework[1]. > > A chaos testing framework is designed to simulate unpredictable and adverse > conditions in distributed systems to validate their robustness and > resilience. This proposal aims to simulate various anomalies and test the > stability of Celeborn in distributed environments via chaos testing. > > Looking forward to everyone's feedback and suggestions. Thank you! > > [1] > https://cwiki.apache.org/confluence/display/CELEBORN/CIP-10+Introduce+Celeborn+Chaos+Testing+Framework > > Regards, > Nicholas Jiang
Re: [VOTE] CIP-9: Celeborn RESTful API Refine
+1 Thanks, Ethan Feng angers zhu 于2024年7月3日周三 10:20写道： > > +1 > > Regards > Angerszh > > Keyong Zhou 于2024年7月3日周三 09:25写道： > > > +1 > > > > Regards, > > Keyong Zhou > > > > Fei Wang 于2024年7月3日周三 02:07写道： > > > > > Hi all, > > > > > > Thanks for all the feedback about the CIP-9 Celeborn RESTful API Refine > > > [1]. > > > The discussion thread is here [2]. > > > > > > I'd like to start a vote for it. The vote will be open for at least 72 > > > hours unless there is an objection or insufficient votes. > > > > > > [1] > > > > > > > > https://docs.google.com/document/d/1LV2vV-w3XtlbJj2Vi4J77mt4IYCr40-8A_JncZLsHqs/edit?usp=sharing > > > [2] https://lists.apache.org/thread/mng0pxst0z4gc9gs7mc1frz4pzpk70jb > > > > > > Best Regards, > > > Fei Wang > > > > >
Re: Re:[DISCUSS] Celeborn RESTful API Refine Proposal
Hi feiwang,
Thanks for this proposal. I agree with the versioned REST APIs.
Thanks,
Ethan Feng
Ethan Feng 于2024年7月2日周二 17:58写道：
>
> Fei Wang 于2024年7月2日周二 13:04写道：
>
> >
> > Hi Keyong,
> >
> > Thanks very much for your suggestions.
> >
> > I have renamed the docs to 'CIP-9 Celeborn RESTful API Refine' and will
> > start a vote thread later.
> >
> > Thank you very much.
> >
> > Regards,
> > Fei Wang
> >
> > On 2024/07/02 04:17:34 Keyong Zhou wrote:
> > > Hi Fei,
> > >
> > > Sorry for the late reply, I reviewed the design doc and it looks good to
> > > me
> > > :)
> > >
> > > In the doc you mentioned CIP-7 Celeborn CLI[1] that relies on the rest
> > > API,
> > > since your design
> > > will reserve the current API, I think there will be no conflicts.
> > >
> > > I suggest to create a CIP for this proposal and starts a vote on it,
> > > like[2][3].
> > >
> > > Regards,
> > > Keyong Zhou
> > >
> > > [1]
> > > https://cwiki.apache.org/confluence/display/CELEBORN/CIP-7+Celeborn+CLI
> > > [2] https://lists.apache.org/thread/xjh8z2kszq0kwj5bdz2bh3b1sotv593p
> > > [3] https://lists.apache.org/thread/bx58h25poypq0znolkb8vlhop4bw1x81
> > >
> > > Fei Wang 于2024年7月2日周二 04:53写道：
> > >
> > > > Hi celeborn community,
> > > > I hope this message finds you well.
> > > >
> > > > I would like to extend my gratitude to those who have taken the time to
> > > > review and discuss for this proposal. Your insights and feedback are
> > > > invaluable to the progression of this project.
> > > >
> > > > As we have not received further discussions for some time, I believe it
> > > > is
> > > > appropriate to move forward with the next step in the process.
> > > >
> > > > Best Regards,
> > > > Fei Wang
> > > >
> > > > On 2024/06/25 19:18:27 Fei Wang wrote:
> > > > > Hi,
> > > > >
> > > > > Thanks Nicholas for the comments.
> > > > >
> > > > > > 1. Could you summary all /api/v1 interfaces in Public Interfaces
> > > > section? Meanwhile, could you also the definition of the parameter and
> > > > return type class like the fields of POJO?
> > > > >
> > > > > I have updated the docs and complete the parameters and response POJO.
> > > > >
> > > > > > 2. Could some interfaces merged into one interface like
> > > > /${version}/workers/lost, /${version}/workers/excluded and
> > > > /${version}/workers/shutdown? Should the refined REST API be mapping to
> > > > origin interface one by one?
> > > > >
> > > > > Thanks we can merge these apis into `/${version}/workers`. The
> > > > > response
> > > > POJO.
> > > > > Name Type
> > > > > workers List[WorkerData]
> > > > > lostWorkers List[WorkerData]
> > > > > excludedWorkers List[WorkerData]
> > > > > shutdownWorkers List[WorkerData]
> > > > > decommissionWorkers List[WorkerData]
> > > > >
> > > > > And I will mapping the the api one by one.
> > > > >
> > > > >
> > > > > > 3. Could this migration plan describe more detail? For example, the
> > > > origin interfaces returns string, but the refined REST API returns the
> > > > POJO? How does the user migrate the REST API?
> > > > >
> > > > > The migration of a REST API primarily involves mapping the old API to
> > > > the new API. Previously, the old API returned a string, whereas the new
> > > > API
> > > > returns a POJO (Plain Old Java Object) that includes all the fields
> > > > relevant to the content of the previous string response. Therefore, the
> > > > content of the new API's response is essentially consistent with the
> > > > previous response results.
> > > > > In the migration documentation, I will detail the API mappings as well
> > > > as the fields in the new response.
> > > > >
> > > > >
> > > > > > 4. Some interfaces like /${version}/exit do not mentation the HTTP
> > > > method? Could you check all the HTTP method of refined REST API?
>
Re: Re:[DISCUSS] Celeborn RESTful API Refine Proposal
Fei Wang 于2024年7月2日周二 13:04写道：
>
> Hi Keyong,
>
> Thanks very much for your suggestions.
>
> I have renamed the docs to 'CIP-9 Celeborn RESTful API Refine' and will start
> a vote thread later.
>
> Thank you very much.
>
> Regards,
> Fei Wang
>
> On 2024/07/02 04:17:34 Keyong Zhou wrote:
> > Hi Fei,
> >
> > Sorry for the late reply, I reviewed the design doc and it looks good to me
> > :)
> >
> > In the doc you mentioned CIP-7 Celeborn CLI[1] that relies on the rest API,
> > since your design
> > will reserve the current API, I think there will be no conflicts.
> >
> > I suggest to create a CIP for this proposal and starts a vote on it,
> > like[2][3].
> >
> > Regards,
> > Keyong Zhou
> >
> > [1] https://cwiki.apache.org/confluence/display/CELEBORN/CIP-7+Celeborn+CLI
> > [2] https://lists.apache.org/thread/xjh8z2kszq0kwj5bdz2bh3b1sotv593p
> > [3] https://lists.apache.org/thread/bx58h25poypq0znolkb8vlhop4bw1x81
> >
> > Fei Wang 于2024年7月2日周二 04:53写道：
> >
> > > Hi celeborn community,
> > > I hope this message finds you well.
> > >
> > > I would like to extend my gratitude to those who have taken the time to
> > > review and discuss for this proposal. Your insights and feedback are
> > > invaluable to the progression of this project.
> > >
> > > As we have not received further discussions for some time, I believe it is
> > > appropriate to move forward with the next step in the process.
> > >
> > > Best Regards,
> > > Fei Wang
> > >
> > > On 2024/06/25 19:18:27 Fei Wang wrote:
> > > > Hi,
> > > >
> > > > Thanks Nicholas for the comments.
> > > >
> > > > > 1. Could you summary all /api/v1 interfaces in Public Interfaces
> > > section? Meanwhile, could you also the definition of the parameter and
> > > return type class like the fields of POJO?
> > > >
> > > > I have updated the docs and complete the parameters and response POJO.
> > > >
> > > > > 2. Could some interfaces merged into one interface like
> > > /${version}/workers/lost, /${version}/workers/excluded and
> > > /${version}/workers/shutdown? Should the refined REST API be mapping to
> > > origin interface one by one?
> > > >
> > > > Thanks we can merge these apis into `/${version}/workers`. The response
> > > POJO.
> > > > Name Type
> > > > workers List[WorkerData]
> > > > lostWorkers List[WorkerData]
> > > > excludedWorkers List[WorkerData]
> > > > shutdownWorkers List[WorkerData]
> > > > decommissionWorkers List[WorkerData]
> > > >
> > > > And I will mapping the the api one by one.
> > > >
> > > >
> > > > > 3. Could this migration plan describe more detail? For example, the
> > > origin interfaces returns string, but the refined REST API returns the
> > > POJO? How does the user migrate the REST API?
> > > >
> > > > The migration of a REST API primarily involves mapping the old API to
> > > the new API. Previously, the old API returned a string, whereas the new
> > > API
> > > returns a POJO (Plain Old Java Object) that includes all the fields
> > > relevant to the content of the previous string response. Therefore, the
> > > content of the new API's response is essentially consistent with the
> > > previous response results.
> > > > In the migration documentation, I will detail the API mappings as well
> > > as the fields in the new response.
> > > >
> > > >
> > > > > 4. Some interfaces like /${version}/exit do not mentation the HTTP
> > > method? Could you check all the HTTP method of refined REST API?
> > > Meanwhile,
> > > is there any standard or pattern of the naming for path? For example,
> > > /${version}/workers/events not only list the event info for Get method,
> > > but
> > > also supports sending event for POST method without operation name in
> > > path.
> > > >
> > > > Thanks for the reminder, I have added the method for params for all the
> > > APIs.
> > > >
> > > > Using different HTTP methods to correspond to different actions on the
> > > same API endpoint is a common practice in RESTful design. This approach is
> > > in line with the principles of REST, where the resource path remains
> > > consistent while the HTTP method indicates the intended operation. And I
> > > have observed a similar design pattern in Apache Kyuubi Project.
> > > >
> > > >
> > > >
> > > > > 5. /${version}/conf/dynamic uses three parameters without any POJO
> > > parameter type class, but /${version}/workers/events uses
> > > SendWorkerEventRequest as parameter type. Could this parameter type be
> > > unified to POJO?
> > > >
> > > > The api `/${version}/conf/dynamic` method is GET, request parameters
> > > should be fine.
> > > >
> > > >
> > > > Thanks,
> > > > Fei Wang
> > > >
> > > > On 2024/06/25 17:22:51 Nicholas wrote:
> > > > > Hi turboFei,
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Thanks for driving the proposal of RESTful API Refine. I have some
> > > questions about this proposal:
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > 1. Could you summary all /api/v1 interfaces in
[NOTICE] Hotfix about potential performance drop in release 0.5.0
Hello users, We found an issue that may cause a performance drop in the latest release 0.5.0. A new patch release will be ready as soon as possible. Suppose you encountered a performance drop in Celeborn Worker after upgrading to the latest version. There is a PR[0] to rectify the situation. Regards, Ethan Feng On behalf of the Apache Celeborn community [0] https://github.com/apache/celeborn/pull/2591
[ANNOUNCE] Apache Celeborn 0.5.0 available
Hello all, Apache Celeborn community is glad to announce the new release of Apache Celeborn 0.5.0. Celeborn is dedicated to improving the efficiency and elasticity of different map-reduce engines and provides an elastic, highly efficient service for intermediate data including shuffle data, spilled data, result data, etc. Download Link: https://celeborn.apache.org/download/ GitHub Release Tag: - https://github.com/apache/celeborn/releases/tag/v0.5.0 Release Notes: - https://celeborn.apache.org/community/release_notes/release_note_0.5.0 Home Page: https://celeborn.apache.org/ Celeborn Resources: - Issue Management: https://issues.apache.org/jira/projects/CELEBORN - Mailing List: dev@celeborn.apache.org Regards, Ethan Feng On behalf of the Apache Celeborn community
Re: [VOTE] Release Apache Celeborn 0.5.0-rc3
Hi Mridul,
The Celeborn dashboard will not be included in this version
release. Therefore, it should not pose any issues for us at this time.
After some investigation, I discovered that our CI tests are based
on a fixed list of projects, which is why this problem was not
detected earlier.
Thank you for bringing this to our attention. To bypass the lint
exceptions for unit tests, please use the following command:
```
SPARK_BINARY_VERSION=3.1
SPARK_MAJOR_VERSION=${SPARK_BINARY_VERSION%%.*}
PROFILES="-Pgoogle-mirror,spark-3.1"
TEST_MODULES="client-spark/common,client-spark/spark-${SPARK_MAJOR_VERSION},client-spark/spark-${SPARK_MAJOR_VERSION}-shaded,tests/spark-it"
build/mvn $PROFILES -pl $TEST_MODULES -am clean install -DskipTests
build/mvn $PROFILES -pl $TEST_MODULES
-Dspark.shuffle.sort.io.plugin.class=org.apache.spark.shuffle.sort.io.LocalDiskShuffleDataIO
test
```
Additionally, I have created a Jira ticket[0] to track this lint exception.
Thanks,
Ethan Feng
[0] https://issues.apache.org/jira/browse/CELEBORN-1480
Mridul Muralidharan 于2024年6月24日周一 15:35写道：
>
> Forgot to update here.
>
> Signatures, digests, etc check out fine.
> Checked out tag and build/tested with "-Pspark3.1"
>
> I keep getting the following error:
>
> - metrics/prometheus *** FAILED ***
> 200 did not equal 404 (ApiBaseResourceSuite.scala:90)
> - metrics/json *** FAILED ***
> 200 did not equal 404 (ApiBaseResourceSuite.scala:96)
>
> In unit-tests.log, I have this [1]
>
> After explicitly ignoring these two tests, I then ran into lint issues with
> "web" submodule - and could not get around to debugging the issue (I
> installed pnpm - which does not seem to be called out in readme).
>
> Thanks,
> Mridul
>
> [1]
> 24/06/24 01:57:33,545 ERROR [ScalaTest-main-running-DiscoverySuite]
> MetricsConfig: Error loading configuration file
> file:/home/mridul/work/apache/vote/celeborn/service/target/celeborn-service_2.12-0.5.0-tests.jar!/metrics-api.properties
> java.io.FileNotFoundException:
> file:/home/mridul/work/apache/vote/celeborn/service/target/celeborn-service_2.12-0.5.0-tests.jar!/metrics-api.properties
> (No such file or directory)
> at java.io.FileInputStream.open0(Native Method)
> at java.io.FileInputStream.open(FileInputStream.java:195)
> at java.io.FileInputStream.(FileInputStream.java:138)
> at java.io.FileInputStream.(FileInputStream.java:93)
> at
> org.apache.celeborn.common.metrics.MetricsConfig.loadPropertiesFromFile(MetricsConfig.scala:95)
> at
> org.apache.celeborn.common.metrics.MetricsConfig.initialize(MetricsConfig.scala:50)
> at
> org.apache.celeborn.common.metrics.MetricsSystem.(MetricsSystem.scala:53)
> at
> org.apache.celeborn.common.metrics.MetricsSystem$.createMetricsSystem(MetricsSystem.scala:197)
> at
> org.apache.celeborn.service.deploy.master.Master.(Master.scala:66)
> at
> org.apache.celeborn.service.deploy.master.http.api.ApiMasterResourceSuite.beforeAll(ApiMasterResourceSuite.scala:54)
> at
> org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
> at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
> at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
> at
> org.apache.celeborn.server.common.http.ApiBaseResourceSuite.run(ApiBaseResourceSuite.scala:23)
> at org.scalatest.Suite.callExecuteOnSuite$1(Suite.scala:1178)
> at org.scalatest.Suite.$anonfun$runNestedSuites$1(Suite.scala:1225)
> at
> scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
> at
> scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
> at
> scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
> at org.scalatest.Suite.runNestedSuites(Suite.scala:1223)
> at org.scalatest.Suite.runNestedSuites$(Suite.scala:1156)
> at
> org.scalatest.tools.DiscoverySuite.runNestedSuites(DiscoverySuite.scala:30)
> at org.scalatest.Suite.run(Suite.scala:)
> at org.scalatest.Suite.run$(Suite.scala:1096)
> at org.scalatest.tools.DiscoverySuite.run(DiscoverySuite.scala:30)
> at org.scalatest.tools.SuiteRunner.run(SuiteRunner.scala:47)
> at
> org.scalatest.tools.Runner$.$anonfun$doRunRunRunDaDoRunRun$13(Runner.scala:1321)
> at
> org.scalatest.tools.Runner$.$anonfun$doRunRunRunDaDoRunRun$13$adapted(Runner.scala:1315)
> at scala.collection.immutable.List.foreach(List.scala:392)
> at
> org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:1315)
> at
> org.scalatest.tools.Runner$.$anonfun$runOptional
[RESULT][VOTE] Release Apache Celeborn 0.5.0-rc3
Hello, Celeborn community, The vote closes now as 72 hours have passed. The vote PASSES with: (* = binding) +1: Ethan Feng* Keyong Zhou* Nicholas Jiang Rexxiong* Cheng Pan* Fu Chen* Kerwin Zhang* There are no 0 or -1 votes. The voting thread: https://lists.apache.org/thread/vpboxros8bhlt2tmvnc4b3onzczlczc5 Thanks, Ethan Feng
Re: [VOTE] Release Apache Celeborn 0.5.0-rc3
Thanks for your feedback, I will close this vote thread and announce the results soon since 72 hours have passed. Ethan Feng. kerwin zhang 于2024年6月24日周一 15:18写道： > > +1 (binding) > > I checked > - git commit hash is correct. > - links are valid. > - signatures are good. > ``` > gpg --verify apache-celeborn-0.5.0-bin.tgz.asc apache-celeborn-0.5.0-bin.tgz > gpg --verify apache-celeborn-0.5.0-source.tgz.asc > apache-celeborn-0.5.0-source.tgz > ``` > - checksums are good. > ``` > shasum -a 512 apache-celeborn-0.5.0-bin.tgz > shasum -a 512 apache-celeborn-0.5.0-source.tgz > ``` > > - build success from source code (macOS). > > ``` > ./build/make-distribution.sh -Pspark-3.4 > ``` > > Thanks, > Kerwin Zhang > > Fu Chen 于2024年6月24日周一 14:46写道： > > > > +1 > > > > I checked > > - download links are valid. > > - git commit hash is correct. > > - no binary files in the source release. > > - build success from source code (JDK 17). > > ``` > > ./build/make-distribution.sh --sbt-enabled -Pspark-3.5 > > ``` > > > > Cheng Pan 于2024年6月24日周一 14:09写道： > > > > > > +1 (binding) > > > > > > - All links are valid. > > > - LICENSE/NOTICE looks fine. > > > - CI result looks great. > > > - No known blockers. > > > > > > Thanks, > > > Cheng Pan > > > > > > > > > > On Jun 19, 2024, at 12:45, Ethan Feng wrote: > > > > > > > > Hello, Celeborn community, > > > > > > > > This is a call for a vote to release Apache Celeborn > > > > 0.5.0-rc3 > > > > > > > > The git tag to be voted upon: > > > > https://github.com/apache/celeborn/releases/tag/v0.5.0-rc3 > > > > > > > > Source and binary artifacts can be found at: > > > > https://dist.apache.org/repos/dist/dev/celeborn/v0.5.0-rc3 > > > > > > > > The git commit hash: > > > > 048ef207359113247bff05dcc203c70021ccfa10 > > > > > > > > The staging repo: > > > > https://repository.apache.org/content/repositories/orgapacheceleborn-1076/ > > > > > > > > The fingerprint of the PGP key release artifacts is signed with: > > > > FCF20BB29C7BEFDF58F998F76392F71F37356FA0 > > > > > > > > My public key to verify signatures can be found in: > > > > https://dist.apache.org/repos/dist/release/celeborn/KEYS > > > > > > > > The vote will be open for at least 72 hours or until the necessary > > > > number of votes are reached. > > > > > > > > Please vote accordingly: > > > > > > > > [ ] +1 approve > > > > [ ] +0 no opinion > > > > [ ] -1 disapprove (and the reason) > > > > > > > > Steps to validate the release: > > > > https://www.apache.org/info/verification.html > > > > > > > > * Download links, checksums, and PGP signatures are valid. > > > > * Source code distributions have correct names matching the current > > > > release. > > > > * LICENSE and NOTICE files are correct. > > > > * All files have license headers if necessary. > > > > * No unlicensed compiled archives bundled in the source archive. > > > > * The source tarball matches the git tag. > > > > * Build from source is successful. > > > > > > > > There are additional tests: > > > > * Performance test no regression > > > > 1 TB TPC-DS, 0.5.0 VS 0.4.1 : 2042(s) VS 2050(s) > > > > 1.1 TB pure shuffle, 0.5.0 VS 0.4.1 : 11.8min vs 11.8min > > > > > > > > * Result correctness test passed > > > > 1TB TPC-DS runs concurrently, the results are identical. > > > > > > > > * Usability test passed > > > > Rolling upgrade from version 0.4.1 to 0.5.0 succeed. > > > > The metrics system works as expected. > > > > > > > > * Stability test passed > > > > Random worker failures, Celeborn works as expected. > > > > Random master failures, Celeborn works as expected. > > > > Master meta corrupted, Celeborn works as expected. > > > > > > > > * Compatibility test passed > > > > The Celeborn server version of 0.5.0 works fine with the Celeborn > > > > client 0.4.1. > > > > > > > > * Grafana dashboard layout checked > > > > > > > > > > > > Regards, > > > > Ethan Feng > > >
[DISCUSS] CIP-8 Refactor PartitionDataWriter
Hi all, I have written up a proposal about refactoring partition data writer. https://cwiki.apache.org/confluence/display/CELEBORN/CIP-8+Refactor+Partition+Data+Writer Please let me know if you have any comments or questions. TLDR makes partition data writers free of implementation about different storage tiers and extendable. Thanks, Ethan
[VOTE] Release Apache Celeborn 0.5.0-rc3
Hello, Celeborn community, This is a call for a vote to release Apache Celeborn 0.5.0-rc3 The git tag to be voted upon: https://github.com/apache/celeborn/releases/tag/v0.5.0-rc3 Source and binary artifacts can be found at: https://dist.apache.org/repos/dist/dev/celeborn/v0.5.0-rc3 The git commit hash: 048ef207359113247bff05dcc203c70021ccfa10 The staging repo: https://repository.apache.org/content/repositories/orgapacheceleborn-1076/ The fingerprint of the PGP key release artifacts is signed with: FCF20BB29C7BEFDF58F998F76392F71F37356FA0 My public key to verify signatures can be found in: https://dist.apache.org/repos/dist/release/celeborn/KEYS The vote will be open for at least 72 hours or until the necessary number of votes are reached. Please vote accordingly: [ ] +1 approve [ ] +0 no opinion [ ] -1 disapprove (and the reason) Steps to validate the release: https://www.apache.org/info/verification.html * Download links, checksums, and PGP signatures are valid. * Source code distributions have correct names matching the current release. * LICENSE and NOTICE files are correct. * All files have license headers if necessary. * No unlicensed compiled archives bundled in the source archive. * The source tarball matches the git tag. * Build from source is successful. There are additional tests: * Performance test no regression 1 TB TPC-DS, 0.5.0 VS 0.4.1 : 2042(s) VS 2050(s) 1.1 TB pure shuffle, 0.5.0 VS 0.4.1 : 11.8min vs 11.8min * Result correctness test passed 1TB TPC-DS runs concurrently, the results are identical. * Usability test passed Rolling upgrade from version 0.4.1 to 0.5.0 succeed. The metrics system works as expected. * Stability test passed Random worker failures, Celeborn works as expected. Random master failures, Celeborn works as expected. Master meta corrupted, Celeborn works as expected. * Compatibility test passed The Celeborn server version of 0.5.0 works fine with the Celeborn client 0.4.1. * Grafana dashboard layout checked Regards, Ethan Feng
Re: [VOTE] Release Apache Celeborn 0.5.0-rc2
This vote is canceled due to recent bug fixes. Thanks, Ethan Feng Nicholas Jiang 于2024年6月12日周三 16:41写道： > > +1 (non-binding) > > I have checked: > > - Download links are valid. > - git commit hash is correct > - Checksums and signatures are valid. > - No binary files in the source release > - Successfully built binary from source via ./build/make-distribution.sh > -Pspark-3.5/flink-1.19/mr > > Meanwhile, I have upgraded internal version to 0.5.0 in our production > environment. > > Regards, > Nicholas Jiang > > On 2024/06/12 05:56:07 Ethan Feng wrote: > > Hello, Celeborn community, > > > > This is a call for a vote to release Apache Celeborn > > 0.5.0-rc2 > > > > The git tag to be voted upon: > > https://github.com/apache/celeborn/releases/tag/v0.5.0-rc2 > > > > Source and binary artifacts can be found at: > > https://dist.apache.org/repos/dist/dev/celeborn/v0.5.0-rc2 > > > > The git commit hash: > > 68c503eb0023e274f8ae09bf4c2687f6a0c01a25 > > > > The staging repo: > > https://repository.apache.org/content/repositories/orgapacheceleborn-1075/ > > > > The fingerprint of the PGP key release artifacts is signed with: > > FCF20BB29C7BEFDF58F998F76392F71F37356FA0 > > > > My public key to verify signatures can be found in: > > https://dist.apache.org/repos/dist/release/celeborn/KEYS > > > > The vote will be open for at least 72 hours or until the necessary > > number of votes are reached. > > > > Please vote accordingly: > > > > [ ] +1 approve > > [ ] +0 no opinion > > [ ] -1 disapprove (and the reason) > > > > Steps to validate the release: > > https://www.apache.org/info/verification.html > > > > * Download links, checksums, and PGP signatures are valid. > > * Source code distributions have correct names matching the current release. > > * LICENSE and NOTICE files are correct. > > * All files have license headers if necessary. > > * No unlicensed compiled archives bundled in the source archive. > > * The source tarball matches the git tag. > > * Build from source is successful. > > > > There are additional tests: > > * Performance test no regression > > 1 TB TPC-DS, 0.5.0 VS 0.4.1 : 2042(s) VS 2050(s) > > 1.1 TB pure shuffle, 0.5.0 VS 0.4.1 : 11.8min vs 11.8min > > > > * Result correctness test passed > > 1TB TPC-DS runs concurrently, the results are identical. > > > > * Usability test passed > > Rolling upgrade from version 0.4.1 to 0.5.0 succeed. > > The metrics system works as expected. > > > > * Stability test passed > > Random worker failures, Celeborn works as expected. > > Random master failures, Celeborn works as expected. > > Master meta corrupted, Celeborn works as expected. > > > > * Compatibility test passed > > The Celeborn server version of 0.5.0 works fine with the Celeborn client > > 0.4.1. > > > > > > Regards, > > Ethan Feng > >
Re: Re:Re:[VOTE] CIP-6: Support Flink hybrid shuffle integration with Apache Celeborn
+1(binding) I think this CIP would bring performance benefits to Flink users. Thanks, Ethan Nicholas 于2024年6月14日周五 14:50写道： > > +1(non-binding). Sorry for mistake of my non-binding. > > > > > Regards, > > Nicholas Jiang > > > > > At 2024-06-14 14:38:42, "Nicholas" wrote: > > +1(binding). > Regards, > Nicholas Jiang > > > At 2024-06-14 14:05:56, "Nicholas Jiang" wrote: > >+1(non-binding) > > > > > > > > > >Regards, > > > >Nicholas Jiang > > > > > > > > > >At 2024-06-14 11:36:17, "Yuxin Tan" wrote: > >>Hi all, > >> > >>Thanks for all the feedback about the CIP-6: Support Flink > >>hybrid shuffle integration with Apache Celeborn[1]. > >>The discussion thread is here [2]. > >> > >>I'd like to start a vote for it. The vote will be open for at least > >>72 hours unless there is an objection or insufficient votes. > >> > >>[1] > >>https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn > >>[2] https://lists.apache.org/thread/55mwmfsxwprzf5l80so9t2cpny82l4nx > >> > >>Best, > >>Yuxin
Re: Re: [VOTE] Contrinute Apache Celeborn CLI
Hi Aravind, Thanks for your detailed response and insight into the CLI implementation for Celeborn. I appreciate the explanation regarding the choice to use Java/Scala for the CLI. I agree that aligning with the prevalent use of Java/Scala within the Celeborn community is the right approach. Your proposed use cases for the CLI functionalities are comprehensive and interesting. I appreciate your proactive approach in considering various cluster management systems and the need for an abstraction layer to interface uniformly. +1 for the current proposal. Looking forward to the CLI evolution. regards, Ethan Aravind Patnam 于2024年6月13日周四 05:35写道： > > Hi Ethan, > > Thanks for your comments! > > Regarding using Java/Scala for the CLI, I am fine with this. I had believed > that using Python would be an easier/simpler implementation given that many > CLI's are implemented in Python, but the points you make are fair. Most of > the Celeborn community uses Java/Scala, so this would be more beneficial > for the development and evolution of the CLI. > > Yes, I think the CLI should contain capabilities beyond the HTTP endpoints > Celeborn exposes. The Celeborn HTTP endpoints work great for application > specific use cases, such as finding the applications or shuffles on a > particular worker, however it would not work for situations in which we > would need information on the cluster itself. For example, we use K8s and > these are use cases internally I can foresee that require communication > with an external cluster manager: > >- Retrieve all pods running masters/workers and their statuses >- Manually evict an Celeborn unhealthy pod >- SSH into various different Celeborn pods >- Manage ACLs of the cluster >- Manually restart pods >- Wipe Ratis storage if state is messed up >- Wipe shuffle directories if state is messed up >- Adding/removing new nodes into our node pool >- Perform any other manual arbitrary function on a Celeborn pod > > > These are just a few of the use cases I can think of, but I am sure more > will arise as more users adopt Celeborn :) > > Given that users will have various different cluster managers, I think as I > mentioned before there should be an abstraction layer present that exposes > different operations. Based on the cluster manager the user is using, the > user can implement their specific logic. We can have a few default ones > included (e.g. Kubernetes). > > Hope this answers your questions, let me know if you have any more > questions! > > Thanks, > Aravind > > On Tue, Jun 11, 2024 at 11:57 PM Ethan Feng wrote: > > > Hi Aravind, > > > > I hope this message finds you well. I wanted to express my > > appreciation for the energy and creativity you've invested in the > > Celeborn project; the proposal you submitted is intriguing. > > > > I apologize for the delayed feedback on your proposal — it took me a > > bit longer to get to it than anticipated. After reviewing it, I have a > > couple of inquiries that I'd like to discuss in order to gain a > > clearer understanding: > > > > I observed that you're planning to implement the CLI in Python. Could > > you elaborate on the choice behind not leveraging the Java stack for > > this purpose? The Java ecosystem already includes mature tools such as > > "commons-cli" or "Scala CLI," which are capable of facilitating CLI > > tool development. Given the prevalent familiarity with the Java stack > > within our community, I believe leveraging it could accelerate the > > CLI's development and evolution through wider collaboration. > > > > From email discussions, you've indicated an interest in offering a > > generic interface API for Celeborn, which is certainly exciting. > > However, I'm concerned that basing a CLI on HTTP API might not fully > > align with this vision. Could you provide additional insights into how > > you envision the CLI advancing beyond the capabilities of the current > > HTTP REST API? > > > > Based on previous exchanges, the CLI is expected to communicate with > > an external cluster manager. Is there an abstraction layer in place to > > interface uniformly with various external cluster managers, or is this > > something under consideration? > > > > I'm looking forward to learning more about your perspectives and the > > pathway you foresee for the CLI's development. > > > > regards, > > Ethan > > > > Mridul Muralidharan 于2024年6月12日周三 14:36写道： > > > > > > +1 > > > > > > Regards, > > > Mridul >
Re: Re: [VOTE] Contrinute Apache Celeborn CLI
Hi Aravind, I hope this message finds you well. I wanted to express my appreciation for the energy and creativity you've invested in the Celeborn project; the proposal you submitted is intriguing. I apologize for the delayed feedback on your proposal — it took me a bit longer to get to it than anticipated. After reviewing it, I have a couple of inquiries that I'd like to discuss in order to gain a clearer understanding: I observed that you're planning to implement the CLI in Python. Could you elaborate on the choice behind not leveraging the Java stack for this purpose? The Java ecosystem already includes mature tools such as "commons-cli" or "Scala CLI," which are capable of facilitating CLI tool development. Given the prevalent familiarity with the Java stack within our community, I believe leveraging it could accelerate the CLI's development and evolution through wider collaboration. >From email discussions, you've indicated an interest in offering a generic interface API for Celeborn, which is certainly exciting. However, I'm concerned that basing a CLI on HTTP API might not fully align with this vision. Could you provide additional insights into how you envision the CLI advancing beyond the capabilities of the current HTTP REST API? Based on previous exchanges, the CLI is expected to communicate with an external cluster manager. Is there an abstraction layer in place to interface uniformly with various external cluster managers, or is this something under consideration? I'm looking forward to learning more about your perspectives and the pathway you foresee for the CLI's development. regards, Ethan Mridul Muralidharan 于2024年6月12日周三 14:36写道： > > +1 > > Regards, > Mridul > > > On Wed, Jun 12, 2024 at 1:08 AM Shaoyun Chen wrote: > > > +1 > > > > Keyong Zhou 于2024年6月12日周三 13:47写道： > > > > > > +1 > > > > > > Thanks for the proposal! > > > > > > Regards, > > > Keyong Zhou > > > > > > Nicholas Jiang 于2024年6月12日周三 13:02写道： > > > > > > > +1. Looking forward to Celeborn CLI. > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > Nicholas Jiang > > > > > > > > > > > > At 2024-06-12 12:26:34, "Aravind Patnam" wrote: > > > > >Hi all, > > > > > > > > > >Sorry, this is the correct link to the Celeborn CLI CIP > > > > >< > > > > > > https://cwiki.apache.org/confluence/display/CELEBORN/CIP+7+-+Celeborn+CLI> > > > > >. > > > > > > > > > >Thanks, > > > > >Aravind > > > > > > > > > >On Tue, Jun 11, 2024 at 9:24 PM Aravind Patnam > > > > wrote: > > > > > > > > > >> Hi all, > > > > >> > > > > >> This is a call to vote to contribute the Celeborn CLI CIP > > > > >> < > > > > > > https://cwiki.apache.org/confluence/display/CELEBORN/Celeborn+Improvement+Proposals > > > > > > > to > > > > >> Apache Celeborn. > > > > >> > > > > >> Please do vote accordingly: > > > > >> [ ] +1 approve > > > > >> [ ] +0 no opinion > > > > >> [ ] -1 disapprove (and the reason) > > > > >> > > > > >> Thanks once again!! > > > > >> > > > > >> Aravind > > > > >> > > > > > > > > > > > > > > >-- > > > > >Aravind K. Patnam > > > > > >
[VOTE] Release Apache Celeborn 0.5.0-rc2
Hello, Celeborn community, This is a call for a vote to release Apache Celeborn 0.5.0-rc2 The git tag to be voted upon: https://github.com/apache/celeborn/releases/tag/v0.5.0-rc2 Source and binary artifacts can be found at: https://dist.apache.org/repos/dist/dev/celeborn/v0.5.0-rc2 The git commit hash: 68c503eb0023e274f8ae09bf4c2687f6a0c01a25 The staging repo: https://repository.apache.org/content/repositories/orgapacheceleborn-1075/ The fingerprint of the PGP key release artifacts is signed with: FCF20BB29C7BEFDF58F998F76392F71F37356FA0 My public key to verify signatures can be found in: https://dist.apache.org/repos/dist/release/celeborn/KEYS The vote will be open for at least 72 hours or until the necessary number of votes are reached. Please vote accordingly: [ ] +1 approve [ ] +0 no opinion [ ] -1 disapprove (and the reason) Steps to validate the release: https://www.apache.org/info/verification.html * Download links, checksums, and PGP signatures are valid. * Source code distributions have correct names matching the current release. * LICENSE and NOTICE files are correct. * All files have license headers if necessary. * No unlicensed compiled archives bundled in the source archive. * The source tarball matches the git tag. * Build from source is successful. There are additional tests: * Performance test no regression 1 TB TPC-DS, 0.5.0 VS 0.4.1 : 2042(s) VS 2050(s) 1.1 TB pure shuffle, 0.5.0 VS 0.4.1 : 11.8min vs 11.8min * Result correctness test passed 1TB TPC-DS runs concurrently, the results are identical. * Usability test passed Rolling upgrade from version 0.4.1 to 0.5.0 succeed. The metrics system works as expected. * Stability test passed Random worker failures, Celeborn works as expected. Random master failures, Celeborn works as expected. Master meta corrupted, Celeborn works as expected. * Compatibility test passed The Celeborn server version of 0.5.0 works fine with the Celeborn client 0.4.1. Regards, Ethan Feng
Re: [VOTE] Release Apache Celeborn 0.5.0-rc1
This vote is canceled due to newly found license issues. Ethan Feng 于2024年6月11日周二 17:48写道： > > Hello, Celeborn community, > > This is a call for a vote to release Apache Celeborn > 0.5.0-rc1 > > The git tag to be voted upon: > https://github.com/apache/celeborn/releases/tag/v0.5.0-rc1 > > Source and binary artifacts can be found at: > https://dist.apache.org/repos/dist/dev/celeborn/v0.5.0-rc1 > > The git commit hash: > 734c42a81c7ed9ebe0c7dbe389a85f31cba116d2 > > The staging repo: > https://repository.apache.org/content/repositories/orgapacheceleborn-1067/ > > The fingerprint of the PGP key release artifacts is signed with: > FCF20BB29C7BEFDF58F998F76392F71F37356FA0 > > My public key to verify signatures can be found in: > https://dist.apache.org/repos/dist/release/celeborn/KEYS > > The vote will be open for at least 72 hours or until the necessary > number of votes are reached. > > Please vote accordingly: > > [ ] +1 approve > [ ] +0 no opinion > [ ] -1 disapprove (and the reason) > > Steps to validate the release: > https://www.apache.org/info/verification.html > > * Download links, checksums, and PGP signatures are valid. > * Source code distributions have correct names matching the current release. > * LICENSE and NOTICE files are correct. > * All files have license headers if necessary. > * No unlicensed compiled archives bundled in the source archive. > * The source tarball matches the git tag. > * Build from source is successful. > > There are additional tests: > * Performance test no regression > 1 TB TPC-DS, 0.5.0 VS 0.4.1 : 2042(s) VS 2050(s) > 1.1 TB pure shuffle, 0.5.0 VS 0.4.1 : 11.8min vs 11.8min > > * Result correctness test passed > 1TB TPC-DS runs concurrently, the results are identical. > > * Usability test passed > Rolling upgrade from version 0.4.1 to 0.5.0 succeed. > The metrics system works as expected. > > * Stability test passed > Random worker failures, Celeborn works as expected. > Random master failures, Celeborn works as expected. > Master meta corrupted, Celeborn works as expected. > > * Compatibility test passed > The Celeborn server version of 0.5.0 works fine with the Celeborn client > 0.4.1. > > > Regards, > Ethan Feng
[VOTE] Release Apache Celeborn 0.5.0-rc1
Hello, Celeborn community, This is a call for a vote to release Apache Celeborn 0.5.0-rc1 The git tag to be voted upon: https://github.com/apache/celeborn/releases/tag/v0.5.0-rc1 Source and binary artifacts can be found at: https://dist.apache.org/repos/dist/dev/celeborn/v0.5.0-rc1 The git commit hash: 734c42a81c7ed9ebe0c7dbe389a85f31cba116d2 The staging repo: https://repository.apache.org/content/repositories/orgapacheceleborn-1067/ The fingerprint of the PGP key release artifacts is signed with: FCF20BB29C7BEFDF58F998F76392F71F37356FA0 My public key to verify signatures can be found in: https://dist.apache.org/repos/dist/release/celeborn/KEYS The vote will be open for at least 72 hours or until the necessary number of votes are reached. Please vote accordingly: [ ] +1 approve [ ] +0 no opinion [ ] -1 disapprove (and the reason) Steps to validate the release: https://www.apache.org/info/verification.html * Download links, checksums, and PGP signatures are valid. * Source code distributions have correct names matching the current release. * LICENSE and NOTICE files are correct. * All files have license headers if necessary. * No unlicensed compiled archives bundled in the source archive. * The source tarball matches the git tag. * Build from source is successful. There are additional tests: * Performance test no regression 1 TB TPC-DS, 0.5.0 VS 0.4.1 : 2042(s) VS 2050(s) 1.1 TB pure shuffle, 0.5.0 VS 0.4.1 : 11.8min vs 11.8min * Result correctness test passed 1TB TPC-DS runs concurrently, the results are identical. * Usability test passed Rolling upgrade from version 0.4.1 to 0.5.0 succeed. The metrics system works as expected. * Stability test passed Random worker failures, Celeborn works as expected. Random master failures, Celeborn works as expected. Master meta corrupted, Celeborn works as expected. * Compatibility test passed The Celeborn server version of 0.5.0 works fine with the Celeborn client 0.4.1. Regards, Ethan Feng
Re: [VOTE] Release Apache Celeborn 0.5.0-rc0
Due to license issues, this vote is canceled. Thanks, Ethan Feng Keyong Zhou 于2024年6月8日周六 10:26写道： > > Hi Ethan, > > Thanks for the effort! Although others LGTM, I found some issues in the > LICENSE file: > > 1. ratis-metrics-default, > ap-loader-all, jersey-server, jersey-container-servlet-core, > jersey-hk2, jersey-media-json-jackson, jersey-media-multipart > are missing in the LICENSE file > 2. javax.servlet is deleted, so it should also be removed from the > LICENSE-binary file > > Regards, > Keyong Zhou > > Ethan Feng 于2024年6月7日周五 15:56写道： > > > Hello, Celeborn community, > > > > This is a call for a vote to release Apache Celeborn > > 0.5.0-rc0 > > > > The git tag to be voted upon: > > https://github.com/apache/celeborn/releases/tag/v0.5.0-rc0 > > > > Source and binary artifacts can be found at: > > https://dist.apache.org/repos/dist/dev/celeborn/v0.5.0-rc0 > > > > The git commit hash: > > 36567733aace2c83e533cbefcc5cd374ca935c76 > > > > The staging repo: > > https://repository.apache.org/content/repositories/orgapacheceleborn-1066/ > > > > The fingerprint of the PGP key release artifacts is signed with: > > FCF20BB29C7BEFDF58F998F76392F71F37356FA0 > > > > My public key to verify signatures can be found in: > > https://dist.apache.org/repos/dist/release/celeborn/KEYS > > > > The vote will be open for at least 72 hours or until the necessary > > number of votes are reached. > > > > Please vote accordingly: > > > > [ ] +1 approve > > [ ] +0 no opinion > > [ ] -1 disapprove (and the reason) > > > > Steps to validate the release: > > https://www.apache.org/info/verification.html > > > > * Download links, checksums, and PGP signatures are valid. > > * Source code distributions have correct names matching the current > > release. > > * LICENSE and NOTICE files are correct. > > * All files have license headers if necessary. > > * No unlicensed compiled archives bundled in the source archive. > > * The source tarball matches the git tag. > > * Build from source is successful. > > > > There are additional tests: > > * Performance test no regression > > 1 TB TPC-DS, 0.5.0 VS 0.4.1 : 2042(s) VS 2050(s) > > 1.1 TB pure shuffle, 0.5.0 VS 0.4.1 : 11.8min vs 11.8min > > > > * Result correctness test passed > > 1TB TPC-DS runs concurrently, the results are identical. > > > > * Usability test passed > > Rolling upgrade from version 0.4.1 to 0.5.0 succeed. > > The metrics system works as expected. > > > > * Stability test passed > > Random worker failures, Celeborn works as expected. > > Random master failures, Celeborn works as expected. > > Master meta corrupted, Celeborn works as expected. > > > > * Compatibility test passed > > The Celeborn server version of 0.5.0 works fine with the Celeborn client > > 0.4.1. > > > > > > Regards, > > Ethan Feng > >
[VOTE] Release Apache Celeborn 0.5.0-rc0
Hello, Celeborn community, This is a call for a vote to release Apache Celeborn 0.5.0-rc0 The git tag to be voted upon: https://github.com/apache/celeborn/releases/tag/v0.5.0-rc0 Source and binary artifacts can be found at: https://dist.apache.org/repos/dist/dev/celeborn/v0.5.0-rc0 The git commit hash: 36567733aace2c83e533cbefcc5cd374ca935c76 The staging repo: https://repository.apache.org/content/repositories/orgapacheceleborn-1066/ The fingerprint of the PGP key release artifacts is signed with: FCF20BB29C7BEFDF58F998F76392F71F37356FA0 My public key to verify signatures can be found in: https://dist.apache.org/repos/dist/release/celeborn/KEYS The vote will be open for at least 72 hours or until the necessary number of votes are reached. Please vote accordingly: [ ] +1 approve [ ] +0 no opinion [ ] -1 disapprove (and the reason) Steps to validate the release: https://www.apache.org/info/verification.html * Download links, checksums, and PGP signatures are valid. * Source code distributions have correct names matching the current release. * LICENSE and NOTICE files are correct. * All files have license headers if necessary. * No unlicensed compiled archives bundled in the source archive. * The source tarball matches the git tag. * Build from source is successful. There are additional tests: * Performance test no regression 1 TB TPC-DS, 0.5.0 VS 0.4.1 : 2042(s) VS 2050(s) 1.1 TB pure shuffle, 0.5.0 VS 0.4.1 : 11.8min vs 11.8min * Result correctness test passed 1TB TPC-DS runs concurrently, the results are identical. * Usability test passed Rolling upgrade from version 0.4.1 to 0.5.0 succeed. The metrics system works as expected. * Stability test passed Random worker failures, Celeborn works as expected. Random master failures, Celeborn works as expected. Master meta corrupted, Celeborn works as expected. * Compatibility test passed The Celeborn server version of 0.5.0 works fine with the Celeborn client 0.4.1. Regards, Ethan Feng
Re: Re: Re: [DISCUSS] Time for 0.5.0
There will be many tests to run before releasing Celeborn 0.5.0. I'll check out the release branch today and test it. Shaoyun Chen 于2024年5月27日周一 16:47写道： > > +1 > > Nicholas 于2024年5月27日周一 15:50写道： > > > > Hi Ethan Feng, > > > > > > > > > > Thanks for the confirm of dynamic config service. This makes sense to me. > > +1 for releasing 0.5.0. > > > > > > > > > > Regards, > > > > Nicholas Jiang > > > > > > > > > > 在 2024-05-27 15:10:05，"Ethan Feng" 写道： > > >Hi Nicholas, > > > > > >The dynamic config service is production-ready but the database-based > > >config server has not been tested in the production environment. > > > > > >Regards, > > >Ethan Feng > > > > > >Nicholas 于2024年5月26日周日 11:37写道： > > >> > > >> Hi Ethan Feng, thanks for driving volunteering to release 0.5.0. +1 for > > >> releasing 0.5.0. BTW, is the dynamic config service also experimental or > > >> production ready? > > >> > > >> > > >> > > >> > > >> Regards, > > >> > > >> Nicholas Jiang > > >> > > >> > > >> At 2024-05-24 20:11:02, "Keyong Zhou" wrote: > > >> >+1 for releasing 0.5.0. But I think memory file storage is still > > >> >experimental. > > >> > > > >> >Regards, > > >> >Keyong Zhou > > >> > > > >> >Ethan Feng 于2024年5月24日周五 18:15写道： > > >> > > > >> >> Hello, Celeborn community, > > >> >> > > >> >> It has been 4 months since we released the last major version. Some > > >> >> new features, such as SSL support and memory file storage, are now > > >> >> ready. Several optimizations have been merged into the main branch. > > >> >> Many components are updated to the latest version. > > >> >> > > >> >> What do you think? I'm volunteering to be the release manager if no > > >> >> one else has applied. > > >> >> > > >> >> Thanks, > > >> >> Ethan Feng > > >> >>
Re: Re: [DISCUSS] Time for 0.5.0
Hi Nicholas, The dynamic config service is production-ready but the database-based config server has not been tested in the production environment. Regards, Ethan Feng Nicholas 于2024年5月26日周日 11:37写道： > > Hi Ethan Feng, thanks for driving volunteering to release 0.5.0. +1 for > releasing 0.5.0. BTW, is the dynamic config service also experimental or > production ready? > > > > > Regards, > > Nicholas Jiang > > > At 2024-05-24 20:11:02, "Keyong Zhou" wrote: > >+1 for releasing 0.5.0. But I think memory file storage is still > >experimental. > > > >Regards, > >Keyong Zhou > > > >Ethan Feng 于2024年5月24日周五 18:15写道： > > > >> Hello, Celeborn community, > >> > >> It has been 4 months since we released the last major version. Some > >> new features, such as SSL support and memory file storage, are now > >> ready. Several optimizations have been merged into the main branch. > >> Many components are updated to the latest version. > >> > >> What do you think? I'm volunteering to be the release manager if no > >> one else has applied. > >> > >> Thanks, > >> Ethan Feng > >>
Re: [DISCUSS] Time for 0.5.0
The decision to continue with experimental memory file storage is based on the potential benefits it offers in terms of performance. We are continuously testing and improving this feature to ensure its reliability and stability. Regards, Ethan Feng Keyong Zhou 于2024年5月24日周五 20:11写道： > > +1 for releasing 0.5.0. But I think memory file storage is still > experimental. > > Regards, > Keyong Zhou > > Ethan Feng 于2024年5月24日周五 18:15写道： > > > Hello, Celeborn community, > > > > It has been 4 months since we released the last major version. Some > > new features, such as SSL support and memory file storage, are now > > ready. Several optimizations have been merged into the main branch. > > Many components are updated to the latest version. > > > > What do you think? I'm volunteering to be the release manager if no > > one else has applied. > > > > Thanks, > > Ethan Feng > >
[DISCUSS] Time for 0.5.0
Hello, Celeborn community, It has been 4 months since we released the last major version. Some new features, such as SSL support and memory file storage, are now ready. Several optimizations have been merged into the main branch. Many components are updated to the latest version. What do you think? I'm volunteering to be the release manager if no one else has applied. Thanks, Ethan Feng
Re: [VOTE] Release Apache Celeborn 0.4.1-rc1
CORRECTION: There is no DISCLAIMER in the tlp release. I revoke the mail that I just sent. +1(binding) I checked: √ release files in the correct location √ digital signature and hashes correct √ LICENSE and NOTICE files exist and are correct √ the contents of the release match the tag in VCS √ can build the release from the source √ maven artifacts look correct Thanks, Ethan Feng Ethan Feng 于2024年5月22日周三 10:32写道： > > +1(binding) > > I checked: > √ release files in the correct location > √ digital signature and hashes correct > √ DISCLAIMER file exist > √ LICENSE and NOTICE files exist and are correct > √ the contents of the release match the tag in VCS > √ can build the release from the source > √ maven artifacts look correct > > Thanks, > Ethan Feng > > Keyong Zhou 于2024年5月22日周三 10:25写道： > > > > +1 (binding) > > > > I checked > > - git commit hash is correct. > > - links are valid. > > - signatures are good. > > ``` > > gpg --import KEYS > > gpg --verify apache-celeborn-0.4.1-bin.tgz.asc > > gpg --verify apache-celeborn-0.4.1-source.tgz.asc > > ``` > > - checksums are good. > > ``` > > sha512sum --check apache-celeborn-0.4.1-bin.tgz.sha512 > > sha512sum --check apache-celeborn-0.4.1-source.tgz.sha512 > > ``` > > - LICENSE looks good. > > - NOTICE looks good. > > - build success from source code (macOS). > > ``` > > ./build/make-distribution.sh --sbt-enabled --release > > ``` > > > > I also tested performance, no regression: > > 1T TPCDS, 0.4.0 vs. 0.4.1: 2136s vs. 2127s > > 734G pure shuffle, 0.4.0 vs. 0.4.1: 10.3min vs. 10.3min > > > > Regards, > > Keyong Zhou > > > > Cheng Pan 于2024年5月21日周二 14:16写道： > > > > > +1 > > > > > > I have rolled out this version to a small cluster for several days, > > > everything goes well so far. > > > > > > I checked the > > > org.apache.celeborn:celeborn-client-spark-3-shaded_2.12:0.4.1, it does not > > > pull transitive deps now. While there might be some issues with the > > > relocation of Guava classes, specifically, package > > > `com.google.thirdparty.publicsuffix` still be there. > > > > > > Thanks, > > > Cheng Pan > > > > > > > > > > On May 14, 2024, at 12:13, Nicholas Jiang > > > wrote: > > > > > > > > Hi Celeborn community, > > > > > > > > This is a call for a vote to release Apache Celeborn > > > > > > > > 0.4.1-rc1 > > > > > > > > > > > > The git tag to be voted upon: > > > > > > > > https://github.com/apache/celeborn/releases/tag/v0.4.1-rc1 > > > > > > > > The git commit hash: > > > > 641180142c5ef36430a6afcd702c9487a6007458 source and binary artifacts can > > > be > > > > found at: > > > > > > > > https://dist.apache.org/repos/dist/dev/celeborn/v0.4.1-rc1 > > > > > > > > The staging repo: > > > > > > > > > > > https://repository.apache.org/content/repositories/orgapacheceleborn-1055 > > > > > > > > > > > > Fingerprint of the PGP key release artifacts are signed with: > > > > D73CADC1DAB63BD3C770BB6D9476842D24B7C885 > > > > > > > > My public key to verify signatures can be found in: > > > > > > > > https://dist.apache.org/repos/dist/release/celeborn/KEYS > > > > > > > > The vote will be open for at least 72 hours or until the necessary > > > > number of votes are reached. > > > > > > > > Please vote accordingly: > > > > > > > > [ ] +1 approve > > > > [ ] +0 no opinion > > > > [ ] -1 disapprove (and the reason) > > > > > > > > Steps to validate the release: > > > > > > > > https://www.apache.org/info/verification.html > > > > > > > > * Download links, checksums and PGP signatures are valid. > > > > * Source code distributions have correct names matching the current > > > release. > > > > * LICENSE and NOTICE files are correct. > > > > * All files have license headers if necessary. > > > > * No unlicensed compiled archives bundled in source archive. > > > > * The source tarball matches the git tag. > > > > * Build from source is successful. > > > > > > > > Regards, > > > > Nicholas Jiang > > > > > > > > >
Re: [VOTE] Release Apache Celeborn 0.4.1-rc1
+1(binding) I checked: √ release files in the correct location √ digital signature and hashes correct √ DISCLAIMER file exist √ LICENSE and NOTICE files exist and are correct √ the contents of the release match the tag in VCS √ can build the release from the source √ maven artifacts look correct Thanks, Ethan Feng Keyong Zhou 于2024年5月22日周三 10:25写道： > > +1 (binding) > > I checked > - git commit hash is correct. > - links are valid. > - signatures are good. > ``` > gpg --import KEYS > gpg --verify apache-celeborn-0.4.1-bin.tgz.asc > gpg --verify apache-celeborn-0.4.1-source.tgz.asc > ``` > - checksums are good. > ``` > sha512sum --check apache-celeborn-0.4.1-bin.tgz.sha512 > sha512sum --check apache-celeborn-0.4.1-source.tgz.sha512 > ``` > - LICENSE looks good. > - NOTICE looks good. > - build success from source code (macOS). > ``` > ./build/make-distribution.sh --sbt-enabled --release > ``` > > I also tested performance, no regression: > 1T TPCDS, 0.4.0 vs. 0.4.1: 2136s vs. 2127s > 734G pure shuffle, 0.4.0 vs. 0.4.1: 10.3min vs. 10.3min > > Regards, > Keyong Zhou > > Cheng Pan 于2024年5月21日周二 14:16写道： > > > +1 > > > > I have rolled out this version to a small cluster for several days, > > everything goes well so far. > > > > I checked the > > org.apache.celeborn:celeborn-client-spark-3-shaded_2.12:0.4.1, it does not > > pull transitive deps now. While there might be some issues with the > > relocation of Guava classes, specifically, package > > `com.google.thirdparty.publicsuffix` still be there. > > > > Thanks, > > Cheng Pan > > > > > > > On May 14, 2024, at 12:13, Nicholas Jiang > > wrote: > > > > > > Hi Celeborn community, > > > > > > This is a call for a vote to release Apache Celeborn > > > > > > 0.4.1-rc1 > > > > > > > > > The git tag to be voted upon: > > > > > > https://github.com/apache/celeborn/releases/tag/v0.4.1-rc1 > > > > > > The git commit hash: > > > 641180142c5ef36430a6afcd702c9487a6007458 source and binary artifacts can > > be > > > found at: > > > > > > https://dist.apache.org/repos/dist/dev/celeborn/v0.4.1-rc1 > > > > > > The staging repo: > > > > > > > > https://repository.apache.org/content/repositories/orgapacheceleborn-1055 > > > > > > > > > Fingerprint of the PGP key release artifacts are signed with: > > > D73CADC1DAB63BD3C770BB6D9476842D24B7C885 > > > > > > My public key to verify signatures can be found in: > > > > > > https://dist.apache.org/repos/dist/release/celeborn/KEYS > > > > > > The vote will be open for at least 72 hours or until the necessary > > > number of votes are reached. > > > > > > Please vote accordingly: > > > > > > [ ] +1 approve > > > [ ] +0 no opinion > > > [ ] -1 disapprove (and the reason) > > > > > > Steps to validate the release: > > > > > > https://www.apache.org/info/verification.html > > > > > > * Download links, checksums and PGP signatures are valid. > > > * Source code distributions have correct names matching the current > > release. > > > * LICENSE and NOTICE files are correct. > > > * All files have license headers if necessary. > > > * No unlicensed compiled archives bundled in source archive. > > > * The source tarball matches the git tag. > > > * Build from source is successful. > > > > > > Regards, > > > Nicholas Jiang > > > > > >
Re: [DISCUSS] Time for 0.4.1
+1 Thanks, Ethan Feng Yu Li 于2024年4月16日周二 17:20写道： > > +1, thanks for driving this and volunteering as our RM, Nicholas! > > Best Regards, > Yu > > On Sat, 13 Apr 2024 at 10:31, Keyong Zhou wrote: > > > > +1, thanks Nicholas for volunteering! > > > > Regards, > > Keyong Zhou > > > > Shaoyun Chen 于2024年4月12日周五 22:03写道： > > > > > +1 > > > > > > Cheng Pan 于2024年4月12日周五 20:04写道： > > > > > > > > +1, we do need a patch release for 0.4 > > > > > > > > Thanks, > > > > Cheng Pan > > > > > > > > > > > > > On Apr 12, 2024, at 19:59, Nicholas Jiang > > > wrote: > > > > > > > > > > Hey, Celeborn community, > > > > > > > > > > > > > > > It has been a while since the 0.4.0 release, and there are some > > > critical fixes land branch-0.4, for example, [CELEBORN-1252][FOLLOWUP] Fix > > > Worker#computeResourceConsumption NullPointerException for > > > userResourceConsumption that does not contain given userIdentifier. From > > > my > > > perspective, it’s time to prepare for releasing 0.4.1. > > > > > > > > > > > > > > > WDYT? And I’m volunteering to be the release manager if no one has > > > applied. > > > > > > > > > > Regards, > > > > > Nicholas Jiang > > > > > > > > > > >
[ANNOUNCE] Apache Celeborn is graduated to Top Level Project
Hello Celeborn community, I am glad to share that the ASF board has approved a resolution to graduate Celeborn into a full Top Level Project. Thank you all for your help in reaching this milestone. To transition from the Apache Incubator to a new TLP, there are a few action items[1] we need to complete the transition. I have opened an Umbrella Issue[2] to track the tasks, and you are welcome to take on the sub-tasks and leave comments if I have missed anything. Additionally, the GitHub repository migration is already complete[3]. Please update your local git repository to track the new repo[4]. If you named the upstream as "apache", you can run the following command to complete the remote repo tracking migration. ` git remote set-url apache g...@github.com:apache/celeborn.git ` Please find the relevant URLs below: [1] https://incubator.apache.org/guides/transferring.html#life_after_graduation [2] https://github.com/apache/celeborn/issues/2415 [3] https://issues.apache.org/jira/browse/INFRA-25635 [4] https://github.com/apache/celeborn Thanks, Ethan Feng
Re: Re: [ANNOUNCE] Add Chandni Singh as new committer
Congratulations. Thanks, Ethan Feng Cheng Pan 于2024年3月21日周四 23:08写道： > > Welcome, and congrats! > > Thanks, > Cheng Pan > > On Thu, Mar 21, 2024 at 7:46 PM Yihe Li wrote: > > > > Congratulations! > > > > Regards, > > Yihe Li > > > > On 2024/03/21 11:21:52 Fu Chen wrote: > > > Congratulations and thank you to Chandni for the contributions to the > > > community! > > > > > > Regards, > > > Fu Chen > > > > > > Nicholas 于2024年3月21日周四 19:20写道： > > > > > > > > Congratulations Chandni. Well deserved! > > > > > > > > > > > > > > > > > > > > Regards, > > > > Nicholas Jiang > > > > > > > > > > > > > > > > > > > > At 2024-03-21 19:16:25, "Shaoyun Chen" wrote: > > > > >Congratulations! > > > > > > > > > >Mridul Muralidharan 于2024年3月21日周四 16:54写道： > > > > >> > > > > >> Congratulations Chandni ! Great job :-) > > > > >> > > > > >> Regards, > > > > >> Mridul > > > > >> > > > > >> > > > > >> On Thu, Mar 21, 2024 at 3:30 AM Keyong Zhou > > > > >> wrote: > > > > >> > > > > >> > Hi Celeborn Community, > > > > >> > > > > > >> > The Podling Project Management Committee (PPMC) for Apache Celeborn > > > > >> > has invited Chandni Singh to become a committer and we are pleased > > > > >> > to announce that she has accepted. > > > > >> > > > > > >> > Being a committer enables easier contribution to the > > > > >> > project since there is no need to go via the patch > > > > >> > submission process. This should enable better productivity. > > > > >> > A (P)PMC member helps manage and guide the direction of the > > > > >> > project. > > > > >> > > > > > >> > Please join me in congratulating Chandni Singh! > > > > >> > > > > > >> > Thanks, > > > > >> > Keyong Zhou > > > > >> > > > >
Re: Maven 'stuck' in service test compilation ?
Hi Mridul, I've tried your scripts on my local environment(JDK8) and the problem is not reproduced. Both Maven 3.8.8 and 3.9.6 are tested. Changing might help as I've encountered some maven bugs before. I think there needs more information about how to reproduce this problem, like the environmental information, JDK version, etc. Regards, Ethan Feng Mridul Muralidharan 于2024年3月21日周四 15:25写道： > > Hi, > > > I am observing that a maven build gets 'stuck' when compiling "services" > for running tests. > Without tests, this goes through: > > $ ARGS="-Pspark-3.1"; ./build/mvn $ARGS clean 2>&1 | tee clean_output.txt > && ./build/mvn -DskipTests $ARGS package 2>&1 | tee build_output.txt > > This gets stuck indefinitely: > > $ ARGS="-Pspark-3.1"; ./build/mvn $ARGS package 2>&1 | tee test_output.txt > See [1] for output snippet. > > Strangely, running with -X seemed to be fine (the one time I tried it). > > I have made some dependency changes to pom.xml, but no changes to > service module. > > Anything I am missing here ? Any hints would be greatly appreciated :-) > > Thanks ! > Mridul > > [1] > > [INFO] --- maven-resources-plugin:3.2.0:resources (default-resources) @ > celeborn-service_2.12 --- > [INFO] Using 'UTF-8' encoding to copy filtered resources. > [INFO] Using 'UTF-8' encoding to copy filtered properties files. > [INFO] Copying 1 resource > [INFO] Copying 3 resources > [INFO] > [INFO] --- scala-maven-plugin:4.7.2:compile (scala-compile-first) @ > celeborn-service_2.12 --- > [INFO] Using incremental compilation using Mixed compile order > [INFO] Compiler bridge file: > /home/mridul/.sbt/1.0/zinc/org.scala-sbt/org.scala-sbt-compiler-bridge_2.12-1.7.1-bin_2.12.10__61.0-1.7.1_20220712T022208.jar > [INFO] compiler plugin: > BasicArtifact(com.github.ghik,silencer-plugin_2.12.10,1.6.0,null) > [INFO] compiling 4 Scala sources and 9 Java sources to > /home/mridul/work/apache/celeborn/incubator-celeborn/service/target/classes > ... > [INFO] NoPosition: Note: Some input files use unchecked or unsafe > operations. > [INFO] NoPosition: Note: Recompile with -Xlint:unchecked for details. > [INFO] done compiling > [INFO] compiling 2 Scala sources and 2 Java sources to > /home/mridul/work/apache/celeborn/incubator-celeborn/service/target/classes > ... > [INFO] done compiling > [INFO] compiling 1 Scala source and 5 Java sources to > /home/mridul/work/apache/celeborn/incubator-celeborn/service/target/classes > ... > [INFO] NoPosition: Note: Some input files use unchecked or unsafe > operations. > [INFO] NoPosition: Note: Recompile with -Xlint:unchecked for details. > [INFO] done compiling > [INFO] compiling 5 Scala sources and 2 Java sources to > /home/mridul/work/apache/celeborn/incubator-celeborn/service/target/classes > ... > [INFO] done compiling > [INFO] compiling 5 Scala sources and 5 Java sources to > /home/mridul/work/apache/celeborn/incubator-celeborn/service/target/classes > ... > [INFO] NoPosition: Note: Some input files use unchecked or unsafe > operations. > [INFO] NoPosition: Note: Recompile with -Xlint:unchecked for details. > [INFO] done compiling > [INFO] compiling 5 Scala sources and 2 Java sources to > /home/mridul/work/apache/celeborn/incubator-celeborn/service/target/classes > ... > [INFO] done compiling > [INFO] compiling 5 Scala sources and 5 Java sources to > /home/mridul/work/apache/celeborn/incubator-celeborn/service/target/classes > ... > [INFO] NoPosition: Note: Some input files use unchecked or unsafe > operations. > [INFO] NoPosition: Note: Recompile with -Xlint:unchecked for details. > [INFO] done compiling > [INFO] compiling 5 Scala sources and 2 Java sources to > /home/mridul/work/apache/celeborn/incubator-celeborn/service/target/classes > ... > [INFO] done compiling > [INFO] compiling 5 Scala sources and 5 Java sources to > /home/mridul/work/apache/celeborn/incubator-celeborn/service/target/classes > ... > [INFO] NoPosition: Note: Some input files use unchecked or unsafe > operations. > [INFO] NoPosition: Note: Recompile with -Xlint:unchecked for details. > [INFO] done compiling > [INFO] compiling 5 Scala sources and 2 Java sources to > /home/mridul/work/apache/celeborn/incubator-celeborn/service/target/classes > ... > [INFO] done compiling > [INFO] compiling 5 Scala sources and 5 Java sources to > /home/mridul/work/apache/celeborn/incubator-celeborn/service/target/classes > ... > [INFO] NoPosition: Note: Some input files use unchecked or unsafe > operations. > [INFO] NoPosition: Note: Recompile with -Xlint:unchecked for details. > [INFO] done compiling > > And then keeps indefinitely repeating this.
Re: [DISCUSS] Graduate Celeborn as TLP
Hi Yu, Thanks for sharing the statistics and drafted resolution for graduating Celeborn as TLP. From my perspective, the community has worked hard and met all the maturity criteria as outlined in [1]. It's time to move forward with the graduation process. Endorse that Keyong Zhou be the PMC chair. Let's graduate. +1 Thanks, Ethan Feng Yu Li 于2024年2月27日周二 09:40写道： > > Dear Celeborn Devs, > > We, the Celeborn community, began our incubation journey on October > 18, 2022. Since then, with the continuous efforts of you all, our > community has steadily developed and gradually matured, approaching > the graduation criteria [1]. Therefore, I'd like to call a discussion > to graduate Celeborn as TLP. Below are some statistics I collected, > please check it and let me know your thoughts. > > - Currently, our community consists of 19 committers (including > mentors) from more than 10 companies, with 12 serving as PPMC members > [2]. > - So far, we have boasted 81 contributors. > - Throughout the incubation period, we've made 6 releases [3] in 16 > months, at a stable pace. > - We've had 6 different release managers to date. > - Our software is used in production by 10+ well known entities [4]. > - As yet, we have opened 1,286 issues with 1,176 successfully resolved [5]. > - We have submitted a total of 1,816 PRs, out of which 1,805 have been > merged or closed [6]. > - Through self-assessment [7], we have met all maturity criteria as > outlined in [1]. > > And below is the drafted graduation resolution, JFYI: > -- > Establish the Apache Celeborn Project > > WHEREAS, the Board of Directors deems it to be in the best interests of > the Foundation and consistent with the Foundation's purpose to establish > a Project Management Committee charged with the creation and maintenance > of open-source software, for distribution at no charge to the public, > related to an intermediate data service for big data computing engines > to boost performance, stability, and flexibility. > > NOW, THEREFORE, BE IT RESOLVED, that a Project Management Committee > (PMC), to be known as the "Apache Celeborn Project", be and hereby is > established pursuant to Bylaws of the Foundation; and be it further > > RESOLVED, that the Apache Celeborn Project be and hereby is responsible > for the creation and maintenance of software related to an intermediate > data service for big data computing engines to boost performance, > stability, and flexibility; and be it further > > RESOLVED, that the office of "Vice President, Apache Celeborn" be and > hereby is created, the person holding such office to serve at the > direction of the Board of Directors as the chair of the Apache Celeborn > Project, and to have primary responsibility for management of the > projects within the scope of responsibility of the Apache Celeborn > Project; and be it further > > RESOLVED, that the persons listed immediately below be and hereby are > appointed to serve as the initial members of the Apache Celeborn > Project: > > * Becket Qin > * Cheng Pan > * Duo Zhang > * Ethan Feng > * Fu Chen > * Jiashu Xiong > * Kerwin Zhang > * Keyong Zhou > * Lidong Dai > * Willem Ning Jiang > * Wu Wei > * Yi Zhu > * Yu Li > > NOW, THEREFORE, BE IT FURTHER RESOLVED, that Keyong Zhou be appointed to > the office of Vice President, Apache Celeborn, to serve in accordance > with and subject to the direction of the Board of Directors and the > Bylaws of the Foundation until death, resignation, retirement, removal > or disqualification, or until a successor is appointed; and be it > further > > RESOLVED, that the Apache Celeborn Project be and hereby is tasked with > the migration and rationalization of the Apache Incubator Celeborn > podling; and be it further > > RESOLVED, that all responsibilities pertaining to the Apache Incubator > Celeborn podling encumbered upon the Apache Incubator PMC are hereafter > discharged. > -- > > Best Regards, > Yu > > [1] https://incubator.apache.org/guides/graduation.html > [2] https://celeborn.apache.org/community/project_management_committee > [3] > https://issues.apache.org/jira/projects/CELEBORN?selectedItem=com.atlassian.jira.jira-projects-plugin:release-page&status=released > [4] https://github.com/apache/incubator-celeborn/issues/2140 > [5] https://s.apache.org/celeborn_jira_issues > [6] https://github.com/apache/incubator-celeborn/pulls > [7] > https://cwiki.apache.org/confluence/display/CELEBORN/Apache+Maturity+Model+Assessment+for+Celeborn
Re: [VOTE] Release Apache Celeborn(Incubating) 0.4.0-incubating-rc6
+1(binding) I checked: √ release files in the correct location √ release files have the word incubating √ digital signature and hashes correct √ DISCLAIMER file exist √ LICENSE and NOTICE files exist and are correct √ the contents of the release match the tag in VCS √ can build the release from the source √ maven artifacts look correct Thanks, Ethan Feng Wei Wu 于2024年1月30日周二 11:00写道： > > +1 (binding) > > I checked > > - Download links are valid > > - git commit hash is correct > > - Checksums and signatures are valid > > - Release files have the word 'incubating' in their name > > - DISCLAIMER, LICENSE and NOTICE files in tarballs are correct > > - Successfully built the binary from the source on Linux for aarch64 using > openjdk8 with command: > > ./build/make-distribution.sh -Pspark-3.5 > > Thanks, > > Wei Wu > > On Mon, Jan 29, 2024 at 9:46 PM Fu Chen wrote: > > > Hi Celeborn community, > > > > This is a call for a vote to release Apache Celeborn (Incubating) > > 0.4.0-incubating-rc6 > > > > > > The git tag to be voted upon: > > > > https://github.com/apache/incubator-celeborn/releases/tag/v0.4.0-incubating-rc6 > > > > > > The git commit hash: > > 20a8576fc696f0208c24ab52e6ae883f5f0567d5 > > source and binary artifacts can be > > found at: > > > > https://dist.apache.org/repos/dist/dev/incubator/celeborn/v0.4.0-incubating-rc6 > > > > > > The staging repo: > > https://repository.apache.org/content/repositories/orgapacheceleborn-1053 > > > > > > Fingerprint of the PGP key release artifacts are signed with: > > 92AF4750DAFCB5E25B5B83EA76F54B977EB5C09B > > > > > > My public key to verify signatures can be found in: > > https://dist.apache.org/repos/dist/release/incubator/celeborn/KEYS > > > > > > The vote will be open for at least 72 hours or until the necessary > > number of votes are reached. > > > > > > Please vote accordingly: > > > > > > [ ] +1 approve > > [ ] +0 no opinion > > [ ] -1 disapprove (and the reason) > > > > > > Checklist for release: > > > > https://cwiki.apache.org/confluence/display/INCUBATOR/Incubator+Release+Checklist > > Steps to validate the release: > > https://www.apache.org/info/verification.html > > > > > > * Download links, checksums and PGP signatures are valid. > > * Source code distributions have correct names matching the current > > release. > > * Release files have the word incubating in their name. > > * DISCLAIMER, LICENSE and NOTICE files are correct. > > * All files have license headers if necessary. > > * No unlicensed compiled archives bundled in source archive. > > * The source tarball matches the git tag. > > * Build from source is successful. > > > > Please be aware that there has been a transition in the Celeborn project's > > build tool, shifting from Maven to SBT. The SBT build documentation is > > available > > at https://celeborn.apache.org/docs/latest/developers/sbt/. > > > > For illustrative purposes: > > > > Packaging the project > > ``` > > ./build/sbt clean package > > ``` > > > > Creating the distribution > > ``` > > ./build/make-distribution.sh --sbt-enabled --release > > ``` > > > > Thanks, > > Fu Chen > >
Re: [VOTE] Release Apache Celeborn(Incubating) 0.4.0-incubating-rc4
+1(binding) I checked: √ release files in the correct location √ release files have the word incubating √ digital signature and hashes correct √ DISCLAIMER file exist √ LICENSE and NOTICE files exist and are correct √ the contents of the release match the tag in VCS √ can build the release from the source √ maven artifacts look correct Thanks, Ethan Feng Yihe Li 于2024年1月19日周五 14:28写道： > > +1 (non-binding) > I checked the following things: > - git commit hash is correct. > - download links are valid. > - release files are in correct location. > - release files have the word incubating in their name. > - signatures and checksums are good. > - DISCLAIMER, LICENSE and NOTICE files exist. > - build success from source code(ubuntu 16.04). > ``` > ./build/make-distribution.sh --release > real30m46.326s > user154m25.760s > sys 29m14.300s > ./build/make-distribution.sh --sbt-enabled --release > real12m57.246s > user68m48.212s > sys 10m8.060s > ``` > > Thanks, > Yihe Li > > On 2024/01/19 02:24:53 Keyong Zhou wrote: > > +1 (binding) > > > > I checked > > - git commit hash is correct. > > - links are valid. > > - "incubating" is in the name. > > - signatures are good. > > ``` > > gpg --import KEYS > > gpg --verify apache-celeborn-0.4.0-incubating-source.tgz.asc > > gpg --verify apache-celeborn-0.4.0-incubating-bin.tgz.asc > > ``` > > - checksums are good. > > ``` > > sha512sum --check apache-celeborn-0.4.0-incubating-source.tgz.sha512 > > sha512sum --check apache-celeborn-0.4.0-incubating-bin.tgz.sha512 > > ``` > > - LICENSE looks good. > > - NOTICE looks good. > > - DISCLAIMER exists. > > - build success from source code (macOS). > > ``` > > ./build/make-distribution.sh --sbt-enabled --release > > ``` > > > > Thanks, > > Keyong Zhou > > > > Fu Chen 于2024年1月18日周四 21:40写道： > > > > > Hi Celeborn community, > > > > > > This is a call for a vote to release Apache Celeborn (Incubating) > > > 0.4.0-incubating-rc4 > > > > > > > > > The git tag to be voted upon: > > > > > > https://github.com/apache/incubator-celeborn/releases/tag/v0.4.0-incubating-rc4 > > > > > > > > > The git commit hash: > > > 8bc07466dd85a90216820617015e329fb806c7dd source and binary artifacts can > > > be > > > found at: > > > > > > https://dist.apache.org/repos/dist/dev/incubator/celeborn/v0.4.0-incubating-rc4 > > > > > > > > > The staging repo: > > > https://repository.apache.org/content/repositories/orgapacheceleborn-1051 > > > > > > > > > Fingerprint of the PGP key release artifacts are signed with: > > > 92AF4750DAFCB5E25B5B83EA76F54B977EB5C09B > > > > > > > > > My public key to verify signatures can be found in: > > > https://dist.apache.org/repos/dist/release/incubator/celeborn/KEYS > > > > > > > > > The vote will be open for at least 72 hours or until the necessary > > > number of votes are reached. > > > > > > > > > Please vote accordingly: > > > > > > > > > [ ] +1 approve > > > [ ] +0 no opinion > > > [ ] -1 disapprove (and the reason) > > > > > > > > > Checklist for release: > > > > > > https://cwiki.apache.org/confluence/display/INCUBATOR/Incubator+Release+Checklist > > > Steps to validate the release: > > > https://www.apache.org/info/verification.html > > > > > > > > > * Download links, checksums and PGP signatures are valid. > > > * Source code distributions have correct names matching the current > > > release. > > > * Release files have the word incubating in their name. > > > * DISCLAIMER, LICENSE and NOTICE files are correct. > > > * All files have license headers if necessary. > > > * No unlicensed compiled archives bundled in source archive. > > > * The source tarball matches the git tag. > > > * Build from source is successful. > > > > > > Please be aware that there has been a transition in the Celeborn project's > > > build tool, shifting from Maven to SBT. The SBT build documentation is > > > available > > > at https://celeborn.apache.org/docs/latest/developers/sbt/. > > > > > > For illustrative purposes: > > > > > > Packaging the project > > > ``` > > > ./build/sbt clean package > > > ``` > > > > > > Creating the distribution > > > ``` > > > ./build/make-distribution.sh --sbt-enabled --release > > > ``` > > > > > > Thanks, > > > Fu Chen > > > > >
Re: [VOTE] Release Apache Celeborn(Incubating) 0.4.0-incubating-rc3
+1(binding) I checked: √ release files in the correct location √ release files have the word incubating √ digital signature and hashes correct √ DISCLAIMER file exist √ LICENSE and NOTICE files exist and are correct √ the contents of the release match the tag in VCS √ can build the release from the source √ maven artifacts look correct Thanks, Ethan Feng Shaoyun Chen 于2024年1月2日周二 12:43写道： > > +1 (non-binding) > > I checked the following things: > > - signatures are good. > ``` > gpg --import KEYS > gpg --verify apache-celeborn-0.4.0-incubating-source.tgz.asc > gpg --verify apache-celeborn-0.4.0-incubating-bin.tgz.asc > ``` > - checksums are good. > ``` > sha512sum --check apache-celeborn-0.4.0-incubating-source.tgz.sha512 > sha512sum --check apache-celeborn-0.4.0-incubating-bin.tgz.sha512 > ``` > - build success from source code. > ``` > ./build/make-distribution.sh --release > ./build/make-distribution.sh --sbt-enabled --release > ``` > > Keyong Zhou 于2024年1月2日周二 10:05写道： > > > > +1 (binding) > > > > I checked > > - git commit hash is correct. > > - links are valid. > > - "incubating" is in the name. > > - PGP keys are good. > > ``` > > > > gpg --import KEYS > > > > gpg --verify apache-celeborn-0.4.0-incubating-source.tgz.asc > > > > gpg --verify apache-celeborn-0.4.0-incubating-bin.tgz.asc > > ``` > > - hashes are correct. > > ``` > > > > sha512sum --check apache-celeborn-0.4.0-incubating-source.tgz.sha512 > > > > sha512sum --check apache-celeborn-0.4.0-incubating-bin.tgz.sha512 > > ``` > > - LICENSE looks good. > > - NOTICE looks good. > > - DISCLAIMER exists. > > - build success from source code. ``` ./build/make-distribution.sh > > --sbt-enabled --release ``` > > > > Thanks, > > Keyong Zhou > > > > Fu Chen 于2024年1月1日周一 19:42写道： > > > > > Hi Celeborn community, > > > > > > This is a call for a vote to release Apache Celeborn (Incubating) > > > 0.4.0-incubating-rc3 > > > > > > > > > The git tag to be voted upon: > > > > > > https://github.com/apache/incubator-celeborn/releases/tag/v0.4.0-incubating-rc3 > > > > > > > > > The git commit hash: > > > 5d94bf9dea735650b90fc4959390da2bfc67fc37 source and binary artifacts can > > > be > > > found at: > > > > > > https://dist.apache.org/repos/dist/dev/incubator/celeborn/v0.4.0-incubating-rc3 > > > > > > > > > The staging repo: > > > https://repository.apache.org/content/repositories/orgapacheceleborn-1050 > > > > > > > > > Fingerprint of the PGP key release artifacts are signed with: > > > 92AF4750DAFCB5E25B5B83EA76F54B977EB5C09B > > > > > > > > > My public key to verify signatures can be found in: > > > https://dist.apache.org/repos/dist/release/incubator/celeborn/KEYS > > > > > > > > > The vote will be open for at least 72 hours or until the necessary > > > number of votes are reached. > > > > > > > > > Please vote accordingly: > > > > > > > > > [ ] +1 approve > > > [ ] +0 no opinion > > > [ ] -1 disapprove (and the reason) > > > > > > > > > Checklist for release: > > > > > > https://cwiki.apache.org/confluence/display/INCUBATOR/Incubator+Release+Checklist > > > Steps to validate the release: > > > https://www.apache.org/info/verification.html > > > > > > > > > * Download links, checksums and PGP signatures are valid. > > > * Source code distributions have correct names matching the current > > > release. > > > * Release files have the word incubating in their name. > > > * DISCLAIMER, LICENSE and NOTICE files are correct. > > > * All files have license headers if necessary. > > > * No unlicensed compiled archives bundled in source archive. > > > * The source tarball matches the git tag. > > > * Build from source is successful. > > > > > > Please be aware that there has been a transition in the Celeborn project's > > > build tool, shifting from Maven to SBT. The SBT build documentation is > > > available > > > at https://celeborn.apache.org/docs/latest/developers/sbt/. > > > > > > For illustrative purposes: > > > > > > Packaging the project > > > ``` > > > ./build/sbt clean package > > > ``` > > > > > > Creating the distribution > > > ``` > > > ./build/make-distribution.sh --sbt-enabled --release > > > ``` > > > > > > Thanks, > > > Fu Chen > > >
Re: [VOTE] Release Apache Celeborn(Incubating) 0.3.2-incubating-rc2
+1(binding) I checked: √ release files in the correct location √ release files have the word incubating √ digital signature and hashes correct √ DISCLAIMER file exist √ LICENSE and NOTICE files exist and are correct √ the contents of the release match the tag in VCS √ can build the release from the source √ maven artifacts look correct Thanks, Ethan Feng Keyong Zhou 于2024年1月1日周一 09:45写道： > > +1 (binding) > > I checked > - git commit hash is correct. > - links are valid. > - "incubating" is in the name. > - PGP keys are good. > - hashes are correct. > - LICENSE looks good. > - NOTICE looks good. > - DISCLAIMER exists. > - build success from source code (macOS). ``` ./build/make-distribution.sh > --release ``` > > Thanks, > Keyong Zhou > > Keyong Zhou 于2024年1月1日周一 09:44写道： > > > I checked > > - git commit hash is correct. > > - links are valid. > > - "incubating" is in the name. > > - PGP keys are good. > > - hashes are correct. > > - LICENSE looks good. > > - NOTICE looks good. > > - DISCLAIMER exists. > > - build success from source code (macOS). ``` ./build/make-distribution.sh > > --release ``` > > > > Thanks, > > Keyong Zhou > > > > Nicholas Jiang 于2023年12月29日周五 19:50写道： > > > >> Hi Celeborn community, > >> > >> This is a call for a vote to release Apache Celeborn (Incubating) > >> > >> 0.3.2-incubating-rc2 > >> > >> > >> The git tag to be voted upon: > >> > >> > >> https://github.com/apache/incubator-celeborn/releases/tag/v0.3.2-incubating-rc2 > >> > >> > >> The git commit hash: > >> 0dccad38e28554c36a5eef98de2540d996f946f7 source and binary artifacts can > >> be > >> found at: > >> > >> > >> https://dist.apache.org/repos/dist/dev/incubator/celeborn/v0.3.2-incubating-rc2 > >> > >> > >> The staging repo: > >> https://repository.apache.org/content/repositories/orgapacheceleborn-1048 > >> > >> > >> Fingerprint of the PGP key release artifacts are signed with: > >> D73CADC1DAB63BD3C770BB6D9476842D24B7C885 > >> > >> My public key to verify signatures can be found in: > >> > >> https://dist.apache.org/repos/dist/release/incubator/celeborn/KEYS > >> > >> > >> The vote will be open for at least 72 hours or until the necessary > >> number of votes are reached. > >> > >> > >> Please vote accordingly: > >> > >> > >> [ ] +1 approve > >> [ ] +0 no opinion > >> > >> [ ] -1 disapprove (and the reason) > >> > >> > >> Checklist for release: > >> > >> > >> https://cwiki.apache.org/confluence/display/INCUBATOR/Incubator+Release+Checklist > >> Steps to validate the release: > >> > >> https://www.apache.org/info/verification.html > >> > >> * Download links, checksums and PGP signatures are valid. > >> * Source code distributions have correct names matching the current > >> release. > >> * Release files have the word incubating in their name. > >> * DISCLAIMER, LICENSE and NOTICE files are correct. > >> * All files have license headers if necessary. > >> * No unlicensed compiled archives bundled in source archive. > >> * The source tarball matches the git tag. > >> * Build from source is successful. > >> > >> Regards, > >> Nicholas Jiang > > > >
Re: [VOTE] Release Apache Celeborn(Incubating) 0.4.0-incubating-rc1
+1(binding) I checked: √ release files in the correct location √ release files have the word incubating √ digital signature and hashes correct √ DISCLAIMER file exist √ LICENSE and NOTICE files exist and are correct √ the contents of the release match the tag in VCS √ can build the release from the source √ maven artifacts look correct Thanks, Ethan Feng Fu Chen 于2023年12月23日周六 00:05写道： > > Hi Celeborn community, > > This is a call for a vote to release Apache Celeborn (Incubating) > 0.4.0-incubating-rc1 > > > The git tag to be voted upon: > https://github.com/apache/incubator-celeborn/releases/tag/v0.4.0-incubating-rc1 > > > The git commit hash: > 6efa068c6715e80e4ddee6bc37d6aaf5bf5251fa source and binary artifacts can be > found at: > https://dist.apache.org/repos/dist/dev/incubator/celeborn/v0.4.0-incubating-rc1 > > > The staging repo: > https://repository.apache.org/content/repositories/orgapacheceleborn-1047 > > > Fingerprint of the PGP key release artifacts are signed with: > 92AF4750DAFCB5E25B5B83EA76F54B977EB5C09B > > > My public key to verify signatures can be found in: > https://dist.apache.org/repos/dist/release/incubator/celeborn/KEYS > > > The vote will be open for at least 72 hours or until the necessary > number of votes are reached. > > > Please vote accordingly: > > > [ ] +1 approve > [ ] +0 no opinion > [ ] -1 disapprove (and the reason) > > > Checklist for release: > https://cwiki.apache.org/confluence/display/INCUBATOR/Incubator+Release+Checklist > Steps to validate the release: > https://www.apache.org/info/verification.html > > > * Download links, checksums and PGP signatures are valid. > * Source code distributions have correct names matching the current release. > * Release files have the word incubating in their name. > * DISCLAIMER, LICENSE and NOTICE files are correct. > * All files have license headers if necessary. > * No unlicensed compiled archives bundled in source archive. > * The source tarball matches the git tag. > * Build from source is successful. > > Please be aware that there has been a transition in the Celeborn project's > build tool, shifting from Maven to SBT. The SBT build documentation is > available > at https://celeborn.apache.org/docs/latest/developers/sbt/. > > For illustrative purposes: > > Packaging the project > ``` > ./build/sbt clean package > ``` > > Creating the distribution > ``` > ./build/make-distribution.sh --sbt-enabled --release > ``` > > Thanks, > Fu Chen
Re: [VOTE] Release Apache Celeborn(Incubating) 0.3.2-incubating-rc1
+1(binding) I checked: √ release files in the correct location √ release files have the word incubating √ digital signature and hashes correct √ DISCLAIMER file exist √ LICENSE and NOTICE files exist and are correct √ the contents of the release match the tag in VCS √ can build the release from the source √ maven artifacts looks correct Thanks, Ethan Feng Yihe Li 于2023年12月22日周五 15:18写道： > > +1 (non-binding) > I checked the following things: > - git commit hash is correct. > - download links are valid. > - release files are in correct location. > - release files have the word incubating in their name. > - signatures and checksums are good. > - DISCLAIMER, LICENSE and NOTICE files exist. > - build success from source code(ubuntu 16.04). > ``` > ./build/make-distribution.sh --release > ./build/make-distribution.sh -Pspark-3.3 > ``` > > Thanks, > Yihe Li > > On 2023/12/22 05:37:52 Cheng Pan wrote: > > +1 > > > > - all links are valid > > - checksums and signatures are good > > - artifacts name contain “incubating” > > - build success on Ubuntu 22.04 JDK 17 x86 with the command: > > `build/make-distribution.sh -Pspark-3.3` > > > > Cheng Pan > > > > > On Dec 22, 2023, at 12:16, Shaoyun Chen wrote: > > > > > > +1 (non-binding) > > > > > > I checked the following things: > > > > > > - signatures are good. > > > ``` > > > gpg --import KEYS > > > gpg --verify apache-celeborn-0.3.2-incubating-source.tgz.asc > > > gpg --verify apache-celeborn-0.3.2-incubating-bin.tgz.asc > > > ``` > > > - checksums are good. > > > ``` > > > sha512sum --check apache-celeborn-0.3.2-incubating-source.tgz.sha512 > > > sha512sum --check apache-celeborn-0.3.2-incubating-bin.tgz.sha512 > > > ``` > > > - build success from source code. > > > ``` > > > ./build/make-distribution.sh -Pspark-3.2 > > > ./build/make-distribution.sh --release > > > ``` > > > > > > rexxiong 于2023年12月22日周五 11:09写道： > > >> > > >> +1 (binding) > > >> I checked > > >> - Download links are valid. > > >> - git commit hash is correct > > >> - Checksums and signatures are valid. > > >> - No binary files in the source release > > >> - Files have the word incubating in their name. > > >> - DISCLAIMER,LICENSE and NOTICE files exist. > > >> > > >> Thanks, > > >> Jiashu Xiong > > >> > > >> Fu Chen 于2023年12月21日周四 21:57写道： > > >> > > >>> +1 > > >>> > > >>> I checked > > >>> - download links are valid. > > >>> - git commit hash is correct. > > >>> - no binary files in the source release. > > >>> - signatures are good. > > >>> ``` > > >>> gpg --import KEYS > > >>> gpg --verify apache-celeborn-0.3.2-incubating-source.tgz.asc > > >>> gpg --verify apache-celeborn-0.3.2-incubating-bin.tgz.asc > > >>> ``` > > >>> - checksums are good. > > >>> ``` > > >>> sha512sum --check apache-celeborn-0.3.2-incubating-source.tgz.sha512 > > >>> sha512sum --check apache-celeborn-0.3.2-incubating-bin.tgz.sha512 > > >>> ``` > > >>> - build success from source code (Pop!_OS 22.04 LTS). > > >>> ``` > > >>> ./build/mvn clean package -DskipTests -Pspark-3.4 > > >>> ``` > > >>> > > >>> Keyong Zhou 于2023年12月21日周四 16:55写道： > > >>> > > >>>> +1 (binding) > > >>>> > > >>>> I checked > > >>>> - git commit hash is correct. > > >>>> - links are valid. > > >>>> - "incubating" is in the name. > > >>>> - PGP keys are good. > > >>>> - hashes are correct. > > >>>> - LICENSE looks good. > > >>>> - NOTICE looks good. > > >>>> - DISCLAIMER exists. > > >>>> - build success from source code (macOS). ``` > > >>> ./build/make-distribution.sh > > >>>> --release ``` > > >>>> > > >>>> Thanks, > > >>>> Keyong Zhou > > >>>> > > >>>> Nicholas Jiang 于2023年12月21日周四 14:06写道： > > >>>> > > >>>>> Hi Celeborn community, > > >>>>> > > >>>>> > > >
Re: [VOTE] Release Apache Celeborn(Incubating) 0.4.0-incubating-rc0
-1(binding) There is a bug that needs to be fixed. Thanks, Ethan Feng. rexxiong 于2023年12月22日周五 11:35写道： > > +1 (binding) > I checked > - Download links are valid. > - git commit hash is correct > - Checksums and signatures are valid. > - No binary files in the source release > - Files have the word incubating in their name. > - DISCLAIMER,LICENSE and NOTICE files exist. > - Successfully built the binary from the source on MacOs with Command: > ./build/make-distribution.sh -Pspark-3.3 > > Thanks, > Jiashu Xiong > > Nicholas Jiang 于2023年12月22日周五 11:05写道： > > > +1 (non-binding) > > > > I checked > > - Download links are valid. > > - Checksums and signatures are valid. > > - Git commit hash is correct > > - No binary files in the source release > > - Files have the word incubating in names. > > - DISCLAIMER,LICENSE and NOTICE files exist. > > - Successfully built the binary from the source via command: > > ./build/make-distribution.sh --release > > > > Regards, > > Nicholas Jiang > > > > On 2023/12/21 13:41:36 Fu Chen wrote: > > > Hi Celeborn community, > > > > > > This is a call for a vote to release Apache Celeborn (Incubating) > > > 0.4.0-incubating-rc0 > > > > > > > > > The git tag to be voted upon: > > > > > https://github.com/apache/incubator-celeborn/releases/tag/v0.4.0-incubating-rc0 > > > > > > > > > The git commit hash: > > > de6d8d69af3381ee899ba8d92c5d63b332cbdfbf source and binary artifacts can > > be > > > found at: > > > > > https://dist.apache.org/repos/dist/dev/incubator/celeborn/v0.4.0-incubating-rc0 > > > > > > > > > The staging repo: > > > > > https://repository.apache.org/content/repositories/orgapacheceleborn-1046 > > > > > > > > > Fingerprint of the PGP key release artifacts are signed with: > > > 92AF4750DAFCB5E25B5B83EA76F54B977EB5C09B > > > > > > > > > My public key to verify signatures can be found in: > > > https://dist.apache.org/repos/dist/release/incubator/celeborn/KEYS > > > > > > > > > The vote will be open for at least 72 hours or until the necessary > > > number of votes are reached. > > > > > > > > > Please vote accordingly: > > > > > > > > > [ ] +1 approve > > > [ ] +0 no opinion > > > [ ] -1 disapprove (and the reason) > > > > > > > > > Checklist for release: > > > > > https://cwiki.apache.org/confluence/display/INCUBATOR/Incubator+Release+Checklist > > > Steps to validate the release: > > > https://www.apache.org/info/verification.html > > > > > > > > > * Download links, checksums and PGP signatures are valid. > > > * Source code distributions have correct names matching the current > > release. > > > * Release files have the word incubating in their name. > > > * DISCLAIMER, LICENSE and NOTICE files are correct. > > > * All files have license headers if necessary. > > > * No unlicensed compiled archives bundled in source archive. > > > * The source tarball matches the git tag. > > > * Build from source is successful. > > > > > > Please be aware that there has been a transition in the Celeborn > > project's > > > build tool, shifting from Maven to SBT. The SBT build documentation is > > > available > > > at https://celeborn.apache.org/docs/latest/developers/sbt/. > > > > > > For illustrative purposes: > > > > > > Packaging the project > > > ``` > > > ./build/sbt clean package > > > ``` > > > > > > Creating the distribution > > > ``` > > > ./build/make-distribution.sh --sbt-enabled --release > > > ``` > > > > > > Thanks, > > > Fu Chen > > > > >
Re: [VOTE] Release Apache Celeborn(Incubating) 0.3.2-incubating-rc0
Due to a bug found yesterday. Update my vote to -1(binding). Thanks, Ethan Feng angers zhu 于2023年12月20日周三 11:39写道： > > +1 > > - Checked build with all version spark > > Thanks Angerszh > > Cheng Pan 于2023年12月20日周三 10:48写道： > > > +1 > > > > - Checked all links are valid > > - GitHub CI result is good > > > > Thanks, > > Cheng Pan > > > > > On Dec 20, 2023, at 10:38, Mridul Muralidharan wrote: > > > > > > +1 > > > > > > Signatures, digests, license, etc check out fine. > > > Checked out tag and build/tested with -Pspark3.1 and -Pflink-1.17 > > > > > > Regards, > > > Mridul > > > > > > > > > On Tue, Dec 19, 2023 at 8:06 PM rexxiong wrote: > > > > > >> +1 (binding) > > >> I checked > > >> - Download links are valid. > > >> - git commit hash is correct > > >> - Checksums and signatures are valid. > > >> - No binary files in the source release > > >> - Files have the word incubating in their name. > > >> - DISCLAIMER,LICENSE and NOTICE files exist. > > >> - Successfully built the binary from the source on MacOs with Command: > > >> ./build/make-distribution.sh --release > > >> > > >> Thanks, > > >> Jiashu Xiong > > >> > > >> Fu Chen 于2023年12月20日周三 00:18写道： > > >> > > >>> +1 > > >>> > > >>> I checked > > >>> - download links are valid. > > >>> - git commit hash is correct. > > >>> - no binary files in the source release. > > >>> - signatures are good. > > >>> ``` > > >>> gpg --import KEYS > > >>> gpg --verify apache-celeborn-0.3.2-incubating-source.tgz.asc > > >>> gpg --verify apache-celeborn-0.3.2-incubating-bin.tgz.asc > > >>> ``` > > >>> - checksums are good. > > >>> ``` > > >>> sha512sum --check apache-celeborn-0.3.2-incubating-source.tgz.sha512 > > >>> sha512sum --check apache-celeborn-0.3.2-incubating-bin.tgz.sha512 > > >>> ``` > > >>> - build success from source code (Pop!_OS 22.04 LTS). > > >>> ``` > > >>> ./build/mvn clean package -DskipTests -Pspark-3.4 > > >>> ``` > > >>> > > >>> Shaoyun Chen 于2023年12月19日周二 22:45写道： > > >>> > > >>>> +1 (non-binding) > > >>>> > > >>>> I checked the following things: > > >>>> > > >>>> - signatures are good. > > >>>> ``` > > >>>> gpg --import KEYS > > >>>> gpg --verify apache-celeborn-0.3.2-incubating-source.tgz.asc > > >>>> gpg --verify apache-celeborn-0.3.2-incubating-bin.tgz.asc > > >>>> ``` > > >>>> - checksums are good. > > >>>> ``` > > >>>> sha512sum --check apache-celeborn-0.3.2-incubating-source.tgz.sha512 > > >>>> sha512sum --check apache-celeborn-0.3.2-incubating-bin.tgz.sha512 > > >>>> ``` > > >>>> - build success from source code. > > >>>> ``` > > >>>> ./build/make-distribution.sh -Pspark-3.2 > > >>>> ./build/make-distribution.sh --release > > >>>> ``` > > >>>> > > >>>> Yihe Li 于2023年12月19日周二 20:56写道： > > >>>>> > > >>>>> +1 (non-binding) > > >>>>> I checked the following things: > > >>>>> - git commit hash is correct. > > >>>>> - download links are valid. > > >>>>> - release files are in correct location. > > >>>>> - release files have the word incubating in their name. > > >>>>> - signatures and checksums are good. > > >>>>> - DISCLAIMER, LICENSE and NOTICE files exist. > > >>>>> - build success from source code(ubuntu 16.04). > > >>>>> ``` > > >>>>> ./build/make-distribution.sh --release > > >>>>> ./build/make-distribution.sh --Pspark-3.3 > > >>>>> ``` > > >>>>> > > >>>>> Thanks, > > >>>>> Yihe Li > > >>>>> > > >>>>> On 2023/12/19 06:37:40 Ethan Feng wrote: > > >>>>>> +1(binding) > > >>>>>&
Re: [VOTE] Release Apache Celeborn(Incubating) 0.3.2-incubating-rc0
+1(binding) I checked: √ release files in the correct location √ release files have the word incubating √ digital signature and hashes correct √ DISCLAIMER file exist √ LICENSE and NOTICE files exist and are correct √ the contents of the release match the tag in VCS √ can build the release from the source √ maven artifacts looks correct Thanks, Ethan Feng Nicholas Jiang 于2023年12月19日周二 12:32写道： > > Hi Celeborn community, > > > This is a call for a vote to release Apache Celeborn (Incubating) > 0.3.2-incubating-rc0 > > > The git tag to be voted upon: > https://github.com/apache/incubator-celeborn/releases/tag/v0.3.2-incubating-rc0 > > > The git commit hash: > d43411b22adf24679c27004a08e813ab278eaaa3 source and binary artifacts can be > found at: > https://dist.apache.org/repos/dist/dev/incubator/celeborn/v0.3.2-incubating-rc0 > > > The staging repo: > https://repository.apache.org/content/repositories/orgapacheceleborn-1041 > > > Fingerprint of the PGP key release artifacts are signed with: > D73CADC1DAB63BD3C770BB6D9476842D24B7C885 > > > My public key to verify signatures can be found in: > https://dist.apache.org/repos/dist/release/incubator/celeborn/KEYS > > > The vote will be open for at least 72 hours or until the necessary > number of votes are reached. > > > Please vote accordingly: > > > [ ] +1 approve > [ ] +0 no opinion > [ ] -1 disapprove (and the reason) > > > Checklist for release: > https://cwiki.apache.org/confluence/display/INCUBATOR/Incubator+Release+Checklist > > > Steps to validate the release: > https://www.apache.org/info/verification.html > > > * Download links, checksums and PGP signatures are valid. > * Source code distributions have correct names matching the current release. > * Release files have the word incubating in their name. > * DISCLAIMER, LICENSE and NOTICE files are correct. > * All files have license headers if necessary. > * No unlicensed compiled archives bundled in source archive. > * The source tarball matches the git tag. > * Build from source is successful. > > > Regards, > Nicholas Jiang
Re: [DISCUSS] Time for 0.3.2
Hey Nicholas, Thanks for the volunteering. I completely agree that we should prepare for the 0.3.2 release. Thanks, Ethan Feng Nicholas 于2023年12月7日周四 14:31写道： > > Hey, Celeborn community, > > It has been a while since the 0.3.1 release, and there are some critical > fixes land branch-0.3, for example, [CELEBORN-1037] Incorrect output for > metrics of Prometheus. From my perspective, it’s time to prepare for > releasing 0.3.2. > > WDYT? And I’m volunteering to be the release manager if no one has applied. > > Regards, > Nicholas Jiang
Re: Add k8s operator for celeborn
Hi xleoken, I completely agree with your idea. We can definitely use a Kubernetes operator for Celeborn, and it would help us manage the Celeborn clusters more effectively. Thank you for bringing this up, I found that you had already completed the Kubernetes operator on your branch. Can you share your design docs and contribute it to the community? Best regards, Ethan Feng leo65535 于2023年11月8日周三 16:54写道： > > > > Hello everyone, > > > As we known, kubernetes is designed for automation, Kubernetes' operator > pattern concept > lets you extend the cluster's behaviour without modifying the code of > Kubernetes itself > by linking controllers to one or more custom resources. Operators are clients > of the > Kubernetes API that act as controllers for a Custom Resource. > > > So we can use k8s operator sdk to write our own operator for celeborn. the > operator can > help us to manage the celeborn clusters. > > > Please check the jira[1] for more detail. > > > Looking forward to your feedback and suggestions, thanks. > > > Bests, > xleoken > > > [1] > https://issues.apache.org/jira/browse/CELEBORN-1117 >
Re: [DISCUSSION] Support memory file storage.
Hi Gabriel, Thank you for bringing this to my attention. I confirm that I will change the authority of the Google Doc to allow you to write a comment. I apologize for any inconvenience this may have caused. Regards, Ethan Gabriel Lee 于2023年9月21日周四 10:50写道： > > Hi Ethan, > > After viewing this Google Doc, I noticed I don't have access to write a > comment. Could you please change this doc's authority ? > > Best, > Gabriel > > > On Wed, 20 Sept 2023 at 11:27, Ethan Feng wrote: > > > Hello Celeborn community, > > > > I have a proposal to support memory file storage in Celeborn: > > > > https://docs.google.com/document/d/1SM-oOM0JHEIoRHTYhE9PYH60_1D3NMxDR50LZIM7uW0/edit?usp=sharing > > > > Would really appreciate feedback from the community on this proposal. > > > > > > Thanks > > Ethan > >
Re: [DISCUSSION] Support memory file storage.
Hi Mridul, Thank you for your email and your positive feedback on the proposed enhancement to Celeborn. I'm glad you find it promising. To address your queries: a) The proposed enhancement is intended to act as a storage tire, not as a cache. However, it may have certain elements of both. Celeborn currently won't store a whole shuffle file in memory and requires a shuffle file to be written to disks or HDFS before the client can read. This proposal will allow the client to read a shuffle file from the worker's memory directly. I hope this clarifies things for you. b) While your suggestion for a tiered storage layer is interesting, it is a superset of this proposal. As you can see there is an issue(https://github.com/apache/incubator-celeborn/issues/146). Celeborn treats a shuffle partition as a shuffle file instead of segments so a shuffle partition will not be distributed to multiple storage tiers. There will be another proposal to discuss how will Celeborn move existing shuffle files to different storage tires. c) As mentioned above, the enhancement is intended to act as a storage tier that's why I explained the details about how it is handled internally. Thanks again for your email. Please let me know if you have any further questions or concerns. Regards, Ethan Mridul Muralidharan 于2023年9月21日周四 01:09写道： > > Hi, > > This should be a nontrivial improvement to Celeborn imo, thanks Ethan ! > > I had a few queries: > > a) Are we viewing this enhancement as a cache or as a tiered storage layer ? > When going over it, I felt the proposal might be doing both - though > leaning more as a cache, but wanted to get clarity. > > b) If we are modelling it as a tiered storage layer, it would be good to > also think about what the right abstractions should be and not special case > it just for memory. > For example: > Memory -> NVME/SSD -> Spinning Disk -> HDFS/S3 > (With one or more being missing in a deployment) > > This would unify the way we handle evictions from one level to the next > with a tiered view of the storage layer. > Complexity of the implementation is definitely a consideration here though. > > Note, this might be out of scope for this proposal and work for the future > as well - wanted to get your thoughts if it was considered ! > > c) If modelling as a cache, we should change the abstractions in the > proposal slightly and hide the details behind the cache implementation. > Read and write path would not need to worry about how it is handled > internally. > > > Regards, > Mridul > > > > > > On Tue, Sep 19, 2023 at 10:27 PM Ethan Feng wrote: > > > Hello Celeborn community, > > > > I have a proposal to support memory file storage in Celeborn: > > > > https://docs.google.com/document/d/1SM-oOM0JHEIoRHTYhE9PYH60_1D3NMxDR50LZIM7uW0/edit?usp=sharing > > > > Would really appreciate feedback from the community on this proposal. > > > > > > Thanks > > Ethan > >
[DISCUSSION] Support memory file storage.
Hello Celeborn community, I have a proposal to support memory file storage in Celeborn: https://docs.google.com/document/d/1SM-oOM0JHEIoRHTYhE9PYH60_1D3NMxDR50LZIM7uW0/edit?usp=sharing Would really appreciate feedback from the community on this proposal. Thanks Ethan
[NEW FEATURE AVAILABLE] Celeborn support MapReduce engine.
Hello developers and users, I am glad to announce that Celeborn supports the MapReduce engine now. Both Hadoop 2 and 3 are supported. If you are interested, you can just try it and feedback on anything you want. The quick start guide can be found here [ https://celeborn.apache.org/docs/latest/ ]. The design doc can be found here [ https://docs.google.com/document/d/1g4irlBucIAFNI42cFSuOVWYqOWSvuqpw_VBHmyyv8zo/edit?usp=sharing ]. Thanks, Ethan.
Re: [VOTE] Release Apache Celeborn(Incubating) 0.3.1-incubating-rc2
+1 (binding) [√] incubating in name [√] signature and hash fine [√] DISCLAIMER is fine [√] LICENSE and NOTICE are fine [√] No unexpected binary files [√] All source files have ASF headers [√] Can be built from source [√] Results are correct on 1TB TPC-DS Thanks, Ethan Binjie Yang 于2023年9月13日周三 16:18写道： > > +1 > > Checked: > - build binary package with ./build/make-distribution.sh -Pspark-3.3 > - check git commit in RELEASE file > - Build Docker Image with docker build -f docker/Dockerfile . > > Thanks, > Binjie Yang > > On 2023/09/12 07:12:44 Fu Chen wrote: > > +1 > > > > I checked > > - download links are valid. > > - git commit hash is correct. > > - no binary files in the source release. > > - signatures are good. > > ``` > > gpg --import KEYS > > gpg --verify apache-celeborn-0.3.1-incubating-source.tgz.asc > > gpg --verify apache-celeborn-0.3.1-incubating-bin.tgz.asc > > ``` > > - checksums are good. > > ``` > > sha512sum --check apache-celeborn-0.3.1-incubating-source.tgz.sha512 > > sha512sum --check apache-celeborn-0.3.1-incubating-bin.tgz.sha512 > > ``` > > - build success from source code (ubuntu 18.04). > > ``` > > ./build/mvn clean package -DskipTests -Pspark-3.3 > > ``` > > > > > > Shaoyun Chen 于2023年9月12日周二 11:01写道： > > > > > +1 (non-binding) > > > > > > I checked the following things: > > > > > > - signatures are good. > > > ``` > > > gpg --import KEYS > > > gpg --verify apache-celeborn-0.3.1-incubating-source.tgz.asc > > > gpg --verify apache-celeborn-0.3.1-incubating-bin.tgz.asc > > > ``` > > > - checksums are good. > > > ``` > > > sha512sum --check apache-celeborn-0.3.1-incubating-source.tgz.sha512 > > > sha512sum --check apache-celeborn-0.3.1-incubating-bin.tgz.sha512 > > > ``` > > > - build success from source code. > > > ``` > > > ./build/make-distribution.sh -Pspark-3.2 > > > ``` > > > > > > Keyong Zhou 于2023年9月12日周二 09:57写道： > > > > > > > > +1 (binding) > > > > > > > > I checked > > > > - git commit hash is correct. > > > > - links are valid. > > > > - "incubating" is in the name. > > > > - PGP keys are good. > > > > - hashes are correct. > > > > - LICENSE looks good. > > > > - NOTICE looks good. > > > > - DISCLAIMER exists. > > > > - build success from source code (macOS). ``` > > > ./build/make-distribution.sh > > > > --release ``` > > > > > > > > Thanks, > > > > Keyong Zhou > > > > > > > > rexxiong 于2023年9月11日周一 22:08写道： > > > > > > > > > +1 (binding) > > > > > I checked > > > > > - Download links are valid. > > > > > - git commit hash is correct > > > > > - Checksums and signatures are valid. > > > > > - No binary files in the source release > > > > > - Files have the word incubating in their name. > > > > > - DISCLAIMER,LICENSE and NOTICE files exist. > > > > > - Successfully built the binary from the source on MacOs with Command: > > > > > ./build/make-distribution.sh --release > > > > > > > > > > Thanks, > > > > > Jiashu Xiong > > > > > > > > > > Cheng Pan 于2023年9月11日周一 16:27写道： > > > > > > > > > > > Hi Celeborn community, > > > > > > > > > > > > This is a call for a vote to release Apache Celeborn (Incubating) > > > > > > 0.3.1-incubating-rc2 > > > > > > > > > > > > The git tag to be voted upon: > > > > > > > > > > > > > > > > > > > > https://github.com/apache/incubator-celeborn/releases/tag/v0.3.1-incubating-rc2 > > > > > > > > > > > > The git commit hash: > > > > > > 7ec5596748af49ef9cb429d08550e89d94d5cc74 > > > > > > > > > > > > The source and binary artifacts can be found at: > > > > > > > > > > > > > > > > > > > > https://dist.apache.org/repos/dist/dev/incubator/celeborn/v0.3.1-incubating-rc2 > > > > > > > > > > > > The staging repo: > > > > > > > > > > > > > > https://repository.apache.org/content/repositories/orgapacheceleborn-1039 > > > > > > > > > > > > Fingerprint of the PGP key release artifacts are signed with: > > > > > > 8FC8075E1FDC303276C676EE8001952629BCC75D > > > > > > > > > > > > My public key to verify signatures can be found in: > > > > > > https://dist.apache.org/repos/dist/release/incubator/celeborn/KEYS > > > > > > > > > > > > The vote will be open for at least 72 hours or until the necessary > > > > > > number of votes are reached. > > > > > > > > > > > > Please vote accordingly: > > > > > > > > > > > > [ ] +1 approve > > > > > > [ ] +0 no opinion > > > > > > [ ] -1 disapprove (and the reason) > > > > > > > > > > > > Checklist for release: > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/INCUBATOR/Incubator+Release+Checklist > > > > > > > > > > > > Steps to validate the release: > > > > > > https://www.apache.org/info/verification.html > > > > > > > > > > > > Instructions for making binary artifacts from source: > > > > > > build/make-distribution.sh --release > > > > > > > > > > > > Thanks, > > > > > > Cheng Pan > > > > > > > > > >
Re: [VOTE] Release Apache Celeborn(Incubating) 0.3.1-incubating-rc0
The Jira ticket is CELEBORN-941. I fixed it yesterday but I didn't realize that merge_pr.sh failed to copy it into branch-0.3. Regards, Ethan Cheng Pan 于2023年9月1日周五 15:39写道： > > Ethan, could you please create the corresponding JIRA ticket(s) and set the > priority to blocker? > > Thanks, > Cheng Pan > > > > On Sep 1, 2023, at 15:33, Ethan Feng wrote: > > > > -1(bingding) > > The deploy doc in this RC needs to be updated. > > > > Regards, > > Ethan > > > > 在 2023年9月1日星期五，Mridul Muralidharan 写道： > > > >> +1 > >> > >> Signatures, digests, license, etc check out fine. > >> Checked out tag and build/tested with -Pspark-3.1 > >> > >> Regards, > >> Mridul > >> > >> > >> On Thu, Aug 31, 2023 at 11:35 AM Cheng Pan wrote: > >> > >>> Hi Celeborn community, > >>> > >>> This is a call for a vote to release Apache Celeborn (Incubating) > >>> 0.3.1-incubating-rc0 > >>> > >>> The git tag to be voted upon: > >>> > >>> https://github.com/apache/incubator-celeborn/releases/tag/ > >> v0.3.1-incubating-rc0 > >>> > >>> The git commit hash: > >>> b3992274e207959125d8784d9b61a6e8043612fc source and binary artifacts > >> can be > >>> found at: > >>> > >>> https://dist.apache.org/repos/dist/dev/incubator/celeborn/v0 > >> .3.1-incubating-rc0 > >>> > >>> The staging repo: > >>> https://repository.apache.org/content/repositories/orgapache > >> celeborn-1037 > >>> > >>> Fingerprint of the PGP key release artifacts are signed with: > >>> 8FC8075E1FDC303276C676EE8001952629BCC75D > >>> > >>> My public key to verify signatures can be found in: > >>> https://dist.apache.org/repos/dist/release/incubator/celeborn/KEYS > >>> > >>> The vote will be open for at least 72 hours or until the necessary > >>> number of votes are reached. > >>> > >>> Please vote accordingly: > >>> > >>> [ ] +1 approve > >>> [ ] +0 no opinion > >>> [ ] -1 disapprove (and the reason) > >>> > >>> Checklist for release: > >>> > >>> https://cwiki.apache.org/confluence/display/INCUBATOR/Incuba > >> tor+Release+Checklist > >>> Steps to validate the release: > >>> https://www.apache.org/info/verification.html > >>> > >>> * Download links, checksums and PGP signatures are valid. > >>> * Source code distributions have correct names matching the current > >>> release. > >>> * Release files have the word incubating in their name. > >>> * DISCLAIMER, LICENSE and NOTICE files are correct. > >>> * All files have license headers if necessary. > >>> * No unlicensed compiled archives bundled in source archive. > >>> * The source tarball matches the git tag. > >>> * Build from source is successful. > >>> > >>> Thanks, > >>> Cheng Pan > >>> > >> >
[VOTE] Release Apache Celeborn(Incubating) 0.3.1-incubating-rc0
-1(bingding) The deploy doc in this RC needs to be updated. Regards, Ethan 在 2023年9月1日星期五，Mridul Muralidharan 写道： > +1 > > Signatures, digests, license, etc check out fine. > Checked out tag and build/tested with -Pspark-3.1 > > Regards, > Mridul > > > On Thu, Aug 31, 2023 at 11:35 AM Cheng Pan wrote: > > > Hi Celeborn community, > > > > This is a call for a vote to release Apache Celeborn (Incubating) > > 0.3.1-incubating-rc0 > > > > The git tag to be voted upon: > > > > https://github.com/apache/incubator-celeborn/releases/tag/ > v0.3.1-incubating-rc0 > > > > The git commit hash: > > b3992274e207959125d8784d9b61a6e8043612fc source and binary artifacts > can be > > found at: > > > > https://dist.apache.org/repos/dist/dev/incubator/celeborn/v0 > .3.1-incubating-rc0 > > > > The staging repo: > > https://repository.apache.org/content/repositories/orgapache > celeborn-1037 > > > > Fingerprint of the PGP key release artifacts are signed with: > > 8FC8075E1FDC303276C676EE8001952629BCC75D > > > > My public key to verify signatures can be found in: > > https://dist.apache.org/repos/dist/release/incubator/celeborn/KEYS > > > > The vote will be open for at least 72 hours or until the necessary > > number of votes are reached. > > > > Please vote accordingly: > > > > [ ] +1 approve > > [ ] +0 no opinion > > [ ] -1 disapprove (and the reason) > > > > Checklist for release: > > > > https://cwiki.apache.org/confluence/display/INCUBATOR/Incuba > tor+Release+Checklist > > Steps to validate the release: > > https://www.apache.org/info/verification.html > > > > * Download links, checksums and PGP signatures are valid. > > * Source code distributions have correct names matching the current > > release. > > * Release files have the word incubating in their name. > > * DISCLAIMER, LICENSE and NOTICE files are correct. > > * All files have license headers if necessary. > > * No unlicensed compiled archives bundled in source archive. > > * The source tarball matches the git tag. > > * Build from source is successful. > > > > Thanks, > > Cheng Pan > > >
Re: [DISCUSS] Allow external contributors to run CI without approval
Thanks for your support, I'll come up with a JIRA ticket to change this setting to "only requires approval first time". Thanks, Ethan Mridul Muralidharan 于2023年6月16日周五 22:24写道： > > Agree, +1 > > Regards, > Mridul > > On Fri, Jun 16, 2023 at 9:16 AM Cheng Pan wrote: > > > +1 for "only requires approval first time" > > > > Keyong Zhou 于 2023年6月16日周五 下午5:48写道： > > > > > +1 > > > > > > Thanks, > > > Keyong Zhou > > > > > > Ethan Feng 于2023年6月16日周五 16:27写道： > > > > > > > Recent moves by Apache Infra have changed the policy on GitHub Actions > > > from > > > > "Only requires approval first time" to "Requires approval every time". > > > > > > > > I think this is not friendly for getting folks involved in > > > > the project and this increased the cost for committers to process the > > > > pull requests. > > > > > > > > Please respond to this thread if you are in support of going back to > > > > "Only requires approval the first time" or if you don't believe this > > is a > > > > good idea please respond as well. > > > > > > > > Thanks, > > > > Ethan Feng > > > > > > > > >
[DISCUSS] Allow external contributors to run CI without approval
Recent moves by Apache Infra have changed the policy on GitHub Actions from "Only requires approval first time" to "Requires approval every time". I think this is not friendly for getting folks involved in the project and this increased the cost for committers to process the pull requests. Please respond to this thread if you are in support of going back to "Only requires approval the first time" or if you don't believe this is a good idea please respond as well. Thanks, Ethan Feng
Re: [VOTE] Release Apache Celeborn(Incubating) 0.2.1-incubating-rc0
+1 (non-binding) All checks are done and waiting for the incubator community to proceed. Ethan Feng rexxiong 于2023年3月20日周一 21:47写道： > > Hello Incubator Community, > > This is a call for a vote to release Apache Celeborn(Incubating) version > 0.2.1-incubating-rc0 > > The Apache Celeborn community has voted on and approved a proposal to > release > Apache Celeborn(Incubating) version 0.2.1-incubating-rc0 > > We now kindly request the Incubator PMC members review and vote on this > incubator release. > > Celeborn community vote thread: > • https://lists.apache.org/thread/7vnm4p3dv9lq1cklj5xh1fzrho381bzy > > Vote result thread: > • https://lists.apache.org/thread/21gjf2v5wn85h37dwdv4yvhbxz48d37r > > The release candidate: > • > https://dist.apache.org/repos/dist/dev/incubator/celeborn/v0.2.1-incubating-rc0 > > Git tag for the release: > • > https://github.com/apache/incubator-celeborn/releases/tag/v0.2.1-incubating-rc0 > > Public keys file: > • https://dist.apache.org/repos/dist/release/incubator/celeborn/KEYS > > The change log is available in: > • > https://github.com/apache/incubator-celeborn/compare/v0.2.0-incubating...v0.2.1-incubating-rc0 > > The vote will be open for at least 72 hours or until the necessary number > of votes are reached. > > Please vote accordingly: > [ ] +1 approve > [ ] +0 no opinion > [ ] -1 disapprove with the reason > > More detailed checklist please refer: > • > https://cwiki.apache.org/confluence/display/INCUBATOR/Incubator+Release+Checklist > > Steps to validate the release， Please refer to: > • https://www.apache.org/info/verification.html > > Thanks, > rexxiong
Re: [VOTE] Release Apache Celeborn(Incubating) 0.2.1-incubating-rc0
+1 (binding) I checked the following items. [√] incubating in name [√] signature and hash fine [√] DISCLAIMER is fine [√] LICENSE and NOTICE are fine [√] No unexpected binary files [√] All source files have ASF headers [√] Can be built from source(profile spark-3.3) [√] Maven artifacts are fine Thanks, Ethan Feng 陈中强 于2023年3月19日周日 10:12写道： > > +1 (binding) > > > > > > > > I have checked the followings: > > > > > 1. PGP signatures are valid. > > wget https://dist.apache.org/repos/dist/release/incubator/celeborn/KEYS > > gpg --import KEYS > > gpg --verify apache-celeborn-0.2.1-incubating-bin.tgz.asc > > gpg --verify apache-celeborn-0.2.1-incubating-source.tgz.asc > > > > > 2. checksums are correct. > > sha512sum --check apache-celeborn-0.2.1-incubating-bin.tgz.sha512 > > sha512sum --check apache-celeborn-0.2.1-incubating-source.tgz.sha512 > > > > > 3. Source code distributions have correct names matching the current release. > > > > > 4. Release files have the word incubating in their name. > > > > > 5. DISCLAIMER, LICENSE and NOTICE files are correct. > > > > > 6. All files have license headers if necessary. > > > > > 7. No unlicensed compiled archives bundled in source archive. > > > > > 8. The source tarball matches the git tag. > > > > > 9. Build from source is successful. > > > > > > > > > > > > > > > > > > At 2023-03-17 17:17:25, "rexxiong" wrote: > >Hi Celeborn community, > > > >This is a call for a vote to release Apache Celeborn (Incubating) > >0.2.1-incubating-rc0 > > > >The git tag to be voted upon: > >https://github.com/apache/incubator-celeborn/releases/tag/v0.2.1-incubating-rc0 > > > >The git commit hash: > >93898d0020899ba7e98ededaad14a6043cec07c9 > > > >The source and binary artifacts can be found at: > >https://dist.apache.org/repos/dist/dev/incubator/celeborn/v0.2.1-incubating-rc0 > > > >The staging repo: > >https://repository.apache.org/content/repositories/orgapacheceleborn-1010 > > > >Fingerprint of the PGP key release artifacts are signed with: > >B4AF9302A52006F3711C784388BDDBB8C6724EC9 > > > >My public key to verify signatures can be found in: > >https://dist.apache.org/repos/dist/release/incubator/celeborn/KEYS > > > >The vote will be open for at least 72 hours or until the necessary > >number of votes are reached. > > > >Please vote accordingly: > > > >[ ] +1 approve > >[ ] +0 no opinion > >[ ] -1 disapprove (and the reason) > > > >Checklist for release: > >https://cwiki.apache.org/confluence/display/INCUBATOR/Incubator+Release+Checklist > >Steps to validate the release: > >https://www.apache.org/info/verification.html > > > >* Download links, checksums and PGP signatures are valid. > >* Source code distributions have correct names matching the current release. > >* Release files have the word incubating in their name. > >* DISCLAIMER, LICENSE and NOTICE files are correct. > >* All files have license headers if necessary. > >* No unlicensed compiled archives bundled in source archive. > >* The source tarball matches the git tag. > >* Build from source is successful. > > > >Thanks, > >rexxiong
[NOTICE] Fix solution about rare data loss in release 0.2.0.
Hello users, Regretfully to inform you that we found a bug[2] in release 0.2.0 yesterday. The bug[2] caused data loss rarely when reading from skew partitions on a high-pressure cluster. You need to apply this patch[1] to your Celeborn client jar. We'll ship this patch in our next release. Feel free to contact us if you encounter any other questions. Regards, Ethan Feng --- 1. https://github.com/apache/incubator-celeborn/pull/1315 2. https://issues.apache.org/jira/browse/CELEBORN-383
[ANNOUNCE] Apache Celeborn(incubating) 0.2.0 available
Hi all, Apache Celeborn(Incubating) community is glad to announce the new release of Apache Celeborn(Incubating) 0.2.0 Celeborn is dedicated to improving the efficiency and elasticity of different map-reduce engines and provides an elastic, high-efficient service for intermediate data including shuffle data, spilled data, result data, etc. Currently Celeborn supports Spark full-featured and improves Spark job's performance, stability and elasticity. Download Link: https://celeborn.apache.org/download/ GitHub Release Tag: - https://github.com/apache/incubator-celeborn/releases/tag/v0.2.0-incubating-rc5 Release Notes: - https://celeborn.apache.org/community/release_notes/release_note_0.2.0 Website: https://celeborn.apache.org/ Celeborn Resources: - Issue: https://issues.apache.org/jira/projects/CELEBORN - Mailing list: dev@celeborn.apache.org Ethan Feng On behalf of the Apache Celeborn(incubating) community
[RESULT][VOTE] Release Apache Celeborn(incubating) 0.2.0-rc5
Hello Incubator Community, The vote to release Apache Celeborn(incubating) 0.2.0-rc5 has passed with 5 +1 and no +0 or -1 votes. (* = binding) +1: - Justin Mclean* - Kent Yao - Yu Li* - Willem Jiang* - Gang Li* The voting thread: https://lists.apache.org/thread/q8hchs4n2twhjl0qf3rn0ln0hyrvrnnp Thanks for reviewing and voting for our release candidate. We will proceed with publishing the approved artifacts and sending out the announcement soon. Regards, Ethan Feng
[VOTE] Release Apache Celeborn(Incubating) 0.2.0-incubating-rc5
Hello Incubator Community, This is a call for a vote to release Apache Celeborn(Incubating) version 0.2.0-incubating-rc5 The Apache Celeborn community has voted on and approved a proposal to release Apache Celeborn(Incubating) version 0.2.0-incubating-rc5 We now kindly request the Incubator PMC members review and vote on this incubator release. Celeborn community vote thread: • https://lists.apache.org/thread/6f28tyvqsqdpjr0wpg2low4hlh39bx4j Vote result thread: • https://lists.apache.org/thread/9bhflkrwrmk81w1stbhvztzc6mb6gjco The release candidate: • https://dist.apache.org/repos/dist/dev/incubator/celeborn/v0.2.0-incubating-rc5 Git tag for the release: • https://github.com/apache/incubator-celeborn/releases/tag/v0.2.0-incubating-rc5 Public keys file: • https://dist.apache.org/repos/dist/release/incubator/celeborn/KEYS The change log is available in: • https://github.com/apache/incubator-celeborn/compare/v0.1.4...v0.2.0-incubating-rc5 The vote will be open for at least 72 hours or until the necessary number of votes are reached. Please vote accordingly: [ ] +1 approve [ ] +0 no opinion [ ] -1 disapprove with the reason More detailed checklist please refer: • https://cwiki.apache.org/confluence/display/INCUBATOR/Incubator+Release+Checklist Steps to validate the release， Please refer to: • https://www.apache.org/info/verification.html Thanks, Ethan Feng
Re: [RESULT][VOTE] Release Apache Celeborn(Incubating) 0.2.0-incubating-rc5
Hi Celeborn community, The vote closes now as 72hr have passed. The vote PASSES with: (* = binding) +1: Ethan Feng* Kerwin Zhang* zhqiangchen Keyong Zhou* Angers Zhu* wangsheng There are no 0 or -1 votes. The voting thread: https://lists.apache.org/thread/6f28tyvqsqdpjr0wpg2low4hlh39bx4j I will now bring the vote to gene...@incubator.apache.org to get approval by the IPMC. If this vote passes, the release is accepted and published. Thanks, Ethan Feng Ethan Feng 于2023年2月24日周五 10:04写道： > > Hi Celeborn community, > > The vote closes now as 72hr have passed. The vote PASSES with: > > (* = binding) > > +1: > > Ethan Feng* > > Kerwin Zhang* > > zhqiangchen > > Keyong Zhou* > > Angers Zhu* > > wangsheng > > There are no 0 or -1 votes. > > The voting thread: > https://lists.apache.org/thread/7ldgny2nnvops0bcy0l7vjqjkn7nb63r > > I will now bring the vote to gene...@incubator.apache.org to get > approval by the IPMC. > If this vote passes, the release is accepted and published. > > Thanks, > Ethan Feng
[RESULT][VOTE] Release Apache Celeborn(Incubating) 0.2.0-incubating-rc5
Hi Celeborn community, The vote closes now as 72hr have passed. The vote PASSES with: (* = binding) +1: Ethan Feng* Kerwin Zhang* zhqiangchen Keyong Zhou* Angers Zhu* wangsheng There are no 0 or -1 votes. The voting thread: https://lists.apache.org/thread/7ldgny2nnvops0bcy0l7vjqjkn7nb63r I will now bring the vote to gene...@incubator.apache.org to get approval by the IPMC. If this vote passes, the release is accepted and published. Thanks, Ethan Feng
[VOTE] Release Apache Celeborn(Incubating) 0.2.0-incubating-rc5
Hi Celeborn community, This is a call for a vote to release Apache Celeborn (Incubating) 0.2.0-incubating-rc5 The git tag to be voted upon: https://github.com/apache/incubator-celeborn/releases/tag/v0.2.0-incubating-rc5 The git commit hash: 52527756c3d2262cfd3e313272b82216c505c7e2 The source and binary artifacts can be found at: https://dist.apache.org/repos/dist/dev/incubator/celeborn/v0.2.0-incubating-rc5/ Fingerprint of the PGP key release artifacts are signed with: FCF20BB29C7BEFDF58F998F76392F71F37356FA0 My public key to verify signatures can be found in: https://dist.apache.org/repos/dist/release/incubator/celeborn/KEYS The vote will be open for at least 72 hours or until necessary number of votes are reached. Please vote accordingly: [ ] +1 approve [ ] +0 no opinion [ ] -1 disapprove (and the reason) Checklist for release: https://cwiki.apache.org/confluence/display/INCUBATOR/Incubator+Release+Checklist Steps to validate the release: https://www.apache.org/info/verification.html Starting with my +1 (binding): * Download links, checksums and PGP signatures are valid. * Source code distributions have correct names matching the current release. * Release files have the word incubating in their name. * DISCLAIMER, LICENSE and NOTICE files are correct. * All files have license headers if necessary. * No unlicensed compiled archives bundled in source archive. * The source tarball matches the git tag. * Build from source is successful. Thanks, Ethan Feng
Re: [VOTE] Release Apache Celeborn(Incubating) 0.2.0-incubating-rc5
Hi Celeborn community, Due to some issues that we fixed recently. The RC5 will be remade. This vote is closed. Ethan Feng Ethan Feng 于2023年2月13日周一 16:47写道： > > Hi Celeborn community, > > This is a call for a vote to release Apache Celeborn (Incubating) > 0.2.0-incubating-rc5 > > The git tag to be voted upon: > https://github.com/apache/incubator-celeborn/releases/tag/v0.2.0-incubating-rc5 > > The git commit hash: > 368b8d0cb67a71e5af2a29604359e583e38d39cb > > The source and binary artifacts can be found at: > https://dist.apache.org/repos/dist/dev/incubator/celeborn/v0.2.0-incubating-rc5/ > > Fingerprint of the PGP key release artifacts are signed with: > FCF20BB29C7BEFDF58F998F76392F71F37356FA0 > > My public key to verify signatures can be found in: > https://dist.apache.org/repos/dist/release/incubator/celeborn/KEYS > > The vote will be open for at least 72 hours or until necessary > number of votes are reached. > > Please vote accordingly: > > [ ] +1 approve > [ ] +0 no opinion > [ ] -1 disapprove (and the reason) > > Checklist for release: > https://cwiki.apache.org/confluence/display/INCUBATOR/Incubator+Release+Checklist > Steps to validate the release: > https://www.apache.org/info/verification.html > > Starting with my +1 (binding): > > * Download links, checksums and PGP signatures are valid. > * Source code distributions have correct names matching the current release. > * Release files have the word incubating in their name. > * DISCLAIMER, LICENSE and NOTICE files are correct. > * All files have license headers if necessary. > * No unlicensed compiled archives bundled in source archive. > * The source tarball matches the git tag. > * Build from source is successful. > > Thanks, > Ethan Feng
[VOTE] Release Apache Celeborn(Incubating) 0.2.0-incubating-rc5
Hi Celeborn community, This is a call for a vote to release Apache Celeborn (Incubating) 0.2.0-incubating-rc5 The git tag to be voted upon: https://github.com/apache/incubator-celeborn/releases/tag/v0.2.0-incubating-rc5 The git commit hash: 368b8d0cb67a71e5af2a29604359e583e38d39cb The source and binary artifacts can be found at: https://dist.apache.org/repos/dist/dev/incubator/celeborn/v0.2.0-incubating-rc5/ Fingerprint of the PGP key release artifacts are signed with: FCF20BB29C7BEFDF58F998F76392F71F37356FA0 My public key to verify signatures can be found in: https://dist.apache.org/repos/dist/release/incubator/celeborn/KEYS The vote will be open for at least 72 hours or until necessary number of votes are reached. Please vote accordingly: [ ] +1 approve [ ] +0 no opinion [ ] -1 disapprove (and the reason) Checklist for release: https://cwiki.apache.org/confluence/display/INCUBATOR/Incubator+Release+Checklist Steps to validate the release: https://www.apache.org/info/verification.html Starting with my +1 (binding): * Download links, checksums and PGP signatures are valid. * Source code distributions have correct names matching the current release. * Release files have the word incubating in their name. * DISCLAIMER, LICENSE and NOTICE files are correct. * All files have license headers if necessary. * No unlicensed compiled archives bundled in source archive. * The source tarball matches the git tag. * Build from source is successful. Thanks, Ethan Feng
[VOTE] Release Apache Celeborn(Incubating) 0.2.0-incubating-rc4
Hello Incubator Community, This is a call for a vote to release Apache Celeborn(Incubating) version 0.2.0-incubating-rc4 The Apache Celeborn community has voted on and approved a proposal to release Apache Celeborn(Incubating) version 0.2.0-incubating-rc4 We now kindly request the Incubator PMC members review and vote on this incubator release. Celeborn community vote thread: • https://lists.apache.org/thread/3qv3byyy1rqv7l9qsx02gbto1n9ymd1h Vote result thread: • https://lists.apache.org/thread/k4jvqdd0dwk9dc7t6n80tgqv99fmwok7 The release candidate: • https://dist.apache.org/repos/dist/dev/incubator/celeborn/v0.2.0-incubating-rc4 Git tag for the release: • https://github.com/apache/incubator-celeborn/releases/tag/v0.2.0-incubating-rc4 Public keys file: • https://dist.apache.org/repos/dist/release/incubator/celeborn/KEYS The change log is available in: • https://github.com/apache/incubator-celeborn/compare/v0.1.4...v0.2.0-incubating-rc4 The vote will be open for at least 72 hours or until the necessary number of votes are reached. Please vote accordingly: [ ] +1 approve [ ] +0 no opinion [ ] -1 disapprove with the reason More detailed checklist please refer: • https://cwiki.apache.org/confluence/display/INCUBATOR/Incubator+Release+Checklist Steps to validate the release， Please refer to: • https://www.apache.org/info/verification.html Thanks, On behalf of Apache Celeborn(Incubating) community
[RESULT][VOTE] Release Apache Celeborn(Incubating) 0.2.0-incubating-rc4
Hi Celeborn community, The vote closes now as 72hr have passed. The vote PASSES with: (* = binding) +1: Ethan Feng* Keyong Zhou* Cheng Pan* Angers Zhu* Kerwin Zhang* Bingjie Yang There are no 0 or -1 votes. The voting thread: https://lists.apache.org/thread/3qv3byyy1rqv7l9qsx02gbto1n9ymd1h I will now bring the vote to gene...@incubator.apache.org to get approval by the IPMC. If this vote passes, the release is accepted and published. Thanks, Ethan Feng
[VOTE] Release Apache Celeborn(Incubating) 0.2.0-incubating-rc4
Hi Celeborn community, This is a call for a vote to release Apache Celeborn (Incubating) 0.2.0-incubating-rc4 The git tag to be voted upon: https://github.com/apache/incubator-celeborn/releases/tag/v0.2.0-incubating-rc4 The git commit hash: 282e0b0bbc76fa9339e931b0b252b6cbb16dddf5 The source and binary artifacts can be found at: https://dist.apache.org/repos/dist/dev/incubator/celeborn/v0.2.0-incubating-rc4/ Fingerprint of the PGP key release artifacts are signed with: FCF20BB29C7BEFDF58F998F76392F71F37356FA0 My public key to verify signatures can be found in: https://dist.apache.org/repos/dist/release/incubator/celeborn/KEYS The vote will be open for at least 72 hours or until necessary number of votes are reached. Please vote accordingly: [ ] +1 approve [ ] +0 no opinion [ ] -1 disapprove (and the reason) Checklist for release: https://cwiki.apache.org/confluence/display/INCUBATOR/Incubator+Release+Checklist Steps to validate the release: https://www.apache.org/info/verification.html Starting with my +1 (binding): * Download links, checksums and PGP signatures are valid. * Source code distributions have correct names matching the current release. * Release files have the word incubating in their name. * DISCLAIMER, LICENSE and NOTICE files are correct. * All files have license headers if necessary. * No unlicensed compiled archives bundled in source archive. * The source tarball matches the git tag. * Build from source is successful. Thanks, Ethan Feng
Re: Call for UT
Hi Kaijie, I introduced Jacoco and I found that its measurement is not accurate. If you have interest you are welcome to create a pull request and introduce scoverage. Thanks, Ethan Kaijie Chen 于2023年2月2日周四 17:09写道： > > +1, good idea. > > To collect the coverage info, scoverage[1] is preferred over jacoco[2] for > Scala code. > > Kaijie > > [1]: https://github.com/scoverage/scalac-scoverage-plugin > [2]: https://github.com/jacoco/jacoco > > On 2023/01/11 08:10:47 Keyong Zhou wrote: > > Hi community, > > > > Currently the code coverage is quite low, I think it's time to boost the UT > > coverage, any effort will be appreciated, thanks! > > > > Thanks, > > Keyong Zhou > >
[VOTE] Release Apache Celeborn(Incubating) 0.2.0-incubating-rc3
Hello Incubator Community, This is a call for a vote to release Apache Celeborn(Incubating) version 0.2.0-incubating-rc3 The Apache Celeborn community has voted on and approved a proposal to release Apache Celeborn(Incubating) version 0.2.0-incubating-rc3 We now kindly request the Incubator PMC members review and vote on this incubator release. Celeborn community vote thread: • https://lists.apache.org/thread/d964fnnw5wff4zjmrdo2srj6rqgb336g Vote result thread: • https://lists.apache.org/thread/3xc8dz0bp7bcfsk8m6l035mz6cx48yoz The release candidate: • https://dist.apache.org/repos/dist/dev/incubator/celeborn/v0.2.0-incubating-rc3 Git tag for the release: • https://github.com/apache/incubator-celeborn/releases/tag/v0.2.0-incubating-rc3 Public keys file: • https://dist.apache.org/repos/dist/release/incubator/celeborn/KEYS The change log is available in: • https://github.com/apache/incubator-celeborn/compare/v0.1.4...v0.2.0-incubating-rc3 The vote will be open for at least 72 hours or until the necessary number of votes are reached. Please vote accordingly: [ ] +1 approve [ ] +0 no opinion [ ] -1 disapprove with the reason More detailed checklist please refer: • https://cwiki.apache.org/confluence/display/INCUBATOR/Incubator+Release+Checklist Steps to validate the release， Please refer to: • https://www.apache.org/info/verification.html Thanks, On behalf of Apache Celeborn(Incubating) community
[RESULT][VOTE] Release Apache Celeborn(Incubating) 0.2.0-incubating-rc3
Hi Celeborn community, The vote closes now as 72hr have passed. The vote PASSES with: (* = binding) +1: Ethan Feng* Keyong Zhou* Cheng Pan* There are no 0 or -1 votes. The vote thread: https://lists.apache.org/thread/d964fnnw5wff4zjmrdo2srj6rqgb336g I will now bring the vote to gene...@incubator.apache.org to get approval by the IPMC. If this vote passes, the release is accepted and published. Thanks, Ethan Feng
[VOTE] Release Apache Celeborn(Incubating) 0.2.0-incubating-rc3
Hi Celeborn community, This is a call for the vote to release Apache Celeborn (Incubating) 0.2.0-incubating-rc3 The git tag to be voted upon: https://github.com/apache/incubator-celeborn/releases/tag/v0.2.0-incubating-rc3 The git commit hash: 98b356f599b9e8960d755cb6add8f6934345d2a8 The source and binary artifacts can be found at: https://dist.apache.org/repos/dist/dev/incubator/celeborn/v0.2.0-incubating-rc3/ The fingerprint of the PGP key release artifacts is signed with: FCF20BB29C7BEFDF58F998F76392F71F37356FA0 My public key to verify signatures can be found in: https://dist.apache.org/repos/dist/release/incubator/celeborn/KEYS The vote will be open for at least 72 hours or until necessary number of votes are reached. Please vote accordingly: [ ] +1 approve [ ] +0 no opinion [ ] -1 disapprove (and the reason) Checklist for release: https://cwiki.apache.org/confluence/display/INCUBATOR/Incubator+Release+Checklist Steps to validate the release: https://www.apache.org/info/verification.html Starting with my +1 (binding): * Download links, checksums and PGP signatures are valid. * Source code distributions have correct names matching the current release. * Release files have the word incubating in their name. * DISCLAIMER, LICENSE and NOTICE files are correct. * All files have license headers if necessary. * No unlicensed compiled archives bundled in source archive. * The source tarball matches the git tag. * Build from source is successful. Thanks, Ethan Feng
[VOTE] Release Apache Celeborn(Incubating) 0.2.0-incubating-rc2
Hello Incubator Community, This is a call for a vote to release Apache Celeborn(Incubating) version 0.2.0-incubating-rc2 The Apache Celeborn community has voted on and approved a proposal to release Apache Celeborn(Incubating) version 0.2.0-incubating-rc2 We now kindly request the Incubator PMC members review and vote on this incubator release. Celeborn community vote thread: • https://lists.apache.org/thread/v4t7r6h8043s0hvhhvlzhb35nr9gvshr Vote result thread: • https://lists.apache.org/thread/7n0gw4dz8d5p852sfok9f61cqods6h76 The release candidate: • https://dist.apache.org/repos/dist/dev/incubator/celeborn/v0.2.0-incubating-rc2 Git tag for the release: • https://github.com/apache/incubator-celeborn/releases/tag/v0.2.0-incubating-rc2 Public keys file: • https://dist.apache.org/repos/dist/release/incubator/celeborn/KEYS The change log is available in: • https://github.com/apache/incubator-celeborn/compare/v0.1.4...v0.2.0-incubating-rc2 The vote will be open for at least 72 hours or until the necessary number of votes are reached. Please vote accordingly: [ ] +1 approve [ ] +0 no opinion [ ] -1 disapprove with the reason More detailed checklist please refer: • https://cwiki.apache.org/confluence/display/INCUBATOR/Incubator+Release+Checklist Steps to validate the release， Please refer to: • https://www.apache.org/info/verification.html Thanks, On behalf of Apache Celeborn(Incubating) community
[RESULT][VOTE] Release Apache Celeborn(Incubating) 0.2.0-incubating-rc2
Hi Celeborn community, The vote closes now as 72hr have passed. The vote PASSES with: (* = binding) +1: - Ethan Feng* - Cheng Pan* - Keyong Zhou* - Angerszh* - zhqiangchen There are no 0 or -1 votes. The vote thread: https://lists.apache.org/thread/v4t7r6h8043s0hvhhvlzhb35nr9gvshr I will now bring the vote to gene...@incubator.apache.org to get approval by the IPMC. If this vote passes, the release is accepted and published. Thanks, Ethan Feng
[VOTE] Release Apache Celeborn(Incubating) 0.2.0-incubating-rc2
Hi Celeborn community, This is a call for vote to release Apache Celeborn (Incubating) 0.2.0-incubating-rc2 The git tag to be voted upon: https://github.com/apache/incubator-celeborn/releases/tag/v0.2.0-incubating-rc2 The git commit hash: fcd1e8ac8723859648e4597092328fa1ed3b96c2 The source and binary artifacts can be found at: https://dist.apache.org/repos/dist/dev/incubator/celeborn/v0.2.0-incubating-rc2/ Fingerprint of the PGP key release artifacts are signed with: FCF20BB29C7BEFDF58F998F76392F71F37356FA0 My public key to verify signatures can be found in: https://dist.apache.org/repos/dist/release/incubator/celeborn/KEYS The vote will be open for at least 72 hours or until necessary number of votes are reached. Please vote accordingly: [ ] +1 approve [ ] +0 no opinion [ ] -1 disapprove (and the reason) Checklist for release: https://cwiki.apache.org/confluence/display/INCUBATOR/Incubator+Release+Checklist Steps to validate the release: https://www.apache.org/info/verification.html Starting with my +1 (binding): * Download links, checksums and PGP signatures are valid. * Source code distributions have correct names matching the current release. * Release files have the word incubating in their name. * DISCLAIMER, LICENSE and NOTICE files are correct. * All files have license headers if necessary. * No unlicensed compiled archives bundled in source archive. * The source tarball matches the git tag. * Build from source is successful. Thanks, Ethan Feng
Re: [VOTE] Release Apache Celeborn(Incubating) 0.2.0-incubating-rc1
This vote is canceled due to the binary package having some error. 在 2023年1月4日 12:27:06 上，Keyong Zhou 写道： > Also, I think we should not include images in source tarball. > > Thanks, > Keyong Zhou > > Keyong Zhou 于2023年1月4日周三 11:30写道： > > Hi Feng, > > > When I tried to decompress the apache-celeborn-0.2.0-incubating-bin.tgz on > > CentOS I got the following error: > > > ._apache-celeborn-0.2.0-incubating-bin > > tar: 忽略未知的扩展头关键字‘LIBARCHIVE.xattr.com.apple.provenance’ > > apache-celeborn-0.2.0-incubating-bin/ > > apache-celeborn-0.2.0-incubating-bin/._jars > > tar: 忽略未知的扩展头关键字‘LIBARCHIVE.xattr.com.apple.provenance’ > > apache-celeborn-0.2.0-incubating-bin/jars/ > > apache-celeborn-0.2.0-incubating-bin/._docker > > tar: 忽略未知的扩展头关键字‘LIBARCHIVE.xattr.com.apple.provenance’ > > > > I think we should re-build the tarball and restart the vote. > > > Thanks, > > Keyong Zhou > > > Ethan Feng 于2023年1月3日周二 21:00写道： > > > > Hi Celeborn community, > > > > > > This is a call for vote to release Apache Celeborn (Incubating) > > > 0.2.0-incubating-rc1 > > > > > > The git tag to be voted upon: > > > > > > > https://github.com/apache/incubator-celeborn/releases/tag/v0.2.0-incubating-rc1 > > > > > > The git commit hash: > > > 7c0664ccd0c296cb14b63d2f0c9733c0deb538b9 > > > > > > The source and binary artifacts can be found at: > > > > > > > https://dist.apache.org/repos/dist/dev/incubator/celeborn/v0.2.0-incubating-rc1/ > > > > > > Fingerprint of the PGP key release artifacts are signed with: > > > FCF20BB29C7BEFDF58F998F76392F71F37356FA0 > > > > > > My public key to verify signatures can be found in: > > > https://dist.apache.org/repos/dist/release/incubator/celeborn/KEYS > > > > > > The vote will be open for at least 72 hours or until necessary > > > number of votes are reached. > > > > > > Please vote accordingly: > > > > > > [ ] +1 approve > > > [ ] +0 no opinion > > > [ ] -1 disapprove (and the reason) > > > > > > Checklist for release: > > > > > > > https://cwiki.apache.org/confluence/display/INCUBATOR/Incubator+Release+Checklist > > > Steps to validate the release: > > > https://www.apache.org/info/verification.html > > > > > > Starting with my +1 (binding): > > > > > > * Download links, checksums and PGP signatures are valid. > > > * Source code distributions have correct names matching the current > > > release. > > > * Release files have the word incubating in their name. > > > * DISCLAIMER, LICENSE and NOTICE files are correct. > > > * All files have license headers if necessary. > > > * No unlicensed compiled archives bundled in source archive. > > > * The source tarball matches the git tag. > > > * Build from source is successful. > > > > > > Thanks, > > > Ethan Feng > > > > > >
[VOTE] Release Apache Celeborn(Incubating) 0.2.0-incubating-rc1
Hi Celeborn community, This is a call for vote to release Apache Celeborn (Incubating) 0.2.0-incubating-rc1 The git tag to be voted upon: https://github.com/apache/incubator-celeborn/releases/tag/v0.2.0-incubating-rc1 The git commit hash: 7c0664ccd0c296cb14b63d2f0c9733c0deb538b9 The source and binary artifacts can be found at: https://dist.apache.org/repos/dist/dev/incubator/celeborn/v0.2.0-incubating-rc1/ Fingerprint of the PGP key release artifacts are signed with: FCF20BB29C7BEFDF58F998F76392F71F37356FA0 My public key to verify signatures can be found in: https://dist.apache.org/repos/dist/release/incubator/celeborn/KEYS The vote will be open for at least 72 hours or until necessary number of votes are reached. Please vote accordingly: [ ] +1 approve [ ] +0 no opinion [ ] -1 disapprove (and the reason) Checklist for release: https://cwiki.apache.org/confluence/display/INCUBATOR/Incubator+Release+Checklist Steps to validate the release: https://www.apache.org/info/verification.html Starting with my +1 (binding): * Download links, checksums and PGP signatures are valid. * Source code distributions have correct names matching the current release. * Release files have the word incubating in their name. * DISCLAIMER, LICENSE and NOTICE files are correct. * All files have license headers if necessary. * No unlicensed compiled archives bundled in source archive. * The source tarball matches the git tag. * Build from source is successful. Thanks, Ethan Feng
Re: [VOTE] Release Apache Celeborn(Incubating) 0.2.0-rc0
This mail should be ignored. 在 2023年1月3日 17:04:39 上，Ethan Feng 写道： > Hi Celeborn community, > > This is a call for vote to release Apache Uniffle (Incubating) 0.2.0-rc0 > > The git tag to be voted upon: > > https://github.com/apache/incubator-celeborn/releases/tag/v0.2.0-incubating-rc0 > > The git commit hash: > 6c3709ce13aa5172fe1c026e14728663ddab7ef3 > > The source tarballs can be found at: > > https://dist.apache.org/repos/dist/dev/incubator/celeborn/v0.2.0-incubating-rc0/ > > Fingerprint of the PGP key release artifacts are signed with: > FCF20BB29C7BEFDF58F998F76392F71F37356FA0 > > My public key to verify signatures can be found in: > https://dist.apache.org/repos/dist/dev/incubator/celeborn/KEYS > > The vote will be open for at least 72 hours or until necessary > number of votes are reached. > > Please vote accordingly: > > [ ] +1 approve > [ ] +0 no opinion > [ ] -1 disapprove (and the reason) > > Checklist for release: > > https://cwiki.apache.org/confluence/display/INCUBATOR/Incubator+Release+Checklist > Steps to validate the release: > https://linkis.apache.org/community/how-to-verify > > Starting with my +1 (non-binding): > > * Download links, checksums and PGP signatures are valid. > * Source code distributions have correct names matching the current > release. > * Release files have the word incubating in their name. > * DISCLAIMER, LICENSE and NOTICE files are correct. > * All files have license headers if necessary. > * No unlicensed compiled archives bundled in source archive. > * The source tarball matches the git tag. > * Build from source is successful, Uniffle CI passed. [1] > > Thanks, > Ethan Feng >
[VOTE] Release Apache Celeborn(Incubating) 0.2.0-rc0
Hi Celeborn community, This is a call for vote to release Apache Uniffle (Incubating) 0.2.0-rc0 The git tag to be voted upon: https://github.com/apache/incubator-celeborn/releases/tag/v0.2.0-incubating-rc0 The git commit hash: 6c3709ce13aa5172fe1c026e14728663ddab7ef3 The source tarballs can be found at: https://dist.apache.org/repos/dist/dev/incubator/celeborn/v0.2.0-incubating-rc0/ Fingerprint of the PGP key release artifacts are signed with: FCF20BB29C7BEFDF58F998F76392F71F37356FA0 My public key to verify signatures can be found in: https://dist.apache.org/repos/dist/dev/incubator/celeborn/KEYS The vote will be open for at least 72 hours or until necessary number of votes are reached. Please vote accordingly: [ ] +1 approve [ ] +0 no opinion [ ] -1 disapprove (and the reason) Checklist for release: https://cwiki.apache.org/confluence/display/INCUBATOR/Incubator+Release+Checklist Steps to validate the release: https://linkis.apache.org/community/how-to-verify Starting with my +1 (non-binding): * Download links, checksums and PGP signatures are valid. * Source code distributions have correct names matching the current release. * Release files have the word incubating in their name. * DISCLAIMER, LICENSE and NOTICE files are correct. * All files have license headers if necessary. * No unlicensed compiled archives bundled in source archive. * The source tarball matches the git tag. * Build from source is successful, Uniffle CI passed. [1] Thanks, Ethan Feng
Re: [DISCUSS] Prepare first release
Roger that. The check will be done. Ethan. 在 2022年12月14日 +0800 19:41，Keyong Zhou ，写道： > Hi celeborn (-incubating) community: > > Currently we are preparing for the first release (branch-0.2). To ensure > code quality, I would like to test for core-path correctness and stability, > could Angerszh and nafiyaix > help test graceful shutdown and rolling upgrade? And > could Ethan Feng help test load-aware slots > allocation? > > And we would be rather happy if anyone can help test for other modules > (k8s, HA, etc.). > > Thanks, > Keyong
