Hi Mridul,
The Celeborn dashboard will not be included in this version
release. Therefore, it should not pose any issues for us at this time.
After some investigation, I discovered that our CI tests are based
on a fixed list of projects, which is why this problem was not
detected earlier.
Thank you for bringing this to our attention. To bypass the lint
exceptions for unit tests, please use the following command:
```
SPARK_BINARY_VERSION=3.1
SPARK_MAJOR_VERSION=${SPARK_BINARY_VERSION%%.*}
PROFILES="-Pgoogle-mirror,spark-3.1"
TEST_MODULES="client-spark/common,client-spark/spark-${SPARK_MAJOR_VERSION},client-spark/spark-${SPARK_MAJOR_VERSION}-shaded,tests/spark-it"
build/mvn $PROFILES -pl $TEST_MODULES -am clean install -DskipTests
build/mvn $PROFILES -pl $TEST_MODULES
-Dspark.shuffle.sort.io.plugin.class=org.apache.spark.shuffle.sort.io.LocalDiskShuffleDataIO
test
```
Additionally, I have created a Jira ticket[0] to track this lint exception.
Thanks,
Ethan Feng
[0] https://issues.apache.org/jira/browse/CELEBORN-1480
Mridul Muralidharan <mri...@gmail.com> 于2024年6月24日周一 15:35写道：
>
> Forgot to update here.
>
> Signatures, digests, etc check out fine.
> Checked out tag and build/tested with "-Pspark3.1"
>
> I keep getting the following error:
>
> - metrics/prometheus *** FAILED ***
> 200 did not equal 404 (ApiBaseResourceSuite.scala:90)
> - metrics/json *** FAILED ***
> 200 did not equal 404 (ApiBaseResourceSuite.scala:96)
>
> In unit-tests.log, I have this [1]
>
> After explicitly ignoring these two tests, I then ran into lint issues with
> "web" submodule - and could not get around to debugging the issue (I
> installed pnpm - which does not seem to be called out in readme).
>
> Thanks,
> Mridul
>
> [1]
> 24/06/24 01:57:33,545 ERROR [ScalaTest-main-running-DiscoverySuite]
> MetricsConfig: Error loading configuration file
> file:/home/mridul/work/apache/vote/celeborn/service/target/celeborn-service_2.12-0.5.0-tests.jar!/metrics-api.properties
> java.io.FileNotFoundException:
> file:/home/mridul/work/apache/vote/celeborn/service/target/celeborn-service_2.12-0.5.0-tests.jar!/metrics-api.properties
> (No such file or directory)
> at java.io.FileInputStream.open0(Native Method)
> at java.io.FileInputStream.open(FileInputStream.java:195)
> at java.io.FileInputStream.<init>(FileInputStream.java:138)
> at java.io.FileInputStream.<init>(FileInputStream.java:93)
> at
> org.apache.celeborn.common.metrics.MetricsConfig.loadPropertiesFromFile(MetricsConfig.scala:95)
> at
> org.apache.celeborn.common.metrics.MetricsConfig.initialize(MetricsConfig.scala:50)
> at
> org.apache.celeborn.common.metrics.MetricsSystem.<init>(MetricsSystem.scala:53)
> at
> org.apache.celeborn.common.metrics.MetricsSystem$.createMetricsSystem(MetricsSystem.scala:197)
> at
> org.apache.celeborn.service.deploy.master.Master.<init>(Master.scala:66)
> at
> org.apache.celeborn.service.deploy.master.http.api.ApiMasterResourceSuite.beforeAll(ApiMasterResourceSuite.scala:54)
> at
> org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
> at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
> at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
> at
> org.apache.celeborn.server.common.http.ApiBaseResourceSuite.run(ApiBaseResourceSuite.scala:23)
> at org.scalatest.Suite.callExecuteOnSuite$1(Suite.scala:1178)
> at org.scalatest.Suite.$anonfun$runNestedSuites$1(Suite.scala:1225)
> at
> scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
> at
> scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
> at
> scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
> at org.scalatest.Suite.runNestedSuites(Suite.scala:1223)
> at org.scalatest.Suite.runNestedSuites$(Suite.scala:1156)
> at
> org.scalatest.tools.DiscoverySuite.runNestedSuites(DiscoverySuite.scala:30)
> at org.scalatest.Suite.run(Suite.scala:1111)
> at org.scalatest.Suite.run$(Suite.scala:1096)
> at org.scalatest.tools.DiscoverySuite.run(DiscoverySuite.scala:30)
> at org.scalatest.tools.SuiteRunner.run(SuiteRunner.scala:47)
> at
> org.scalatest.tools.Runner$.$anonfun$doRunRunRunDaDoRunRun$13(Runner.scala:1321)
> at
> org.scalatest.tools.Runner$.$anonfun$doRunRunRunDaDoRunRun$13$adapted(Runner.scala:1315)
> at scala.collection.immutable.List.foreach(List.scala:392)
> at
> org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:1315)
> at
> org.scalatest.tools.Runner$.$anonfun$runOptionallyWithPassFailReporter$24(Runner.scala:992)
> at
> org.scalatest.tools.Runner$.$anonfun$runOptionallyWithPassFailReporter$24$adapted(Runner.scala:970)
> at
> org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:1481)
> at
> org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:970)
> at org.scalatest.tools.Runner$.main(Runner.scala:775)
> at org.scalatest.tools.Runner.main(Runner.scala)
>
>
> On Mon, Jun 24, 2024 at 2:22 AM Ethan Feng <ethanf...@apache.org> wrote:
>
> > Thanks for your feedback, I will close this vote
> > thread and announce the results soon since 72 hours have passed.
> >
> >
> > Ethan Feng.
> >
> > kerwin zhang <kerwin.libra...@gmail.com> 于2024年6月24日周一 15:18写道：
> > >
> > > +1 (binding)
> > >
> > > I checked
> > > - git commit hash is correct.
> > > - links are valid.
> > > - signatures are good.
> > > ```
> > > gpg --verify apache-celeborn-0.5.0-bin.tgz.asc
> > apache-celeborn-0.5.0-bin.tgz
> > > gpg --verify apache-celeborn-0.5.0-source.tgz.asc
> > > apache-celeborn-0.5.0-source.tgz
> > > ```
> > > - checksums are good.
> > > ```
> > > shasum -a 512 apache-celeborn-0.5.0-bin.tgz
> > > shasum -a 512 apache-celeborn-0.5.0-source.tgz
> > > ```
> > >
> > > - build success from source code (macOS).
> > >
> > > ```
> > > ./build/make-distribution.sh -Pspark-3.4
> > > ```
> > >
> > > Thanks,
> > > Kerwin Zhang
> > >
> > > Fu Chen <cfmcgr...@gmail.com> 于2024年6月24日周一 14:46写道：
> > > >
> > > > +1
> > > >
> > > > I checked
> > > > - download links are valid.
> > > > - git commit hash is correct.
> > > > - no binary files in the source release.
> > > > - build success from source code (JDK 17).
> > > > ```
> > > > ./build/make-distribution.sh --sbt-enabled -Pspark-3.5
> > > > ```
> > > >
> > > > Cheng Pan <pan3...@gmail.com> 于2024年6月24日周一 14:09写道：
> > > > >
> > > > > +1 (binding)
> > > > >
> > > > > - All links are valid.
> > > > > - LICENSE/NOTICE looks fine.
> > > > > - CI result looks great.
> > > > > - No known blockers.
> > > > >
> > > > > Thanks,
> > > > > Cheng Pan
> > > > >
> > > > >
> > > > > > On Jun 19, 2024, at 12:45, Ethan Feng <ethanf...@apache.org>
> > wrote:
> > > > > >
> > > > > > Hello, Celeborn community,
> > > > > >
> > > > > > This is a call for a vote to release Apache Celeborn
> > > > > > 0.5.0-rc3
> > > > > >
> > > > > > The git tag to be voted upon:
> > > > > > https://github.com/apache/celeborn/releases/tag/v0.5.0-rc3
> > > > > >
> > > > > > Source and binary artifacts can be found at:
> > > > > > https://dist.apache.org/repos/dist/dev/celeborn/v0.5.0-rc3
> > > > > >
> > > > > > The git commit hash:
> > > > > > 048ef207359113247bff05dcc203c70021ccfa10
> > > > > >
> > > > > > The staging repo:
> > > > > >
> > https://repository.apache.org/content/repositories/orgapacheceleborn-1076/
> > > > > >
> > > > > > The fingerprint of the PGP key release artifacts is signed with:
> > > > > > FCF20BB29C7BEFDF58F998F76392F71F37356FA0
> > > > > >
> > > > > > My public key to verify signatures can be found in:
> > > > > > https://dist.apache.org/repos/dist/release/celeborn/KEYS
> > > > > >
> > > > > > The vote will be open for at least 72 hours or until the necessary
> > > > > > number of votes are reached.
> > > > > >
> > > > > > Please vote accordingly:
> > > > > >
> > > > > > [ ] +1 approve
> > > > > > [ ] +0 no opinion
> > > > > > [ ] -1 disapprove (and the reason)
> > > > > >
> > > > > > Steps to validate the release:
> > > > > > https://www.apache.org/info/verification.html
> > > > > >
> > > > > > * Download links, checksums, and PGP signatures are valid.
> > > > > > * Source code distributions have correct names matching the
> > current release.
> > > > > > * LICENSE and NOTICE files are correct.
> > > > > > * All files have license headers if necessary.
> > > > > > * No unlicensed compiled archives bundled in the source archive.
> > > > > > * The source tarball matches the git tag.
> > > > > > * Build from source is successful.
> > > > > >
> > > > > > There are additional tests:
> > > > > > * Performance test no regression
> > > > > > 1 TB TPC-DS, 0.5.0 VS 0.4.1 : 2042(s) VS 2050(s)
> > > > > > 1.1 TB pure shuffle, 0.5.0 VS 0.4.1 : 11.8min vs 11.8min
> > > > > >
> > > > > > * Result correctness test passed
> > > > > > 1TB TPC-DS runs concurrently, the results are identical.
> > > > > >
> > > > > > * Usability test passed
> > > > > > Rolling upgrade from version 0.4.1 to 0.5.0 succeed.
> > > > > > The metrics system works as expected.
> > > > > >
> > > > > > * Stability test passed
> > > > > > Random worker failures, Celeborn works as expected.
> > > > > > Random master failures, Celeborn works as expected.
> > > > > > Master meta corrupted, Celeborn works as expected.
> > > > > >
> > > > > > * Compatibility test passed
> > > > > > The Celeborn server version of 0.5.0 works fine with the Celeborn
> > client 0.4.1.
> > > > > >
> > > > > > * Grafana dashboard layout checked
> > > > > >
> > > > > >
> > > > > > Regards,
> > > > > > Ethan Feng
> > > > >
> >
