./build/mvn clean package I'm using ubuntu rolling, python 3.11 openjdk 17
CompatibilitySuite: - compatibility MiMa tests *** FAILED *** java.lang.AssertionError: assertion failed: Failed to find the jar inside folder: /home/bjorn/spark-3.4.0/connector/connect/client/jvm/target at scala.Predef$.assert(Predef.scala:223) at org.apache.spark.sql.connect.client.util.IntegrationTestUtils$.findJar(IntegrationTestUtils.scala:67) at org.apache.spark.sql.connect.client.CompatibilitySuite.clientJar$lzycompute(CompatibilitySuite.scala:57) at org.apache.spark.sql.connect.client.CompatibilitySuite.clientJar(CompatibilitySuite.scala:53) at org.apache.spark.sql.connect.client.CompatibilitySuite.$anonfun$new$1(CompatibilitySuite.scala:69) at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85) at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83) at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104) at org.scalatest.Transformer.apply(Transformer.scala:22) at org.scalatest.Transformer.apply(Transformer.scala:20) ... - compatibility API tests: Dataset *** FAILED *** java.lang.AssertionError: assertion failed: Failed to find the jar inside folder: /home/bjorn/spark-3.4.0/connector/connect/client/jvm/target at scala.Predef$.assert(Predef.scala:223) at org.apache.spark.sql.connect.client.util.IntegrationTestUtils$.findJar(IntegrationTestUtils.scala:67) at org.apache.spark.sql.connect.client.CompatibilitySuite.clientJar$lzycompute(CompatibilitySuite.scala:57) at org.apache.spark.sql.connect.client.CompatibilitySuite.clientJar(CompatibilitySuite.scala:53) at org.apache.spark.sql.connect.client.CompatibilitySuite.$anonfun$new$7(CompatibilitySuite.scala:110) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85) at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83) at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104) at org.scalatest.Transformer.apply(Transformer.scala:22) ... SparkConnectClientSuite: - Placeholder test: Create SparkConnectClient - Test connection - Test connection string - Check URI: sc://host, isCorrect: true - Check URI: sc://localhost/, isCorrect: true - Check URI: sc://localhost:1234/, isCorrect: true - Check URI: sc://localhost/;, isCorrect: true - Check URI: sc://host:123, isCorrect: true - Check URI: sc://host:123/;user_id=a94, isCorrect: true - Check URI: scc://host:12, isCorrect: false - Check URI: http://host, isCorrect: false - Check URI: sc:/host:1234/path, isCorrect: false - Check URI: sc://host/path, isCorrect: false - Check URI: sc://host/;parm1;param2, isCorrect: false - Check URI: sc://host:123;user_id=a94, isCorrect: false - Check URI: sc:///user_id=123, isCorrect: false - Check URI: sc://host:-4, isCorrect: false - Check URI: sc://:123/, isCorrect: false - Non user-id parameters throw unsupported errors DatasetSuite: - limit - select - filter - write UserDefinedFunctionSuite: - udf and encoder serialization Run completed in 21 seconds, 944 milliseconds. Total number of tests run: 389 Suites: completed 10, aborted 0 Tests: succeeded 386, failed 3, canceled 0, ignored 0, pending 0 *** 3 TESTS FAILED *** [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary for Spark Project Parent POM 3.4.0: [INFO] [INFO] Spark Project Parent POM ........................... SUCCESS [ 47.096 s] [INFO] Spark Project Tags ................................. SUCCESS [ 14.759 s] [INFO] Spark Project Sketch ............................... SUCCESS [ 21.628 s] [INFO] Spark Project Local DB ............................. SUCCESS [ 20.311 s] [INFO] Spark Project Networking ........................... SUCCESS [01:07 min] [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 15.921 s] [INFO] Spark Project Unsafe ............................... SUCCESS [ 16.020 s] [INFO] Spark Project Launcher ............................. SUCCESS [ 10.873 s] [INFO] Spark Project Core ................................. SUCCESS [37:10 min] [INFO] Spark Project ML Local Library ..................... SUCCESS [ 40.841 s] [INFO] Spark Project GraphX ............................... SUCCESS [02:39 min] [INFO] Spark Project Streaming ............................ SUCCESS [05:53 min] [INFO] Spark Project Catalyst ............................. SUCCESS [11:22 min] [INFO] Spark Project SQL .................................. SUCCESS [ 02:27 h] [INFO] Spark Project ML Library ........................... SUCCESS [22:45 min] [INFO] Spark Project Tools ................................ SUCCESS [ 7.263 s] [INFO] Spark Project Hive ................................. SUCCESS [ 01:21 h] [INFO] Spark Project REPL ................................. SUCCESS [02:07 min] [INFO] Spark Project Assembly ............................. SUCCESS [ 11.704 s] [INFO] Kafka 0.10+ Token Provider for Streaming ........... SUCCESS [ 26.748 s] [INFO] Spark Integration for Kafka 0.10 ................... SUCCESS [01:44 min] [INFO] Kafka 0.10+ Source for Structured Streaming ........ SUCCESS [33:27 min] [INFO] Spark Project Examples ............................. SUCCESS [01:17 min] [INFO] Spark Integration for Kafka 0.10 Assembly .......... SUCCESS [ 27.292 s] [INFO] Spark Avro ......................................... SUCCESS [02:18 min] [INFO] Spark Project Connect Common ....................... SUCCESS [ 43.728 s] [INFO] Spark Project Connect Server ....................... SUCCESS [01:19 min] [INFO] Spark Project Connect Client ....................... FAILURE [ 53.524 s] [INFO] Spark Protobuf ..................................... SKIPPED [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 05:58 h [INFO] Finished at: 2023-02-22T22:28:38+01:00 [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.scalatest:scalatest-maven-plugin:2.2.0:test (test) on project spark-connect-client-jvm_2.12: There are test failures -> [Help 1] ons. 22. feb. 2023 kl. 21:41 skrev Mridul Muralidharan <mri...@gmail.com>: > > Signatures, digests, etc check out fine - thanks for updating them ! > Checked out tag and build/tested with -Phive -Pyarn -Pmesos -Pkubernetes > > > The test ClientE2ETestSuite.simple udf failed [1] in "Connect Client " > module ... yet to test "Spark Protobuf" module due to the failure. > > > Regards, > Mridul > > [1] > > - simple udf *** FAILED *** > > io.grpc.StatusRuntimeException: INTERNAL: > org.apache.spark.sql.ClientE2ETestSuite > > at io.grpc.Status.asRuntimeException(Status.java:535) > > at > io.grpc.stub.ClientCalls$BlockingResponseStream.hasNext(ClientCalls.java:660) > > at org.apache.spark.sql.connect.client.SparkResult.org > $apache$spark$sql$connect$client$SparkResult$$processResponses(SparkResult.scala:50) > > at > org.apache.spark.sql.connect.client.SparkResult.length(SparkResult.scala:95) > > at > org.apache.spark.sql.connect.client.SparkResult.toArray(SparkResult.scala:112) > > at org.apache.spark.sql.Dataset.$anonfun$collect$1(Dataset.scala:2037) > > at org.apache.spark.sql.Dataset.withResult(Dataset.scala:2267) > > at org.apache.spark.sql.Dataset.collect(Dataset.scala:2036) > > at > org.apache.spark.sql.ClientE2ETestSuite.$anonfun$new$5(ClientE2ETestSuite.scala:65) > > at > scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) > > ... > > > > > > On Wed, Feb 22, 2023 at 2:07 AM Mridul Muralidharan <mri...@gmail.com> > wrote: > >> >> Thanks Xinrong ! >> The signature verifications are fine now ... will continue with testing >> the release. >> >> >> Regards, >> Mridul >> >> >> On Wed, Feb 22, 2023 at 1:27 AM Xinrong Meng <xinrong.apa...@gmail.com> >> wrote: >> >>> Hi Mridul, >>> >>> Would you please try that again? It should work now. >>> >>> On Wed, Feb 22, 2023 at 2:04 PM Mridul Muralidharan <mri...@gmail.com> >>> wrote: >>> >>>> >>>> Hi Xinrong, >>>> >>>> Was it signed with the same key as present in KEYS [1] ? >>>> I am seeing errors with gpg when validating. For example: >>>> >>>> >>>> $ gpg --verify pyspark-3.4.0.tar.gz.asc >>>> >>>> gpg: assuming signed data in 'pyspark-3.4.0.tar.gz' >>>> >>>> gpg: Signature made Tue 21 Feb 2023 05:56:05 AM CST >>>> >>>> gpg: using RSA key >>>> CC68B3D16FE33A766705160BA7E57908C7A4E1B1 >>>> >>>> gpg: issuer "xinr...@apache.org" >>>> >>>> gpg: Can't check signature: No public key >>>> >>>> >>>> >>>> Regards, >>>> Mridul >>>> >>>> [1] https://dist.apache.org/repos/dist/dev/spark/KEYS >>>> >>>> >>>> On Tue, Feb 21, 2023 at 10:36 PM Xinrong Meng <xinrong.apa...@gmail.com> >>>> wrote: >>>> >>>>> Please vote on releasing the following candidate as Apache Spark >>>>> version 3.4.0. >>>>> >>>>> The vote is open until 11:59pm Pacific time *February 27th* and >>>>> passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes. >>>>> >>>>> [ ] +1 Release this package as Apache Spark 3.4.0 >>>>> [ ] -1 Do not release this package because ... >>>>> >>>>> To learn more about Apache Spark, please see http://spark.apache.org/ >>>>> >>>>> The tag to be voted on is *v3.4.0-rc1* (commit >>>>> e2484f626bb338274665a49078b528365ea18c3b): >>>>> https://github.com/apache/spark/tree/v3.4.0-rc1 >>>>> >>>>> The release files, including signatures, digests, etc. can be found at: >>>>> https://dist.apache.org/repos/dist/dev/spark/v3.4.0-rc1-bin/ >>>>> >>>>> Signatures used for Spark RCs can be found in this file: >>>>> https://dist.apache.org/repos/dist/dev/spark/KEYS >>>>> >>>>> The staging repository for this release can be found at: >>>>> https://repository.apache.org/content/repositories/orgapachespark-1435 >>>>> >>>>> The documentation corresponding to this release can be found at: >>>>> https://dist.apache.org/repos/dist/dev/spark/v3.4.0-rc1-docs/ >>>>> >>>>> The list of bug fixes going into 3.4.0 can be found at the following >>>>> URL: >>>>> https://issues.apache.org/jira/projects/SPARK/versions/12351465 >>>>> >>>>> This release is using the release script of the tag v3.4.0-rc1. >>>>> >>>>> >>>>> FAQ >>>>> >>>>> ========================= >>>>> How can I help test this release? >>>>> ========================= >>>>> If you are a Spark user, you can help us test this release by taking >>>>> an existing Spark workload and running on this release candidate, then >>>>> reporting any regressions. >>>>> >>>>> If you're working in PySpark you can set up a virtual env and install >>>>> the current RC and see if anything important breaks, in the Java/Scala >>>>> you can add the staging repository to your projects resolvers and test >>>>> with the RC (make sure to clean up the artifact cache before/after so >>>>> you don't end up building with a out of date RC going forward). >>>>> >>>>> =========================================== >>>>> What should happen to JIRA tickets still targeting 3.4.0? >>>>> =========================================== >>>>> The current list of open tickets targeted at 3.4.0 can be found at: >>>>> https://issues.apache.org/jira/projects/SPARK and search for "Target >>>>> Version/s" = 3.4.0 >>>>> >>>>> Committers should look at those and triage. Extremely important bug >>>>> fixes, documentation, and API tweaks that impact compatibility should >>>>> be worked on immediately. Everything else please retarget to an >>>>> appropriate release. >>>>> >>>>> ================== >>>>> But my bug isn't fixed? >>>>> ================== >>>>> In order to make timely releases, we will typically not hold the >>>>> release unless the bug in question is a regression from the previous >>>>> release. That being said, if there is something which is a regression >>>>> that has not been correctly targeted please ping me or a committer to >>>>> help target the issue. >>>>> >>>>> Thanks, >>>>> Xinrong Meng >>>>> >>>> -- Bjørn Jørgensen Vestre Aspehaug 4, 6010 Ålesund Norge +47 480 94 297
