Thanks! I was wondering about that ClientE2ETestSuite failure today, so I'm glad to know that it's also being experienced by others.
On a similar note, I am experiencing the following error when running the Python tests with Python 3.7: + ./python/run-tests --python-executables=python3 Running PySpark tests. Output is in /home/ec2-user/spark/python/unit-tests.log Will test against the following Python executables: ['python3'] Will test the following Python modules: ['pyspark-connect', 'pyspark-core', 'pyspark-errors', 'pyspark-ml', 'pyspark-mllib', 'pyspark-pandas', 'pyspark-pandas-slow', 'pyspark-resource', 'pyspark-sql', 'pyspark-streaming'] python3 python_implementation is CPython python3 version is: Python 3.7.16 Starting test(python3): pyspark.ml.tests.test_feature (temp output: /home/ec2-user/spark/python/target/8ca9ab1a-05cc-4845-bf89-30d9001510bc/python3__pyspark.ml.tests.test_feature__kg6sseie.log) Starting test(python3): pyspark.ml.tests.test_base (temp output: /home/ec2-user/spark/python/target/f2264f3b-6b26-4e61-9452-8d6ddd7eb002/python3__pyspark.ml.tests.test_base__0902zf9_.log) Starting test(python3): pyspark.ml.tests.test_algorithms (temp output: /home/ec2-user/spark/python/target/d1dc4e07-e58c-4c03-abe5-09d8fab22e6a/python3__pyspark.ml.tests.test_algorithms__lh3wb2u8.log) Starting test(python3): pyspark.ml.tests.test_evaluation (temp output: /home/ec2-user/spark/python/target/3f42dc79-c945-4cf2-a1eb-83e72b40a9ee/python3__pyspark.ml.tests.test_evaluation__89idc7fa.log) Finished test(python3): pyspark.ml.tests.test_base (16s) Starting test(python3): pyspark.ml.tests.test_functions (temp output: /home/ec2-user/spark/python/target/5a3b90f0-216b-4edd-9d15-6619d3e03300/python3__pyspark.ml.tests.test_functions__g5u1290s.log) Traceback (most recent call last): File "/usr/lib64/python3.7/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/usr/lib64/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/ec2-user/spark/python/pyspark/ml/tests/test_functions.py", line 21, in <module> from pyspark.ml.functions import predict_batch_udf File "/home/ec2-user/spark/python/pyspark/ml/functions.py", line 38, in <module> from typing import Any, Callable, Iterator, List, Mapping, Protocol, TYPE_CHECKING, Tuple, Union ImportError: cannot import name 'Protocol' from 'typing' (/usr/lib64/python3.7/typing.py) Had test failures in pyspark.ml.tests.test_functions with python3; see logs. I know we should move on to a newer version of Python, but isn't Python 3.7 still officially supported? Thank you, Jonathan Kelly On Wed, Feb 22, 2023 at 1:47 PM Herman van Hovell <her...@databricks.com.invalid> wrote: > Hi All, > > Thanks for testing the 3.4.0 RC! I apologize for the maven testing > failures for the Spark Connect Scala Client. We will try to get those > sorted as soon as possible. > > This is an artifact of having multiple build systems, and only running CI > for one (SBT). That, however, is a debate for another day :)... > > Cheers, > Herman > > On Wed, Feb 22, 2023 at 5:32 PM Bjørn Jørgensen <bjornjorgen...@gmail.com> > wrote: > >> ./build/mvn clean package >> >> I'm using ubuntu rolling, python 3.11 openjdk 17 >> >> CompatibilitySuite: >> - compatibility MiMa tests *** FAILED *** >> java.lang.AssertionError: assertion failed: Failed to find the jar >> inside folder: /home/bjorn/spark-3.4.0/connector/connect/client/jvm/target >> at scala.Predef$.assert(Predef.scala:223) >> at >> org.apache.spark.sql.connect.client.util.IntegrationTestUtils$.findJar(IntegrationTestUtils.scala:67) >> at >> org.apache.spark.sql.connect.client.CompatibilitySuite.clientJar$lzycompute(CompatibilitySuite.scala:57) >> at >> org.apache.spark.sql.connect.client.CompatibilitySuite.clientJar(CompatibilitySuite.scala:53) >> at >> org.apache.spark.sql.connect.client.CompatibilitySuite.$anonfun$new$1(CompatibilitySuite.scala:69) >> at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85) >> at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83) >> at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104) >> at org.scalatest.Transformer.apply(Transformer.scala:22) >> at org.scalatest.Transformer.apply(Transformer.scala:20) >> ... >> - compatibility API tests: Dataset *** FAILED *** >> java.lang.AssertionError: assertion failed: Failed to find the jar >> inside folder: /home/bjorn/spark-3.4.0/connector/connect/client/jvm/target >> at scala.Predef$.assert(Predef.scala:223) >> at >> org.apache.spark.sql.connect.client.util.IntegrationTestUtils$.findJar(IntegrationTestUtils.scala:67) >> at >> org.apache.spark.sql.connect.client.CompatibilitySuite.clientJar$lzycompute(CompatibilitySuite.scala:57) >> at >> org.apache.spark.sql.connect.client.CompatibilitySuite.clientJar(CompatibilitySuite.scala:53) >> at >> org.apache.spark.sql.connect.client.CompatibilitySuite.$anonfun$new$7(CompatibilitySuite.scala:110) >> at >> scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) >> at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85) >> at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83) >> at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104) >> at org.scalatest.Transformer.apply(Transformer.scala:22) >> ... >> SparkConnectClientSuite: >> - Placeholder test: Create SparkConnectClient >> - Test connection >> - Test connection string >> - Check URI: sc://host, isCorrect: true >> - Check URI: sc://localhost/, isCorrect: true >> - Check URI: sc://localhost:1234/, isCorrect: true >> - Check URI: sc://localhost/;, isCorrect: true >> - Check URI: sc://host:123, isCorrect: true >> - Check URI: sc://host:123/;user_id=a94, isCorrect: true >> - Check URI: scc://host:12, isCorrect: false >> - Check URI: http://host, isCorrect: false >> - Check URI: sc:/host:1234/path, isCorrect: false >> - Check URI: sc://host/path, isCorrect: false >> - Check URI: sc://host/;parm1;param2, isCorrect: false >> - Check URI: sc://host:123;user_id=a94, isCorrect: false >> - Check URI: sc:///user_id=123, isCorrect: false >> - Check URI: sc://host:-4, isCorrect: false >> - Check URI: sc://:123/, isCorrect: false >> - Non user-id parameters throw unsupported errors >> DatasetSuite: >> - limit >> - select >> - filter >> - write >> UserDefinedFunctionSuite: >> - udf and encoder serialization >> Run completed in 21 seconds, 944 milliseconds. >> Total number of tests run: 389 >> Suites: completed 10, aborted 0 >> Tests: succeeded 386, failed 3, canceled 0, ignored 0, pending 0 >> *** 3 TESTS FAILED *** >> [INFO] >> ------------------------------------------------------------------------ >> [INFO] Reactor Summary for Spark Project Parent POM 3.4.0: >> [INFO] >> [INFO] Spark Project Parent POM ........................... SUCCESS [ >> 47.096 s] >> [INFO] Spark Project Tags ................................. SUCCESS [ >> 14.759 s] >> [INFO] Spark Project Sketch ............................... SUCCESS [ >> 21.628 s] >> [INFO] Spark Project Local DB ............................. SUCCESS [ >> 20.311 s] >> [INFO] Spark Project Networking ........................... SUCCESS >> [01:07 min] >> [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ >> 15.921 s] >> [INFO] Spark Project Unsafe ............................... SUCCESS [ >> 16.020 s] >> [INFO] Spark Project Launcher ............................. SUCCESS [ >> 10.873 s] >> [INFO] Spark Project Core ................................. SUCCESS >> [37:10 min] >> [INFO] Spark Project ML Local Library ..................... SUCCESS [ >> 40.841 s] >> [INFO] Spark Project GraphX ............................... SUCCESS >> [02:39 min] >> [INFO] Spark Project Streaming ............................ SUCCESS >> [05:53 min] >> [INFO] Spark Project Catalyst ............................. SUCCESS >> [11:22 min] >> [INFO] Spark Project SQL .................................. SUCCESS [ >> 02:27 h] >> [INFO] Spark Project ML Library ........................... SUCCESS >> [22:45 min] >> [INFO] Spark Project Tools ................................ SUCCESS [ >> 7.263 s] >> [INFO] Spark Project Hive ................................. SUCCESS [ >> 01:21 h] >> [INFO] Spark Project REPL ................................. SUCCESS >> [02:07 min] >> [INFO] Spark Project Assembly ............................. SUCCESS [ >> 11.704 s] >> [INFO] Kafka 0.10+ Token Provider for Streaming ........... SUCCESS [ >> 26.748 s] >> [INFO] Spark Integration for Kafka 0.10 ................... SUCCESS >> [01:44 min] >> [INFO] Kafka 0.10+ Source for Structured Streaming ........ SUCCESS >> [33:27 min] >> [INFO] Spark Project Examples ............................. SUCCESS >> [01:17 min] >> [INFO] Spark Integration for Kafka 0.10 Assembly .......... SUCCESS [ >> 27.292 s] >> [INFO] Spark Avro ......................................... SUCCESS >> [02:18 min] >> [INFO] Spark Project Connect Common ....................... SUCCESS [ >> 43.728 s] >> [INFO] Spark Project Connect Server ....................... SUCCESS >> [01:19 min] >> [INFO] Spark Project Connect Client ....................... FAILURE [ >> 53.524 s] >> [INFO] Spark Protobuf ..................................... SKIPPED >> [INFO] >> ------------------------------------------------------------------------ >> [INFO] BUILD FAILURE >> [INFO] >> ------------------------------------------------------------------------ >> [INFO] Total time: 05:58 h >> [INFO] Finished at: 2023-02-22T22:28:38+01:00 >> [INFO] >> ------------------------------------------------------------------------ >> [ERROR] Failed to execute goal >> org.scalatest:scalatest-maven-plugin:2.2.0:test (test) on project >> spark-connect-client-jvm_2.12: There are test failures -> [Help 1] >> >> ons. 22. feb. 2023 kl. 21:41 skrev Mridul Muralidharan <mri...@gmail.com >> >: >> >>> >>> Signatures, digests, etc check out fine - thanks for updating them ! >>> Checked out tag and build/tested with -Phive -Pyarn -Pmesos -Pkubernetes >>> >>> >>> The test ClientE2ETestSuite.simple udf failed [1] in "Connect Client " >>> module ... yet to test "Spark Protobuf" module due to the failure. >>> >>> >>> Regards, >>> Mridul >>> >>> [1] >>> >>> - simple udf *** FAILED *** >>> >>> io.grpc.StatusRuntimeException: INTERNAL: >>> org.apache.spark.sql.ClientE2ETestSuite >>> >>> at io.grpc.Status.asRuntimeException(Status.java:535) >>> >>> at >>> io.grpc.stub.ClientCalls$BlockingResponseStream.hasNext(ClientCalls.java:660) >>> >>> at org.apache.spark.sql.connect.client.SparkResult.org >>> $apache$spark$sql$connect$client$SparkResult$$processResponses(SparkResult.scala:50) >>> >>> at >>> org.apache.spark.sql.connect.client.SparkResult.length(SparkResult.scala:95) >>> >>> at >>> org.apache.spark.sql.connect.client.SparkResult.toArray(SparkResult.scala:112) >>> >>> at org.apache.spark.sql.Dataset.$anonfun$collect$1(Dataset.scala:2037) >>> >>> at org.apache.spark.sql.Dataset.withResult(Dataset.scala:2267) >>> >>> at org.apache.spark.sql.Dataset.collect(Dataset.scala:2036) >>> >>> at >>> org.apache.spark.sql.ClientE2ETestSuite.$anonfun$new$5(ClientE2ETestSuite.scala:65) >>> >>> at >>> scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) >>> >>> ... >>> >>> >>> >>> >>> >>> On Wed, Feb 22, 2023 at 2:07 AM Mridul Muralidharan <mri...@gmail.com> >>> wrote: >>> >>>> >>>> Thanks Xinrong ! >>>> The signature verifications are fine now ... will continue with testing >>>> the release. >>>> >>>> >>>> Regards, >>>> Mridul >>>> >>>> >>>> On Wed, Feb 22, 2023 at 1:27 AM Xinrong Meng <xinrong.apa...@gmail.com> >>>> wrote: >>>> >>>>> Hi Mridul, >>>>> >>>>> Would you please try that again? It should work now. >>>>> >>>>> On Wed, Feb 22, 2023 at 2:04 PM Mridul Muralidharan <mri...@gmail.com> >>>>> wrote: >>>>> >>>>>> >>>>>> Hi Xinrong, >>>>>> >>>>>> Was it signed with the same key as present in KEYS [1] ? >>>>>> I am seeing errors with gpg when validating. For example: >>>>>> >>>>>> >>>>>> $ gpg --verify pyspark-3.4.0.tar.gz.asc >>>>>> >>>>>> gpg: assuming signed data in 'pyspark-3.4.0.tar.gz' >>>>>> >>>>>> gpg: Signature made Tue 21 Feb 2023 05:56:05 AM CST >>>>>> >>>>>> gpg: using RSA key >>>>>> CC68B3D16FE33A766705160BA7E57908C7A4E1B1 >>>>>> >>>>>> gpg: issuer "xinr...@apache.org" >>>>>> >>>>>> gpg: Can't check signature: No public key >>>>>> >>>>>> >>>>>> >>>>>> Regards, >>>>>> Mridul >>>>>> >>>>>> [1] https://dist.apache.org/repos/dist/dev/spark/KEYS >>>>>> >>>>>> >>>>>> On Tue, Feb 21, 2023 at 10:36 PM Xinrong Meng < >>>>>> xinrong.apa...@gmail.com> wrote: >>>>>> >>>>>>> Please vote on releasing the following candidate as Apache Spark >>>>>>> version 3.4.0. >>>>>>> >>>>>>> The vote is open until 11:59pm Pacific time *February 27th* and >>>>>>> passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 >>>>>>> votes. >>>>>>> >>>>>>> [ ] +1 Release this package as Apache Spark 3.4.0 >>>>>>> [ ] -1 Do not release this package because ... >>>>>>> >>>>>>> To learn more about Apache Spark, please see >>>>>>> http://spark.apache.org/ >>>>>>> >>>>>>> The tag to be voted on is *v3.4.0-rc1* (commit >>>>>>> e2484f626bb338274665a49078b528365ea18c3b): >>>>>>> https://github.com/apache/spark/tree/v3.4.0-rc1 >>>>>>> >>>>>>> The release files, including signatures, digests, etc. can be found >>>>>>> at: >>>>>>> https://dist.apache.org/repos/dist/dev/spark/v3.4.0-rc1-bin/ >>>>>>> >>>>>>> Signatures used for Spark RCs can be found in this file: >>>>>>> https://dist.apache.org/repos/dist/dev/spark/KEYS >>>>>>> >>>>>>> The staging repository for this release can be found at: >>>>>>> >>>>>>> https://repository.apache.org/content/repositories/orgapachespark-1435 >>>>>>> >>>>>>> The documentation corresponding to this release can be found at: >>>>>>> https://dist.apache.org/repos/dist/dev/spark/v3.4.0-rc1-docs/ >>>>>>> >>>>>>> The list of bug fixes going into 3.4.0 can be found at the following >>>>>>> URL: >>>>>>> https://issues.apache.org/jira/projects/SPARK/versions/12351465 >>>>>>> >>>>>>> This release is using the release script of the tag v3.4.0-rc1. >>>>>>> >>>>>>> >>>>>>> FAQ >>>>>>> >>>>>>> ========================= >>>>>>> How can I help test this release? >>>>>>> ========================= >>>>>>> If you are a Spark user, you can help us test this release by taking >>>>>>> an existing Spark workload and running on this release candidate, >>>>>>> then >>>>>>> reporting any regressions. >>>>>>> >>>>>>> If you're working in PySpark you can set up a virtual env and install >>>>>>> the current RC and see if anything important breaks, in the >>>>>>> Java/Scala >>>>>>> you can add the staging repository to your projects resolvers and >>>>>>> test >>>>>>> with the RC (make sure to clean up the artifact cache before/after so >>>>>>> you don't end up building with a out of date RC going forward). >>>>>>> >>>>>>> =========================================== >>>>>>> What should happen to JIRA tickets still targeting 3.4.0? >>>>>>> =========================================== >>>>>>> The current list of open tickets targeted at 3.4.0 can be found at: >>>>>>> https://issues.apache.org/jira/projects/SPARK and search for >>>>>>> "Target Version/s" = 3.4.0 >>>>>>> >>>>>>> Committers should look at those and triage. Extremely important bug >>>>>>> fixes, documentation, and API tweaks that impact compatibility should >>>>>>> be worked on immediately. Everything else please retarget to an >>>>>>> appropriate release. >>>>>>> >>>>>>> ================== >>>>>>> But my bug isn't fixed? >>>>>>> ================== >>>>>>> In order to make timely releases, we will typically not hold the >>>>>>> release unless the bug in question is a regression from the previous >>>>>>> release. That being said, if there is something which is a regression >>>>>>> that has not been correctly targeted please ping me or a committer to >>>>>>> help target the issue. >>>>>>> >>>>>>> Thanks, >>>>>>> Xinrong Meng >>>>>>> >>>>>> >> >> -- >> Bjørn Jørgensen >> Vestre Aspehaug 4, 6010 Ålesund >> Norge >> >> +47 480 94 297 >> >