Hi All,
Considering there are no notable release blockers now, I am going to cut
the RC2 for Apache Spark 3.4.0.
On Fri, Feb 24, 2023 at 10:44 AM Hyukjin Kwon wrote:
> Yes we should fix. I will take a look
>
> On Thu, 23 Feb 2023 at 07:32, Jonathan Kelly
> wrote:
>
>> Thanks! I was wondering about that ClientE2ETestSuite failure today, so
>> I'm glad to know that it's also being experienced by others.
>>
>> On a similar note, I am experiencing the following error when running the
>> Python tests with Python 3.7:
>>
>> + ./python/run-tests --python-executables=python3
>> Running PySpark tests. Output is in
>> /home/ec2-user/spark/python/unit-tests.log
>> Will test against the following Python executables: ['python3']
>> Will test the following Python modules: ['pyspark-connect',
>> 'pyspark-core', 'pyspark-errors', 'pyspark-ml', 'pyspark-mllib',
>> 'pyspark-pandas', 'pyspark-pandas-slow', 'pyspark-resource', 'pyspark-sql',
>> 'pyspark-streaming']
>> python3 python_implementation is CPython
>> python3 version is: Python 3.7.16
>> Starting test(python3): pyspark.ml.tests.test_feature (temp output:
>> /home/ec2-user/spark/python/target/8ca9ab1a-05cc-4845-bf89-30d9001510bc/python3__pyspark.ml.tests.test_feature__kg6sseie.log)
>> Starting test(python3): pyspark.ml.tests.test_base (temp output:
>> /home/ec2-user/spark/python/target/f2264f3b-6b26-4e61-9452-8d6ddd7eb002/python3__pyspark.ml.tests.test_base__0902zf9_.log)
>> Starting test(python3): pyspark.ml.tests.test_algorithms (temp output:
>> /home/ec2-user/spark/python/target/d1dc4e07-e58c-4c03-abe5-09d8fab22e6a/python3__pyspark.ml.tests.test_algorithms__lh3wb2u8.log)
>> Starting test(python3): pyspark.ml.tests.test_evaluation (temp output:
>> /home/ec2-user/spark/python/target/3f42dc79-c945-4cf2-a1eb-83e72b40a9ee/python3__pyspark.ml.tests.test_evaluation__89idc7fa.log)
>> Finished test(python3): pyspark.ml.tests.test_base (16s)
>> Starting test(python3): pyspark.ml.tests.test_functions (temp output:
>> /home/ec2-user/spark/python/target/5a3b90f0-216b-4edd-9d15-6619d3e03300/python3__pyspark.ml.tests.test_functions__g5u1290s.log)
>> Traceback (most recent call last):
>> File "/usr/lib64/python3.7/runpy.py", line 193, in _run_module_as_main
>> "__main__", mod_spec)
>> File "/usr/lib64/python3.7/runpy.py", line 85, in _run_code
>> exec(code, run_globals)
>> File "/home/ec2-user/spark/python/pyspark/ml/tests/test_functions.py",
>> line 21, in
>> from pyspark.ml.functions import predict_batch_udf
>> File "/home/ec2-user/spark/python/pyspark/ml/functions.py", line 38, in
>>
>> from typing import Any, Callable, Iterator, List, Mapping, Protocol,
>> TYPE_CHECKING, Tuple, Union
>> ImportError: cannot import name 'Protocol' from 'typing'
>> (/usr/lib64/python3.7/typing.py)
>> Had test failures in pyspark.ml.tests.test_functions with python3; see
>> logs.
>>
>> I know we should move on to a newer version of Python, but isn't Python
>> 3.7 still officially supported?
>>
>> Thank you,
>> Jonathan Kelly
>>
>> On Wed, Feb 22, 2023 at 1:47 PM Herman van Hovell
>> wrote:
>>
>>> Hi All,
>>>
>>> Thanks for testing the 3.4.0 RC! I apologize for the maven testing
>>> failures for the Spark Connect Scala Client. We will try to get those
>>> sorted as soon as possible.
>>>
>>> This is an artifact of having multiple build systems, and only running
>>> CI for one (SBT). That, however, is a debate for another day :)...
>>>
>>> Cheers,
>>> Herman
>>>
>>> On Wed, Feb 22, 2023 at 5:32 PM Bjørn Jørgensen <
>>> bjornjorgen...@gmail.com> wrote:
>>>
./build/mvn clean package
I'm using ubuntu rolling, python 3.11 openjdk 17
CompatibilitySuite:
- compatibility MiMa tests *** FAILED ***
java.lang.AssertionError: assertion failed: Failed to find the jar
inside folder: /home/bjorn/spark-3.4.0/connector/connect/client/jvm/target
at scala.Predef$.assert(Predef.scala:223)
at
org.apache.spark.sql.connect.client.util.IntegrationTestUtils$.findJar(IntegrationTestUtils.scala:67)
at
org.apache.spark.sql.connect.client.CompatibilitySuite.clientJar$lzycompute(CompatibilitySuite.scala:57)
at
org.apache.spark.sql.connect.client.CompatibilitySuite.clientJar(CompatibilitySuite.scala:53)
at
org.apache.spark.sql.connect.client.CompatibilitySuite.$anonfun$new$1(CompatibilitySuite.scala:69)
at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
at org.scalatest.Transformer.apply(Transformer.scala:22)
at org.scalatest.Transformer.apply(Transformer.scala:20)
...
- compatibility API tests: Dataset *** FAILED ***
java.lang.AssertionError: assertion failed: Failed to find the jar
inside folder: /home/bjorn/spark-3.4.0/connector/connect/client/jvm/target
at scala.Predef$.assert(Predef.scala:223)
at
org
