I am seeing a bunch of python related (43) failures in the sql module (for
example [1]) ... I am currently on Python 3.11.6, java 8.
Not sure if ubuntu modified anything from under me, thoughts ?
I am currently testing this against an older branch to make sure it is not
an issue with my desktop.
Regards,
Mridul
[1]
org.apache.spark.sql.IntegratedUDFTestUtils.shouldTestGroupedAggPandasUDFs
was false (QueryCompilationErrorsSuite.scala:112)
Traceback (most recent call last):
File "/home/mridul/work/apache/vote/spark/python/pyspark/serializers.py",
line 458, in dumps
return cloudpickle.dumps(obj, pickle_protocol)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/mridul/work/apache/vote/spark/python/pyspark/cloudpickle/cloudpickle_fast.py",
line 73, in dumps
cp.dump(obj)
File
"/home/mridul/work/apache/vote/spark/python/pyspark/cloudpickle/cloudpickle_fast.py",
line 602, in dump
return Pickler.dump(self, obj)
^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/mridul/work/apache/vote/spark/python/pyspark/cloudpickle/cloudpickle_fast.py",
line 692, in reducer_override
return self._function_reduce(obj)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/mridul/work/apache/vote/spark/python/pyspark/cloudpickle/cloudpickle_fast.py",
line 565, in _function_reduce
return self._dynamic_function_reduce(obj)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/mridul/work/apache/vote/spark/python/pyspark/cloudpickle/cloudpickle_fast.py",
line 546, in _dynamic_function_reduce
state = _function_getstate(func)
^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/mridul/work/apache/vote/spark/python/pyspark/cloudpickle/cloudpickle_fast.py",
line 157, in _function_getstate
f_globals_ref = _extract_code_globals(func.__code__)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/mridul/work/apache/vote/spark/python/pyspark/cloudpickle/cloudpickle.py",
line 334, in _extract_code_globals
out_names = {names[oparg]: None for _, oparg in _walk_global_ops(co)}
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/mridul/work/apache/vote/spark/python/pyspark/cloudpickle/cloudpickle.py",
line 334, in <dictcomp>
out_names = {names[oparg]: None for _, oparg in _walk_global_ops(co)}
~~~~~^^^^^^^
IndexError: tuple index out of range
Traceback (most recent call last):
File "/home/mridul/work/apache/vote/spark/python/pyspark/serializers.py",
line 458, in dumps
return cloudpickle.dumps(obj, pickle_protocol)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/mridul/work/apache/vote/spark/python/pyspark/cloudpickle/cloudpickle_fast.py",
line 73, in dumps
cp.dump(obj)
File
"/home/mridul/work/apache/vote/spark/python/pyspark/cloudpickle/cloudpickle_fast.py",
line 602, in dump
return Pickler.dump(self, obj)
^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/mridul/work/apache/vote/spark/python/pyspark/cloudpickle/cloudpickle_fast.py",
line 692, in reducer_override
return self._function_reduce(obj)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/mridul/work/apache/vote/spark/python/pyspark/cloudpickle/cloudpickle_fast.py",
line 565, in _function_reduce
return self._dynamic_function_reduce(obj)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/mridul/work/apache/vote/spark/python/pyspark/cloudpickle/cloudpickle_fast.py",
line 546, in _dynamic_function_reduce
state = _function_getstate(func)
^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/mridul/work/apache/vote/spark/python/pyspark/cloudpickle/cloudpickle_fast.py",
line 157, in _function_getstate
f_globals_ref = _extract_code_globals(func.__code__)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/mridul/work/apache/vote/spark/python/pyspark/cloudpickle/cloudpickle.py",
line 334, in _extract_code_globals
out_names = {names[oparg]: None for _, oparg in _walk_global_ops(co)}
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/mridul/work/apache/vote/spark/python/pyspark/cloudpickle/cloudpickle.py",
line 334, in <dictcomp>
out_names = {names[oparg]: None for _, oparg in _walk_global_ops(co)}
~~~~~^^^^^^^
IndexError: tuple index out of range
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/home/mridul/work/apache/vote/spark/python/pyspark/serializers.py",
line 468, in dumps
raise pickle.PicklingError(msg)
_pickle.PicklingError: Could not serialize object: IndexError: tuple index
out of range
- UNSUPPORTED_FEATURE: Using Python UDF with unsupported join condition ***
FAILED ***
On Sun, Dec 10, 2023 at 9:05 PM L. C. Hsieh <vii...@gmail.com> wrote:
> +1
>
> On Sun, Dec 10, 2023 at 6:15 PM Kent Yao <y...@apache.org> wrote:
> >
> > +1(non-binding
> >
> > Kent Yao
> >
> > Yuming Wang <yumw...@apache.org> 于2023年12月11日周一 09:33写道：
> > >
> > > +1
> > >
> > > On Mon, Dec 11, 2023 at 5:55 AM Dongjoon Hyun <dongj...@apache.org>
> wrote:
> > >>
> > >> +1
> > >>
> > >> Dongjoon
> > >>
> > >> On 2023/12/08 21:41:00 Dongjoon Hyun wrote:
> > >> > Please vote on releasing the following candidate as Apache Spark
> version
> > >> > 3.3.4.
> > >> >
> > >> > The vote is open until December 15th 1AM (PST) and passes if a
> majority +1
> > >> > PMC votes are cast, with a minimum of 3 +1 votes.
> > >> >
> > >> > [ ] +1 Release this package as Apache Spark 3.3.4
> > >> > [ ] -1 Do not release this package because ...
> > >> >
> > >> > To learn more about Apache Spark, please see
> https://spark.apache.org/
> > >> >
> > >> > The tag to be voted on is v3.3.4-rc1 (commit
> > >> > 18db204995b32e87a650f2f09f9bcf047ddafa90)
> > >> > https://github.com/apache/spark/tree/v3.3.4-rc1
> > >> >
> > >> > The release files, including signatures, digests, etc. can be found
> at:
> > >> >
> > >> > https://dist.apache.org/repos/dist/dev/spark/v3.3.4-rc1-bin/
> > >> >
> > >> >
> > >> > Signatures used for Spark RCs can be found in this file:
> > >> >
> > >> > https://dist.apache.org/repos/dist/dev/spark/KEYS
> > >> >
> > >> >
> > >> > The staging repository for this release can be found at:
> > >> >
> > >> >
> https://repository.apache.org/content/repositories/orgapachespark-1451/
> > >> >
> > >> >
> > >> > The documentation corresponding to this release can be found at:
> > >> >
> > >> > https://dist.apache.org/repos/dist/dev/spark/v3.3.4-rc1-docs/
> > >> >
> > >> >
> > >> > The list of bug fixes going into 3.3.4 can be found at the
> following URL:
> > >> >
> > >> > https://issues.apache.org/jira/projects/SPARK/versions/12353505
> > >> >
> > >> >
> > >> > This release is using the release script of the tag v3.3.4-rc1.
> > >> >
> > >> >
> > >> > FAQ
> > >> >
> > >> >
> > >> > =========================
> > >> >
> > >> > How can I help test this release?
> > >> >
> > >> > =========================
> > >> >
> > >> >
> > >> >
> > >> > If you are a Spark user, you can help us test this release by taking
> > >> >
> > >> > an existing Spark workload and running on this release candidate,
> then
> > >> >
> > >> > reporting any regressions.
> > >> >
> > >> >
> > >> >
> > >> > If you're working in PySpark you can set up a virtual env and
> install
> > >> >
> > >> > the current RC and see if anything important breaks, in the
> Java/Scala
> > >> >
> > >> > you can add the staging repository to your projects resolvers and
> test
> > >> >
> > >> > with the RC (make sure to clean up the artifact cache before/after
> so
> > >> >
> > >> > you don't end up building with a out of date RC going forward).
> > >> >
> > >> >
> > >> >
> > >> > ===========================================
> > >> >
> > >> > What should happen to JIRA tickets still targeting 3.3.4?
> > >> >
> > >> > ===========================================
> > >> >
> > >> >
> > >> >
> > >> > The current list of open tickets targeted at 3.3.4 can be found at:
> > >> >
> > >> > https://issues.apache.org/jira/projects/SPARK and search for
> "Target
> > >> > Version/s" = 3.3.4
> > >> >
> > >> >
> > >> > Committers should look at those and triage. Extremely important bug
> > >> >
> > >> > fixes, documentation, and API tweaks that impact compatibility
> should
> > >> >
> > >> > be worked on immediately. Everything else please retarget to an
> > >> >
> > >> > appropriate release.
> > >> >
> > >> >
> > >> >
> > >> > ==================
> > >> >
> > >> > But my bug isn't fixed?
> > >> >
> > >> > ==================
> > >> >
> > >> >
> > >> >
> > >> > In order to make timely releases, we will typically not hold the
> > >> >
> > >> > release unless the bug in question is a regression from the previous
> > >> >
> > >> > release. That being said, if there is something which is a
> regression
> > >> >
> > >> > that has not been correctly targeted please ping me or a committer
> to
> > >> >
> > >> > help target the issue.
> > >> >
> > >>
> > >> ---------------------------------------------------------------------
> > >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
> > >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>
