Taragolis commented on issue #35307:
URL: https://github.com/apache/airflow/issues/35307#issuecomment-1788030750
Assumption 1
---
Maybe it happen because we use
[dill==3.1.1.1](https://github.com/apache/airflow/blob/eb0f00cf9c4c71fd871c970857da481d097a4733/constraints-3.11.txt#L263)
which was released at Sep 28, 2019 and Python 3.11 was released Oct 24, 2022?
```console
root@567a6cdeef08:/opt/airflow# pipdeptree --packages dill -r
Warning!! Cyclic dependencies found:
* sphinxcontrib-applehelp => sphinx => sphinxcontrib-applehelp
* sphinxcontrib-devhelp => sphinx => sphinxcontrib-devhelp
* sphinxcontrib-htmlhelp => sphinx => sphinxcontrib-htmlhelp
* sphinxcontrib-qthelp => sphinx => sphinxcontrib-qthelp
* sphinxcontrib-serializinghtml => sphinx => sphinxcontrib-serializinghtml
* sphinx => sphinxcontrib-applehelp => sphinx
------------------------------------------------------------------------
dill==0.3.1.1
├── apache-airflow==2.8.0.dev0 [requires: dill>=0.2.2]
└── apache-beam==2.51.0 [requires: dill>=0.3.1.1,<0.3.2]
```
Ok, let's break dependencies, install dill==0.3.7 and have a look what is
happen
```console
warnings summary:
============================
tests/operators/test_python.py::TestPythonVirtualenvOperator.test_airflow_context:1004
-> connexion/decorators/validation.py:16 DeprecationWarning('Accessing
jsonschema.draft4_format_checker is deprecated and will be removed in a future
release. Instead, use the FORMAT_CHECKER attribute on the corresponding
Validator.')
tests/operators/test_python.py::TestPythonVirtualenvOperator.test_airflow_context:1004
-> connexion/json_schema.py:16 DeprecationWarning('jsonschema.RefResolver is
deprecated as of v4.18.0, in favor of the
https://github.com/python-jsonschema/referencing library, which provides more
compliant referencing behavior as well as more flexible APIs for customization.
A future release will remove RefResolver. Please file a feature request (on
referencing) if you are missing an API for the kind of customization you need.')
tests/operators/test_python.py::TestPythonVirtualenvOperator.test_airflow_context:1004
-> connexion/json_schema.py:17
DeprecationWarning('jsonschema.exceptions.RefResolutionError is deprecated as
of version 4.18.0. If you wish to catch potential reference resolution errors,
directly catch referencing.exceptions.Unresolvable.')
tests/operators/test_python.py::TestPythonVirtualenvOperator.test_airflow_context:1004
-> dill/_dill.py:412 PicklingWarning('Cannot locate reference to <class
'pendulum.duration.Duration'>.')
tests/operators/test_python.py::TestPythonVirtualenvOperator.test_airflow_context:1004
-> dill/_dill.py:412 PicklingWarning('Cannot pickle <class
'pendulum.duration.Duration'>: pendulum.duration.Duration has recursive
self-references that trigger a RecursionError.')
tests/operators/test_python.py::TestPythonVirtualenvOperator.test_airflow_context:1004
-> dill/_dill.py:412 PicklingWarning('Cannot locate reference to <class
'pendulum.datetime.DateTime'>.')
tests/operators/test_python.py::TestPythonVirtualenvOperator.test_airflow_context:1004
-> dill/_dill.py:412 PicklingWarning('Cannot pickle <class
'pendulum.datetime.DateTime'>: pendulum.datetime.DateTime has recursive
self-references that trigger a RecursionError.')
tests/operators/test_python.py::TestPythonVirtualenvOperator.test_airflow_context:1004
-> dill/_dill.py:412 PicklingWarning('Cannot locate reference to <class
'pendulum.time.Time'>.')
tests/operators/test_python.py::TestPythonVirtualenvOperator.test_airflow_context:1004
-> dill/_dill.py:412 PicklingWarning('Cannot pickle <class
'pendulum.time.Time'>: pendulum.time.Time has recursive self-references that
trigger a RecursionError.')
All Warning errors can be found in the warnings.txt file.
==============================================================================
short test summary info
==============================================================================
FAILED
tests/operators/test_python.py::TestPythonVirtualenvOperator::test_airflow_context
- subprocess.CalledProcessError: Command '['/tmp/venvngz2drh_/bin/python',
'/tmp/venvngz2drh_/script.py', '/tmp/venvngz2drh_/script.in',
'/tmp/venvngz2drh_/script.out', '/tmp/venvngz2drh_/string_args.txt',
'/tmp/venvngz2drh_/termination.log']' returned non-zero exit status 1.
=================================================================================
1 failed in 6.54s
=================================================================================
root@6c8587e657c9:/opt/airflow#
```
Assumption 2
---
Hmm, in the new version I could clearly see the warnings about pendulum.
Let's try pendulum 3.0.0b1, for allow run Airflow with this version I
switched to https://github.com/apache/airflow/pull/34744
And with combination of `pendulum==3.0.0b1` and `dill=0.3.7` the test is pass
```
warnings summary:
============================
tests/operators/test_python.py::TestPythonVirtualenvOperator.test_airflow_context:1004
-> connexion/decorators/validation.py:16 DeprecationWarning('Accessing
jsonschema.draft4_format_checker is deprecated and will be removed in a future
release. Instead, use the FORMAT_CHECKER attribute on the corresponding
Validator.')
tests/operators/test_python.py::TestPythonVirtualenvOperator.test_airflow_context:1004
-> connexion/json_schema.py:16 DeprecationWarning('jsonschema.RefResolver is
deprecated as of v4.18.0, in favor of the
https://github.com/python-jsonschema/referencing library, which provides more
compliant referencing behavior as well as more flexible APIs for customization.
A future release will remove RefResolver. Please file a feature request (on
referencing) if you are missing an API for the kind of customization you need.')
tests/operators/test_python.py::TestPythonVirtualenvOperator.test_airflow_context:1004
-> connexion/json_schema.py:17
DeprecationWarning('jsonschema.exceptions.RefResolutionError is deprecated as
of version 4.18.0. If you wish to catch potential reference resolution errors,
directly catch referencing.exceptions.Unresolvable.')
tests/operators/test_python.py::TestPythonVirtualenvOperator.test_airflow_context:1004
-> dill/_dill.py:412 PicklingWarning('Cannot locate reference to <class
'pendulum.duration.Duration'>.')
tests/operators/test_python.py::TestPythonVirtualenvOperator.test_airflow_context:1004
-> dill/_dill.py:412 PicklingWarning('Cannot pickle <class
'pendulum.duration.Duration'>: pendulum.duration.Duration has recursive
self-references that trigger a RecursionError.')
tests/operators/test_python.py::TestPythonVirtualenvOperator.test_airflow_context:1004
-> dill/_dill.py:412 PicklingWarning('Cannot locate reference to <class
'pendulum.datetime.DateTime'>.')
tests/operators/test_python.py::TestPythonVirtualenvOperator.test_airflow_context:1004
-> dill/_dill.py:412 PicklingWarning('Cannot pickle <class
'pendulum.datetime.DateTime'>: pendulum.datetime.DateTime has recursive
self-references that trigger a RecursionError.')
tests/operators/test_python.py::TestPythonVirtualenvOperator.test_airflow_context:1004
-> dill/_dill.py:412 PicklingWarning('Cannot locate reference to <class
'pendulum.time.Time'>.')
tests/operators/test_python.py::TestPythonVirtualenvOperator.test_airflow_context:1004
-> dill/_dill.py:412 PicklingWarning('Cannot pickle <class
'pendulum.time.Time'>: pendulum.time.Time has recursive self-references that
trigger a RecursionError.')
All Warning errors can be found in the warnings.txt file.
=================================================================================
1 passed in 6.39s
=================================================================================
```
If return any of this packages back then test failed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
