potiuk commented on PR #50417: URL: https://github.com/apache/airflow/pull/50417#issuecomment-2873359762
> Why did this start failing suddenly? That code has not changed since 2023? Or has it always been failing to generate docs since then? > > Is it possible to share the exception that was being raised? Tose examples simply failed to be imported - and that's what caused the issue. I fixed generation of source links in amazon provider that were missing because of that. You can see it here: https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/operators/dms.html -> most of the amazon exmples have now links to sources. I regenerated the docs for past few releases after the fix but you can see how it looked before https://airflow.apache.org/docs/apache-airflow-providers-amazon/9.4.0/operators/dms.html (see the missing [sources] link in examples). The thing is that our "exampleinclude" will silently swallow such exceptions and the only thing you see is missing `[sources]' - I might want to fix it soon and have it fail in this case, I have few small improvements to the exampleinclude that I want to impleement. The error was caused by `hook.conn.exceptions` trying to actually connect to SSM without having credentials - because some of the wrapper around `exceptions` property. It tries to get environment and set a default value for it from SSM. It's easy to repro by reverting the change and using the new sphinx autobuild workflow (you will see many exceptions thrown and the generated docs will have no `[sources]` link: ``` git revert c5ed10010a5a5a7b34d5fb0d8d70a6d70b35ddb3 cd providers/amazon uv run --group docs build-docs --autobuild ``` That's the exception: ```python Traceback (most recent call last): File "/home/jarek/code/airflow/.venv/lib/python3.12/site-packages/sphinx/pycode/__init__.py", line 38, in get_module_source mod = import_module(modname) ^^^^^^^^^^^^^^^^^^^^^^ File "/home/jarek/.local/share/uv/python/cpython-3.12.7-linux-x86_64-gnu/lib/python3.12/importlib/__init__.py", line 90, in import_module return _bootstrap._gcd_import(name[level:], package, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "<frozen importlib._bootstrap>", line 1387, in _gcd_import File "<frozen importlib._bootstrap>", line 1360, in _find_and_load File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 935, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 995, in exec_module File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed File "/home/jarek/code/airflow/providers/amazon/tests/system/amazon/aws/example_sql_to_s3.py", line 47, in <module> SystemTestContextBuilder().add_variable(SECURITY_GROUP_KEY).add_variable(CLUSTER_SUBNET_GROUP_KEY).build() ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/jarek/code/airflow/providers/amazon/tests/system/amazon/aws/utils/__init__.py", line 191, in __init__ self.env_id = set_env_id() ^^^^^^^^^^^^ File "/home/jarek/code/airflow/providers/amazon/tests/system/amazon/aws/utils/__init__.py", line 275, in set_env_id env_id: str = str(fetch_variable(ENV_ID_ENVIRON_KEY, DEFAULT_ENV_ID)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/jarek/code/airflow/providers/amazon/tests/system/amazon/aws/utils/__init__.py", line 258, in fetch_variable value: str | None = os.getenv(key, _fetch_from_ssm(key, test_name)) or default_value ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/jarek/code/airflow/providers/amazon/tests/system/amazon/aws/utils/__init__.py", line 116, in _fetch_from_ssm except hook.conn.exceptions.ParameterNotFound as e: ^^^^^^^^^ File "/home/jarek/.local/share/uv/python/cpython-3.12.7-linux-x86_64-gnu/lib/python3.12/functools.py", line 993, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/home/jarek/code/airflow/providers/amazon/src/airflow/providers/amazon/aws/hooks/base_aws.py", line 746, in conn return self.get_client_type(region_name=self.region_name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/jarek/code/airflow/providers/amazon/src/airflow/providers/amazon/aws/hooks/base_aws.py", line 708, in get_client_type if not isinstance(session, boto3.session.Session): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ TypeError: isinstance() arg 2 must be a type, a tuple of types, or a union The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/jarek/code/airflow/devel-common/src/sphinx_exts/exampleinclude.py", line 151, in register_source analyzer = ModuleAnalyzer.for_module(modname) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/jarek/code/airflow/.venv/lib/python3.12/site-packages/sphinx/pycode/__init__.py", line 114, in for_module filename, source = cls.get_module_source(modname) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/jarek/code/airflow/.venv/lib/python3.12/site-packages/sphinx/pycode/__init__.py", line 40, in get_module_source raise PycodeError('error importing %r' % modname, err) from err sphinx.errors.PycodeError: error importing 'system.amazon.aws.example_sql_to_s3' (exception was: TypeError('isinstance() arg 2 must be a type, a tuple of types, or a union')) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
