Re: [PR] Move AWS auth dependencies to python3-saml extra [airflow]
potiuk merged PR #50449: URL: https://github.com/apache/airflow/pull/50449 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Move AWS auth dependencies to python3-saml extra [airflow]
eladkal commented on PR #50449: URL: https://github.com/apache/airflow/pull/50449#issuecomment-2900929905 @potiuk CI is green -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Move AWS auth dependencies to python3-saml extra [airflow]
potiuk commented on PR #50449: URL: https://github.com/apache/airflow/pull/50449#issuecomment-2897572327 https://github.com/apache/airflow/blob/2442b215db2e800b72af06bba80b327fce5a41d1/providers/amazon/tests/unit/amazon/aws/auth_manager/routes/test_login.py#L29 > > but this import should be available when the `apache-airflow-providers-amazon[python3-saml]` is installed in the CI. So I guess I am missing something to make this extra installed in the CI @eladkal - Yes. The problem is that in the compatibility tests and amazon tests with lowest dependencies we do not use the extra when installing the provider. In those tests we are installing just `provider` - no optional extras. Since this is now effectively an optional feature we can make the tests optional via - `importorskip` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Move AWS auth dependencies to python3-saml extra [airflow]
eladkal commented on PR #50449:
URL: https://github.com/apache/airflow/pull/50449#issuecomment-2896785688
@potiuk I am missing something here
I see
```
ERRORS
_ ERROR collecting
providers/amazon/tests/unit/amazon/aws/auth_manager/routes/test_login.py _
ImportError while importing test module
'/opt/airflow/providers/amazon/tests/unit/amazon/aws/auth_manager/routes/test_login.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/local/lib/python3.9/importlib/__init__.py:127: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
providers/amazon/tests/unit/amazon/aws/auth_manager/routes/test_login.py:29:
in
from onelogin.saml2.idp_metadata_parser import
OneLogin_Saml2_IdPMetadataParser
E ModuleNotFoundError: No module named 'onelogin'
-- generated xml file: /files/test_result-providers_amazon-sqlite.xml
--
=== short test summary info
ERROR
providers/amazon/tests/unit/amazon/aws/auth_manager/routes/test_login.py
Interrupted: 1 error during collection
=== 1 skipped, 1 warning, 1 error in 33.28s
```
This is due to
https://github.com/apache/airflow/blob/2442b215db2e800b72af06bba80b327fce5a41d1/providers/amazon/tests/unit/amazon/aws/auth_manager/routes/test_login.py#L29
but this import should be available when the
`apache-airflow-providers-amazon[python3-saml]` is installed in the CI. So I
guess I am missing something to make this extra installed in the CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] Move AWS auth dependencies to python3-saml extra [airflow]
potiuk commented on PR #50449: URL: https://github.com/apache/airflow/pull/50449#issuecomment-2873168852 Restructuring PR here https://github.com/apache/airflow/pull/50495 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Move AWS auth dependencies to python3-saml extra [airflow]
potiuk commented on PR #50449: URL: https://github.com/apache/airflow/pull/50449#issuecomment-2873170165 > We have google-auth. I don't think we can have python3-saml as Airflow extra. We can have it as amazon extra. Sure - indeed we have other cases like that. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Move AWS auth dependencies to python3-saml extra [airflow]
potiuk commented on PR #50449: URL: https://github.com/apache/airflow/pull/50449#issuecomment-2873133704 > Also some docs need to be updated. why? We still have python3-saml as extra for amazon. What we add is aws-auth extra for apache-airflow. Becuase we have `apache-airlfow` extras documented here https://airflow.apache.org/docs/apache-airflow/stable/extra-packages-ref.html#core-airflow-extras - including google-auth. Likely this page should be restructured a bit - because we split the extras to "core" and "meta-package" extras though. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Move AWS auth dependencies to python3-saml extra [airflow]
eladkal commented on PR #50449: URL: https://github.com/apache/airflow/pull/50449#issuecomment-2871107459 > Not sure if that's a good idea to add different name - it's better to keep it consistent, so maybe we should change both to be aws-auth ? We have` google-auth`. I don't think we can have `python3-saml` as Airflow extra. We can have it as `amazon` extra. > Also some docs need to be updated. why? We still have `python3-saml` as extra for amazon. What we add is `aws-auth` extra for apache-airflow. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Move AWS auth dependencies to python3-saml extra [airflow]
potiuk commented on PR #50449: URL: https://github.com/apache/airflow/pull/50449#issuecomment-2870195070 Also compatibility tests will have problem in this case - we might need to add some special case for the compat tests to pass -- the extra is not defined in Airflow 3.0.0, so we cannot really install it via `extra` - we should likely add separate installation of the missing dependencies in case of 3.0 tests - until at least the 3.0 * version tested in back-compat has the same extra defined. Or maybe just conditionally skipping the tests with `importorskip` - that woudl also work. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Move AWS auth dependencies to python3-saml extra [airflow]
potiuk commented on PR #50449: URL: https://github.com/apache/airflow/pull/50449#issuecomment-2870193803 Not sure if that's a good idea to add different name - it's better to keep it consistent, so maybe we should change both to be `aws-auth` ? Also some docs need to be updated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Move AWS auth dependencies to python3-saml extra [airflow]
eladkal commented on PR #50449: URL: https://github.com/apache/airflow/pull/50449#issuecomment-2869812703 > "python3-saml" = [ "apache-airflow-providers-amazon[python3-saml]", ] I named it `aws-auth` to avoid confusion -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Move AWS auth dependencies to python3-saml extra [airflow]
potiuk commented on PR #50449: URL: https://github.com/apache/airflow/pull/50449#issuecomment-2869652659 Fix to selective check https://github.com/apache/airflow/pull/50451 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Move AWS auth dependencies to python3-saml extra [airflow]
potiuk commented on PR #50449: URL: https://github.com/apache/airflow/pull/50449#issuecomment-2869647581 Yep. That's what I thought. You will need to add `python3-saml` extra to the `apache-airflow` meta-project `pyproject.toml` (the one in the root package, similarly to the s3fs one: ``` "s3fs" = [ # This is required for support of S3 file system which uses aiobotocore # which can have a conflict with boto3 as mentioned in aiobotocore extra "apache-airflow-providers-amazon[s3fs]", ] ``` Like this: ``` "python3-saml" = [ "apache-airflow-providers-amazon[python3-saml]", ] ``` And it should be added as an extra installed when `all` extras are installed (same pyproject.toml) : ``` "all" = [ "apache-airflow[aiobotocore,apache-atlas,apache-webhdfs,async,cloudpickle,github-enterprise,google-auth,graphviz,kerberos,ldap,otel,pandas,polars,rabbitmq,s3fs,sentry,statsd,uv]", ``` Otherwise CI image wil not install this extra. We use `all` extra to build the `ci` image - and if the extra is not specified in `all` - directly or transitively, it will not be installed when `ci` image is built. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Move AWS auth dependencies to python3-saml extra [airflow]
potiuk commented on PR #50449: URL: https://github.com/apache/airflow/pull/50449#issuecomment-2869630879 Closed/reopened to trigger full tests -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Move AWS auth dependencies to python3-saml extra [airflow]
potiuk commented on PR #50449: URL: https://github.com/apache/airflow/pull/50449#issuecomment-2869630671 Hm.. I think this one will fail provider tests - selective checks did not trigger provider tests on this one (fix is coming to that one) but I have a feeling this one will fail if we run full tests. Let's see. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Move AWS auth dependencies to python3-saml extra [airflow]
potiuk closed pull request #50449: Move AWS auth dependencies to python3-saml extra URL: https://github.com/apache/airflow/pull/50449 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] Move AWS auth dependencies to python3-saml extra [airflow]
eladkal opened a new pull request, #50449: URL: https://github.com/apache/airflow/pull/50449 The workaround of set xmlsec and lxml is relevant only for aws auth manager thus we need these as part of the python3-saml extra rather than as direct dependency of the provider -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
