aneagoe opened a new issue #14659: URL: https://github.com/apache/airflow/issues/14659
**Apache Airflow version**: 1.10.12 **Kubernetes version: 1.16 **What happened**: As per https://issues.apache.org/jira/browse/AIRFLOW-5164, when using mongodb replica set URIs (of form mongodb://mongodb0.example.com:27017,mongodb1.example.com:27017,mongodb2.example.com:27017/admin?replicaSet=myRepl) then an error is thrown as airflow tries to cram everything after `:` as port. (ie `ValueError: invalid literal for int() with base 10: '27017,mongodb1.example.com:27017,mongodb2.example.com:27017`). **What you expected to happen**: Airflow should properly interpret the replicaset mongodb URI formats. **How to reproduce it**: ``` import os os.environ["AIRFLOW_CONN_MONGO_STANDALONE"] = "mongodb://mongodb0.example.com:27017/admin" os.environ["AIRFLOW_CONN_MONGO_REPLICASET"] = "mongodb://mongodb0.example.com:27017,mongodb1.example.com:27017,mongodb2.example.com:27017/admin?replicaSet=myRepl" from airflow.contrib.hooks.mongo_hook import MongoHook hook = MongoHook('mongo_standalone') print(hook) hook = MongoHook('mongo_replicaset') print(hook) ``` Example: ``` >>> import os >>> os.environ["AIRFLOW_CONN_MONGO_STANDALONE"] = "mongodb://mongodb0.example.com:27017/admin" >>> os.environ["AIRFLOW_CONN_MONGO_REPLICASET"] = "mongodb://mongodb0.example.com:27017,mongodb1.example.com:27017,mongodb2.example.com:27017/admin?replicaSet=myRepl" >>> from airflow.contrib.hooks.mongo_hook import MongoHook >>> hook = MongoHook('mongo_standalone') [2021-03-08 09:34:06,166] {base_hook.py:89} INFO - Using connection to: id: mongo_standalone. Host: mongodb0.example.com, Port: 27017, Schema: admin, Login: None, Password: None, extra: None >>> print(hook) <airflow.contrib.hooks.mongo_hook.MongoHook object at 0x7f8dccc0aa50> >>> hook = MongoHook('mongo_replicaset') Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/airflow/.local/lib/python3.7/site-packages/airflow/contrib/hooks/mongo_hook.py", line 44, in __init__ self.connection = self.get_connection(conn_id) File "/home/airflow/.local/lib/python3.7/site-packages/airflow/hooks/base_hook.py", line 87, in get_connection conn = random.choice(list(cls.get_connections(conn_id))) File "/home/airflow/.local/lib/python3.7/site-packages/airflow/hooks/base_hook.py", line 83, in get_connections return secrets.get_connections(conn_id) File "/home/airflow/.local/lib/python3.7/site-packages/airflow/secrets/__init__.py", line 55, in get_connections conn_list = secrets_backend.get_connections(conn_id=conn_id) File "/home/airflow/.local/lib/python3.7/site-packages/airflow/secrets/base_secrets.py", line 67, in get_connections conn = Connection(conn_id=conn_id, uri=conn_uri) File "<string>", line 4, in __init__ File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/state.py", line 433, in _initialize_instance manager.dispatch.init_failure(self, args, kwargs) File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ with_traceback=exc_tb, File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 182, in raise_ raise exception File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/state.py", line 430, in _initialize_instance return manager.original_init(*mixed[1:], **kwargs) File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/connection.py", line 121, in __init__ self.parse_from_uri(uri) File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/connection.py", line 146, in parse_from_uri self.port = uri_parts.port File "/usr/local/lib/python3.7/urllib/parse.py", line 169, in port port = int(port, 10) ValueError: invalid literal for int() with base 10: '27017,mongodb1.example.com:27017,mongodb2.example.com:27017' >>> print(hook) <airflow.contrib.hooks.mongo_hook.MongoHook object at 0x7f8dccc0aa50> >>> ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
