[jira] [Commented] (AIRFLOW-1330) Connection.parse_from_uri doesn't work for google_cloud_platform and so on
[
https://issues.apache.org/jira/browse/AIRFLOW-1330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235496#comment-16235496
]
Cedrik Neumann commented on AIRFLOW-1330:
-
I just wanted to add that option (2) also provides the flexibility to add more
exotic connections (jdbc, ...) without making assumptions on the form of the
connection string.
> Connection.parse_from_uri doesn't work for google_cloud_platform and so on
> --
>
> Key: AIRFLOW-1330
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1330
> Project: Apache Airflow
> Issue Type: Bug
> Components: cli
>Reporter: Yu Ishikawa
>Assignee: Shintaro Murakami
>Priority: Major
> Fix For: 1.9.0
>
>
> h2. Overview
> {{Connection.parse_from_uri}} doesn't work for some types like
> {{google_cloud_platform}} whose type name includes under scores. Since
> `urllib.parse.urlparse()` which is used in {{Connection.parse_from_url}}
> doesn't support a schema name which include under scores.
> So, airflow's CLI doesn't work when a given connection URI includes under
> scores like {{google_cloud_platform://X}}.
> h3. Workaround
> https://medium.com/@yuu.ishikawa/apache-airflow-how-to-add-a-connection-to-google-cloud-with-cli-af2cc8df138d
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AIRFLOW-1330) Connection.parse_from_uri doesn't work for google_cloud_platform and so on
[
https://issues.apache.org/jira/browse/AIRFLOW-1330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16217303#comment-16217303
]
ASF subversion and git services commented on AIRFLOW-1330:
--
Commit 6e5e9d282e2048c16cde75d7dabff74cd5717646 in incubator-airflow's branch
refs/heads/v1-9-test from [~mrkm4ntr]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=6e5e9d2 ]
[AIRFLOW-1330] Add conn_type argument to CLI when adding connection
Closes #2525 from mrkm4ntr/airflow-1330
> Connection.parse_from_uri doesn't work for google_cloud_platform and so on
> --
>
> Key: AIRFLOW-1330
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1330
> Project: Apache Airflow
> Issue Type: Bug
> Components: cli
>Reporter: Yu Ishikawa
>Assignee: Shintaro Murakami
> Fix For: 1.10.0
>
>
> h2. Overview
> {{Connection.parse_from_uri}} doesn't work for some types like
> {{google_cloud_platform}} whose type name includes under scores. Since
> `urllib.parse.urlparse()` which is used in {{Connection.parse_from_url}}
> doesn't support a schema name which include under scores.
> So, airflow's CLI doesn't work when a given connection URI includes under
> scores like {{google_cloud_platform://X}}.
> h3. Workaround
> https://medium.com/@yuu.ishikawa/apache-airflow-how-to-add-a-connection-to-google-cloud-with-cli-af2cc8df138d
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AIRFLOW-1330) Connection.parse_from_uri doesn't work for google_cloud_platform and so on
[
https://issues.apache.org/jira/browse/AIRFLOW-1330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211311#comment-16211311
]
ASF subversion and git services commented on AIRFLOW-1330:
--
Commit 2f107d8a30910fd025774004d5c4c95407ed55c5 in incubator-airflow's branch
refs/heads/master from [~mrkm4ntr]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=2f107d8 ]
[AIRFLOW-1330] Add conn_type argument to CLI when adding connection
Closes #2525 from mrkm4ntr/airflow-1330
> Connection.parse_from_uri doesn't work for google_cloud_platform and so on
> --
>
> Key: AIRFLOW-1330
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1330
> Project: Apache Airflow
> Issue Type: Bug
> Components: cli
>Reporter: Yu Ishikawa
>Assignee: Shintaro Murakami
>
> h2. Overview
> {{Connection.parse_from_uri}} doesn't work for some types like
> {{google_cloud_platform}} whose type name includes under scores. Since
> `urllib.parse.urlparse()` which is used in {{Connection.parse_from_url}}
> doesn't support a schema name which include under scores.
> So, airflow's CLI doesn't work when a given connection URI includes under
> scores like {{google_cloud_platform://X}}.
> h3. Workaround
> https://medium.com/@yuu.ishikawa/apache-airflow-how-to-add-a-connection-to-google-cloud-with-cli-af2cc8df138d
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AIRFLOW-1330) Connection.parse_from_uri doesn't work for google_cloud_platform and so on
[
https://issues.apache.org/jira/browse/AIRFLOW-1330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16174120#comment-16174120
]
Shintaro Murakami commented on AIRFLOW-1330:
Fixed it with the option (2). In addition, made it possible to parse even the
scheme containing underscores. Since we did not want to add all setters to
Connection class, made {{conn_uri}} and {{conn_type}} be exclusive.
> Connection.parse_from_uri doesn't work for google_cloud_platform and so on
> --
>
> Key: AIRFLOW-1330
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1330
> Project: Apache Airflow
> Issue Type: Bug
> Components: cli
>Reporter: Yu Ishikawa
>Assignee: Shintaro Murakami
>
> h2. Overview
> {{Connection.parse_from_uri}} doesn't work for some types like
> {{google_cloud_platform}} whose type name includes under scores. Since
> `urllib.parse.urlparse()` which is used in {{Connection.parse_from_url}}
> doesn't support a schema name which include under scores.
> So, airflow's CLI doesn't work when a given connection URI includes under
> scores like {{google_cloud_platform://X}}.
> h3. Workaround
> https://medium.com/@yuu.ishikawa/apache-airflow-how-to-add-a-connection-to-google-cloud-with-cli-af2cc8df138d
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AIRFLOW-1330) Connection.parse_from_uri doesn't work for google_cloud_platform and so on
[
https://issues.apache.org/jira/browse/AIRFLOW-1330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16154209#comment-16154209
]
Yu Ishikawa commented on AIRFLOW-1330:
--
That's a good idea. That would work.
FYI, we have another approach not to allow to include underscores in a
connection type name, such as `google-cloud-platform`. It would be more
natural, because the issue was caused by connection types as invalid schemas.
> Connection.parse_from_uri doesn't work for google_cloud_platform and so on
> --
>
> Key: AIRFLOW-1330
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1330
> Project: Apache Airflow
> Issue Type: Bug
> Components: cli
>Reporter: Yu Ishikawa
>Assignee: Shintaro Murakami
>
> h2. Overview
> {{Connection.parse_from_uri}} doesn't work for some types like
> {{google_cloud_platform}} whose type name includes under scores. Since
> `urllib.parse.urlparse()` which is used in {{Connection.parse_from_url}}
> doesn't support a schema name which include under scores.
> So, airflow's CLI doesn't work when a given connection URI includes under
> scores like {{google_cloud_platform://X}}.
> h3. Workaround
> https://medium.com/@yuu.ishikawa/apache-airflow-how-to-add-a-connection-to-google-cloud-with-cli-af2cc8df138d
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AIRFLOW-1330) Connection.parse_from_uri doesn't work for google_cloud_platform and so on
[
https://issues.apache.org/jira/browse/AIRFLOW-1330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16145674#comment-16145674
]
Yu Ishikawa commented on AIRFLOW-1330:
--
Let's summarize our discussion. We have 3 options to fix the issue, offering
another command line options to register a connection ID. I think {{3}} would
be the most natural.
What do you think about it? If [~criccomini] agree with me, we would like
[~mrkm4ntr] to modify his PR to fit to my proposal.
1. Support {{\-\-conn_url}} and {{--conn_type}} like [~mrkm4ntr] 's PR
{noformat}
airflow connections --add --conn_url //mysql-host:3306 --conn_type mysql
{noformat}
* Pros
** We don't add more command line options except for --conn_type
* Cons
** the format of --conn_url is a little confusing
2. Support both of {{\-\-conn_url}} and combinations of {{--conn_type}} and
like this
{noformat}
airflow connections --add --conn_id --conn_url mysql://fake-host:3306
or
airflow connections --add --conn_id --conn_type mysql --conn_host fake-host
--conn_port 3306
{noformat}
* Pros
** Users keep using the existing way to register a connection ID, as far as the
URL schema is valid
* Cons
** It's a little annoying for users to understand cases that the existing way
doesn't work.
3. Support only combinations {{--con_type}} and like this option
{noformat}
airflow connections --add --conn_id --conn_type mysql --conn_host fake-host
--conn_port 3306
{noformat}
* Pros
** All users have to do is to learn the new way to register a connection ID. It
would not be confusing.
* Cons
** We throw away the existing way to do that.
> Connection.parse_from_uri doesn't work for google_cloud_platform and so on
> --
>
> Key: AIRFLOW-1330
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1330
> Project: Apache Airflow
> Issue Type: Bug
> Components: cli
>Reporter: Yu Ishikawa
>
> h2. Overview
> {{Connection.parse_from_uri}} doesn't work for some types like
> {{google_cloud_platform}} whose type name includes under scores. Since
> `urllib.parse.urlparse()` which is used in {{Connection.parse_from_url}}
> doesn't support a schema name which include under scores.
> So, airflow's CLI doesn't work when a given connection URI includes under
> scores like {{google_cloud_platform://X}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AIRFLOW-1330) Connection.parse_from_uri doesn't work for google_cloud_platform and so on
[
https://issues.apache.org/jira/browse/AIRFLOW-1330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137840#comment-16137840
]
Yu Ishikawa commented on AIRFLOW-1330:
--
[~criccomini] Of course, we can also throw away {{--conn_url}}, because it
should be confusing with begginers who want to register connections whose
schema doesn't follow the RFC. What do you think?
> Connection.parse_from_uri doesn't work for google_cloud_platform and so on
> --
>
> Key: AIRFLOW-1330
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1330
> Project: Apache Airflow
> Issue Type: Bug
> Components: cli
>Reporter: Yu Ishikawa
>
> h2. Overview
> {{Connection.parse_from_uri}} doesn't work for some types like
> {{google_cloud_platform}} whose type name includes under scores. Since
> `urllib.parse.urlparse()` which is used in {{Connection.parse_from_url}}
> doesn't support a schema name which include under scores.
> So, airflow's CLI doesn't work when a given connection URI includes under
> scores like {{google_cloud_platform://X}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AIRFLOW-1330) Connection.parse_from_uri doesn't work for google_cloud_platform and so on
[
https://issues.apache.org/jira/browse/AIRFLOW-1330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130013#comment-16130013
]
Yu Ishikawa commented on AIRFLOW-1330:
--
Good point. I think we should consider backwards compatibility. We can offer
both of the following types to expose options for a connection type, host and
port.
{noformat}
airflow connections --add --conn_id --conn_url mysql://fake-host:3306
airflow connections --add --conn_id --conn_type mysql --conn_host fake-host
--conn_port 3306
{noformat}
> Connection.parse_from_uri doesn't work for google_cloud_platform and so on
> --
>
> Key: AIRFLOW-1330
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1330
> Project: Apache Airflow
> Issue Type: Bug
> Components: cli
>Reporter: Yu Ishikawa
>
> h2. Overview
> {{Connection.parse_from_uri}} doesn't work for some types like
> {{google_cloud_platform}} whose type name includes under scores. Since
> `urllib.parse.urlparse()` which is used in {{Connection.parse_from_url}}
> doesn't support a schema name which include under scores.
> So, airflow's CLI doesn't work when a given connection URI includes under
> scores like {{google_cloud_platform://X}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AIRFLOW-1330) Connection.parse_from_uri doesn't work for google_cloud_platform and so on
[
https://issues.apache.org/jira/browse/AIRFLOW-1330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16057929#comment-16057929
]
Yu Ishikawa commented on AIRFLOW-1330:
--
I think we need to change the command line options for a command to add
connection on our CLI. Because the command requires connection URI whose parser
is implemented on {{urllib.parse.urlparse}}. As I mentioned above, however, the
specification of URL by RFC doesn't allow include under score(s). So we have
two approaches to improve the command to add a connection.
1. Have a kind of dictionary between a connection type and its alias. For
example, when we want to add a connection for google cloud platform, we could
define one like {{'google+cloud+platform' -> 'google_cloud_platform'}}.
2. Change the command line options to add a connection to specify a connection
type without a connection URI like bellow.
{noformat}
airflow connections --add --conn_id google_cloud_default --conn_type
google_cloud_platform ...
{noformat}
The first option could be a little hacky, because we have to add new mapping if
the connection type includes under scores. So, the later option would work.
> Connection.parse_from_uri doesn't work for google_cloud_platform and so on
> --
>
> Key: AIRFLOW-1330
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1330
> Project: Apache Airflow
> Issue Type: Bug
> Components: cli
>Reporter: Yu Ishikawa
>
> h2. Overview
> {{Connection.parse_from_uri}} doesn't work for some types like
> {{google_cloud_platform}} whose type name includes under scores. Since
> `urllib.parse.urlparse()` which is used in {{Connection.parse_from_url}}
> doesn't support a schema name which include under scores.
> So, airflow's CLI doesn't work when a given connection URI includes under
> scores like {{google_cloud_platform://X}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AIRFLOW-1330) Connection.parse_from_uri doesn't work for google_cloud_platform and so on
[
https://issues.apache.org/jira/browse/AIRFLOW-1330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16057857#comment-16057857
]
Yu Ishikawa commented on AIRFLOW-1330:
--
If we want to any connection whose connection URI includes under score ({{_}}),
we can't add it with our CLI. Because the specification of URI's scheme doesn't
allow us to include under score.
{quote}
Scheme names consist of a sequence of characters. The lower case
letters "a"--"z", digits, and the characters plus ("+"), period
("."), and hyphen ("-") are allowed. For resiliency, programs
interpreting URLs should treat upper case letters as equivalent to
lower case in scheme names (e.g., allow "HTTP" as well as "http").
{quote}
Cited by https://www.ietf.org/rfc/rfc1738.txt
> Connection.parse_from_uri doesn't work for google_cloud_platform and so on
> --
>
> Key: AIRFLOW-1330
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1330
> Project: Apache Airflow
> Issue Type: Bug
> Components: cli
>Reporter: Yu Ishikawa
>
> h2. Overview
> {{Connection.parse_from_uri}} doesn't work for some types like
> {{google_cloud_platform}} whose type name includes under scores. Since
> `urllib.parse.urlparse()` which is used in {{Connection.parse_from_url}}
> doesn't support a schema name which include under scores.
> So, airflow's CLI doesn't work when a given connection URI includes under
> scores like {{google_cloud_platform://X}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
