[jira] [Commented] (AIRFLOW-4226) DataProcPySparkOperator gets project ID from gcp_conn_id rather than project_id
[ https://issues.apache.org/jira/browse/AIRFLOW-4226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16982900#comment-16982900 ] Kaxil Naik commented on AIRFLOW-4226: - Yes in 1.10.6 it is still the case. > DataProcPySparkOperator gets project ID from gcp_conn_id rather than > project_id > --- > > Key: AIRFLOW-4226 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4226 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 1.10.3, 1.10.4, 1.10.5 >Reporter: Aaron Liblong >Assignee: Kaxil Naik >Priority: Minor > Fix For: 1.10.7 > > > DataProcPySparkOperator gets [the project ID for the cluster it > creates|https://github.com/apache/airflow/blob/master/airflow/contrib/operators/dataproc_operator.py#L1334] > from [the hook instantiated from > gcp_conn_id|https://github.com/apache/airflow/blob/master/airflow/contrib/operators/dataproc_operator.py#L1324] > rather than from its own project_id property. This is contrary to how every > other operator in this module works. Not sure if it's a bug or intended. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4226) DataProcPySparkOperator gets project ID from gcp_conn_id rather than project_id
[ https://issues.apache.org/jira/browse/AIRFLOW-4226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16982895#comment-16982895 ] Kamil Bregula commented on AIRFLOW-4226: [~kaxilnaik] Is this ticket still valid? > DataProcPySparkOperator gets project ID from gcp_conn_id rather than > project_id > --- > > Key: AIRFLOW-4226 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4226 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 1.10.3, 1.10.4, 1.10.5 >Reporter: Aaron Liblong >Assignee: Kaxil Naik >Priority: Minor > Fix For: 1.10.7 > > > DataProcPySparkOperator gets [the project ID for the cluster it > creates|https://github.com/apache/airflow/blob/master/airflow/contrib/operators/dataproc_operator.py#L1334] > from [the hook instantiated from > gcp_conn_id|https://github.com/apache/airflow/blob/master/airflow/contrib/operators/dataproc_operator.py#L1324] > rather than from its own project_id property. This is contrary to how every > other operator in this module works. Not sure if it's a bug or intended. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4226) DataProcPySparkOperator gets project ID from gcp_conn_id rather than project_id
[ https://issues.apache.org/jira/browse/AIRFLOW-4226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937970#comment-16937970 ] Kaxil Naik commented on AIRFLOW-4226: - Thanks [~kamil.bregula] > DataProcPySparkOperator gets project ID from gcp_conn_id rather than > project_id > --- > > Key: AIRFLOW-4226 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4226 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 1.10.3, 1.10.4, 1.10.5 >Reporter: Aaron Liblong >Assignee: Kaxil Naik >Priority: Minor > Fix For: 1.10.6 > > > DataProcPySparkOperator gets [the project ID for the cluster it > creates|https://github.com/apache/airflow/blob/master/airflow/contrib/operators/dataproc_operator.py#L1334] > from [the hook instantiated from > gcp_conn_id|https://github.com/apache/airflow/blob/master/airflow/contrib/operators/dataproc_operator.py#L1324] > rather than from its own project_id property. This is contrary to how every > other operator in this module works. Not sure if it's a bug or intended. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4226) DataProcPySparkOperator gets project ID from gcp_conn_id rather than project_id
[ https://issues.apache.org/jira/browse/AIRFLOW-4226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937683#comment-16937683 ] Kaxil Naik commented on AIRFLOW-4226: - Thanks [~jackjack10]. [~liblonga] I agree this should change and made consistent, allowing users to override project on task/operator level. I will assign this to myself and try to fix this soon. > DataProcPySparkOperator gets project ID from gcp_conn_id rather than > project_id > --- > > Key: AIRFLOW-4226 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4226 > Project: Apache Airflow > Issue Type: Bug > Components: gcp >Reporter: Aaron Liblong >Priority: Minor > > DataProcPySparkOperator gets [the project ID for the cluster it > creates|https://github.com/apache/airflow/blob/master/airflow/contrib/operators/dataproc_operator.py#L1334] > from [the hook instantiated from > gcp_conn_id|https://github.com/apache/airflow/blob/master/airflow/contrib/operators/dataproc_operator.py#L1324] > rather than from its own project_id property. This is contrary to how every > other operator in this module works. Not sure if it's a bug or intended. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4226) DataProcPySparkOperator gets project ID from gcp_conn_id rather than project_id
[ https://issues.apache.org/jira/browse/AIRFLOW-4226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937597#comment-16937597 ] jack commented on AIRFLOW-4226: --- [~kaxilnaik] ping :) > DataProcPySparkOperator gets project ID from gcp_conn_id rather than > project_id > --- > > Key: AIRFLOW-4226 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4226 > Project: Apache Airflow > Issue Type: Bug > Components: gcp >Reporter: Aaron Liblong >Priority: Minor > > DataProcPySparkOperator gets [the project ID for the cluster it > creates|https://github.com/apache/airflow/blob/master/airflow/contrib/operators/dataproc_operator.py#L1334] > from [the hook instantiated from > gcp_conn_id|https://github.com/apache/airflow/blob/master/airflow/contrib/operators/dataproc_operator.py#L1324] > rather than from its own project_id property. This is contrary to how every > other operator in this module works. Not sure if it's a bug or intended. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4226) DataProcPySparkOperator gets project ID from gcp_conn_id rather than project_id
[
https://issues.apache.org/jira/browse/AIRFLOW-4226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16895476#comment-16895476
]
Aaron Liblong commented on AIRFLOW-4226:
It was more of a comment on a perceived inconsistency in the API. It's been a
while since I wrote the issue, and looking back through the source code, it
looks like I may have gotten some of the details wrong, so allow me to rephrase:
Is there a particular reason {{DataprocOperationBaseOperator}} takes a
{{project_id}} param and {{DataProcJobBaseOperator}} does not? Both take a
{{gcp_conn_id}}, but only the latter uses it to get a {{project_id}}.
> DataProcPySparkOperator gets project ID from gcp_conn_id rather than
> project_id
> ---
>
> Key: AIRFLOW-4226
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4226
> Project: Apache Airflow
> Issue Type: Bug
> Components: gcp
>Reporter: Aaron Liblong
>Priority: Minor
>
> DataProcPySparkOperator gets [the project ID for the cluster it
> creates|https://github.com/apache/airflow/blob/master/airflow/contrib/operators/dataproc_operator.py#L1334]
> from [the hook instantiated from
> gcp_conn_id|https://github.com/apache/airflow/blob/master/airflow/contrib/operators/dataproc_operator.py#L1324]
> rather than from its own project_id property. This is contrary to how every
> other operator in this module works. Not sure if it's a bug or intended.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (AIRFLOW-4226) DataProcPySparkOperator gets project ID from gcp_conn_id rather than project_id
[
https://issues.apache.org/jira/browse/AIRFLOW-4226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16895080#comment-16895080
]
Kaxil Naik commented on AIRFLOW-4226:
-
[~liblonga] All the DataProc{*}SparkOperator uses the project_id from the hook.
Is the use-case to override it?
> DataProcPySparkOperator gets project ID from gcp_conn_id rather than
> project_id
> ---
>
> Key: AIRFLOW-4226
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4226
> Project: Apache Airflow
> Issue Type: Bug
> Components: gcp
>Reporter: Aaron Liblong
>Priority: Minor
>
> DataProcPySparkOperator gets [the project ID for the cluster it
> creates|https://github.com/apache/airflow/blob/master/airflow/contrib/operators/dataproc_operator.py#L1334]
> from [the hook instantiated from
> gcp_conn_id|https://github.com/apache/airflow/blob/master/airflow/contrib/operators/dataproc_operator.py#L1324]
> rather than from its own project_id property. This is contrary to how every
> other operator in this module works. Not sure if it's a bug or intended.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (AIRFLOW-4226) DataProcPySparkOperator gets project ID from gcp_conn_id rather than project_id
[ https://issues.apache.org/jira/browse/AIRFLOW-4226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16876397#comment-16876397 ] Kamil Bregula commented on AIRFLOW-4226: [~jackjack10] Thanks. You're awesome! > DataProcPySparkOperator gets project ID from gcp_conn_id rather than > project_id > --- > > Key: AIRFLOW-4226 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4226 > Project: Apache Airflow > Issue Type: Bug > Components: gcp >Reporter: Aaron Liblong >Priority: Minor > > DataProcPySparkOperator gets [the project ID for the cluster it > creates|https://github.com/apache/airflow/blob/master/airflow/contrib/operators/dataproc_operator.py#L1334] > from [the hook instantiated from > gcp_conn_id|https://github.com/apache/airflow/blob/master/airflow/contrib/operators/dataproc_operator.py#L1324] > rather than from its own project_id property. This is contrary to how every > other operator in this module works. Not sure if it's a bug or intended. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4226) DataProcPySparkOperator gets project ID from gcp_conn_id rather than project_id
[ https://issues.apache.org/jira/browse/AIRFLOW-4226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16876239#comment-16876239 ] jack commented on AIRFLOW-4226: --- [~kamil.bregula] [~kaxilnaik] do you know if this is intentional? also need to add gcp component to the ticket > DataProcPySparkOperator gets project ID from gcp_conn_id rather than > project_id > --- > > Key: AIRFLOW-4226 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4226 > Project: Apache Airflow > Issue Type: Bug >Reporter: Aaron Liblong >Priority: Minor > > DataProcPySparkOperator gets [the project ID for the cluster it > creates|https://github.com/apache/airflow/blob/master/airflow/contrib/operators/dataproc_operator.py#L1334] > from [the hook instantiated from > gcp_conn_id|https://github.com/apache/airflow/blob/master/airflow/contrib/operators/dataproc_operator.py#L1324] > rather than from its own project_id property. This is contrary to how every > other operator in this module works. Not sure if it's a bug or intended. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
