[GitHub] [airflow] milton0825 commented on a change in pull request #5890: [AIRFLOW-5274] dag loading duration metric name too long
milton0825 commented on a change in pull request #5890: [AIRFLOW-5274] dag
loading duration metric name too long
URL: https://github.com/apache/airflow/pull/5890#discussion_r317458205
##
File path: airflow/models/dagbag.py
##
@@ -408,13 +405,9 @@ def collect_dags(
self.dagbag_stats = sorted(
stats, key=lambda x: x.duration, reverse=True)
for file_stat in self.dagbag_stats:
-dag_ids = dags_by_name[file_stat.dags]
-if file_stat.dag_num >= 1:
-# if we found multiple dags per file, the stat is 'dag_id1 _
dag_id2'
-dag_names = '_'.join(dag_ids)
-Stats.timing('dag.loading-duration.{}'.
- format(dag_names),
- file_stat.duration)
+Stats.timing('dag.loading-duration.{}'.
+ format(file_stat.file),
Review comment:
Question: Is `file_stat.file` just the file name or it is the file path that
contains `/`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] feng-tao commented on issue #5890: [AIRFLOW-5274] dag loading duration metric name too long
feng-tao commented on issue #5890: [AIRFLOW-5274] dag loading duration metric name too long URL: https://github.com/apache/airflow/pull/5890#issuecomment-524723110 @kaxil @ashb @milton0825 PR updated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-5309) Use assert_called_once or has_calls in tests
[ https://issues.apache.org/jira/browse/AIRFLOW-5309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915481#comment-16915481 ] ASF subversion and git services commented on AIRFLOW-5309: -- Commit 23d104203e28b60a83172949b38f7c27bf947519 in airflow's branch refs/heads/master from Tomek [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=23d1042 ] [AIRFLOW-5309] Use assert_called_once or has_calls in tests (#5912) Using mock.assert_call_with method can result in flaky tests (ex. iterating through dict in python 3.5 which does not store order of elements). That's why it's better to use assert_called_once_with or has_calls methods. > Use assert_called_once or has_calls in tests > > > Key: AIRFLOW-5309 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5309 > Project: Apache Airflow > Issue Type: Bug > Components: tests >Affects Versions: 1.10.4 >Reporter: Tomasz Urbaszek >Assignee: Tomasz Urbaszek >Priority: Major > > Using mock.assert_call_with method can result in flaky tests (ex. iterating > through dict in python 3.5 which does not store order of elements). That's > why it's better to use assert_called_once_with or has_calls methods. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [airflow] BasPH merged pull request #5912: [AIRFLOW-5309] Use assert_called_once or has_calls in tests
BasPH merged pull request #5912: [AIRFLOW-5309] Use assert_called_once or has_calls in tests URL: https://github.com/apache/airflow/pull/5912 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-5309) Use assert_called_once or has_calls in tests
[ https://issues.apache.org/jira/browse/AIRFLOW-5309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915480#comment-16915480 ] ASF GitHub Bot commented on AIRFLOW-5309: - BasPH commented on pull request #5912: [AIRFLOW-5309] Use assert_called_once or has_calls in tests URL: https://github.com/apache/airflow/pull/5912 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Use assert_called_once or has_calls in tests > > > Key: AIRFLOW-5309 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5309 > Project: Apache Airflow > Issue Type: Bug > Components: tests >Affects Versions: 1.10.4 >Reporter: Tomasz Urbaszek >Assignee: Tomasz Urbaszek >Priority: Major > > Using mock.assert_call_with method can result in flaky tests (ex. iterating > through dict in python 3.5 which does not store order of elements). That's > why it's better to use assert_called_once_with or has_calls methods. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [airflow] Ryan-Miao closed pull request #5888: Set charset utf-8 for webserver log
Ryan-Miao closed pull request #5888: Set charset utf-8 for webserver log URL: https://github.com/apache/airflow/pull/5888 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #5907: [AIRFLOW-5303] Use project_id from GCP credentials
mik-laj commented on a change in pull request #5907: [AIRFLOW-5303] Use
project_id from GCP credentials
URL: https://github.com/apache/airflow/pull/5907#discussion_r317420718
##
File path: airflow/contrib/hooks/gcp_api_base_hook.py
##
@@ -169,7 +179,11 @@ def project_id(self) -> Optional[str]:
:return: id of the project
:rtype: str
"""
-return self._get_field('project')
+overridden_project_id = self._get_field('project')
Review comment:
Updated.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (AIRFLOW-5303) Use project_id from GCP credentials
[ https://issues.apache.org/jira/browse/AIRFLOW-5303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula updated AIRFLOW-5303: --- Summary: Use project_id from GCP credentials (was: Return project_id for other authorization methods) > Use project_id from GCP credentials > --- > > Key: AIRFLOW-5303 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5303 > Project: Apache Airflow > Issue Type: Bug > Components: gcp >Affects Versions: 1.10.4 >Reporter: Kamil Bregula >Assignee: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [airflow] mik-laj commented on issue #5906: [AIRFLOW-5304][depends on AIRFLOW-5306] Fix extra links in BigQueryOperator for task with muliple queriess
mik-laj commented on issue #5906: [AIRFLOW-5304][depends on AIRFLOW-5306] Fix extra links in BigQueryOperator for task with muliple queriess URL: https://github.com/apache/airflow/pull/5906#issuecomment-524674823 I would like GCP operators to better support these feature. I hope that we will be able to find hours in the near future to add support for this. I hope to find hours in the near future to add support for this feature to more operators. I am currently working on one set of operators and It will implement this functionality. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #5906: [AIRFLOW-5304][depends on AIRFLOW-5306] Fix extra links in BigQueryOperator for task with muliple queriess
mik-laj commented on a change in pull request #5906: [AIRFLOW-5304][depends on
AIRFLOW-5306] Fix extra links in BigQueryOperator for task with muliple queriess
URL: https://github.com/apache/airflow/pull/5906#discussion_r317420179
##
File path: airflow/contrib/operators/bigquery_operator.py
##
@@ -45,6 +45,30 @@ def get_link(self, operator, dttm):
job_id=job_id) if job_id else ''
+class BigQueryConsoleIndexableLink(BaseOperatorLink):
+"""
+Helper class for constructing BigQuery link.
+"""
+
+def __init__(self, index) -> None:
+super().__init__()
+self.index = index
+
+@property
+def name(self) -> str:
+return 'BigQuery Console #{index}'.format(index=self.index + 1)
+
+def get_link(self, operator, dttm):
+ti = TaskInstance(task=operator, execution_date=dttm)
+job_ids = ti.xcom_pull(task_ids=operator.task_id, key='job_id')
+if not job_ids:
+return None
+if len(job_ids) < self.index:
+return None
+job_id = job_ids[self.index]
+return
'https://console.cloud.google.com/bigquery?j={job_id}'.format(job_id=job_id)
Review comment:
Updated. :-D
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] milton0825 commented on a change in pull request #5906: [AIRFLOW-5304][depends on AIRFLOW-5306] Fix extra links in BigQueryOperator for task with muliple queriess
milton0825 commented on a change in pull request #5906: [AIRFLOW-5304][depends
on AIRFLOW-5306] Fix extra links in BigQueryOperator for task with muliple
queriess
URL: https://github.com/apache/airflow/pull/5906#discussion_r317419518
##
File path: airflow/contrib/operators/bigquery_operator.py
##
@@ -45,6 +45,30 @@ def get_link(self, operator, dttm):
job_id=job_id) if job_id else ''
+class BigQueryConsoleIndexableLink(BaseOperatorLink):
+"""
+Helper class for constructing BigQuery link.
+"""
+
+def __init__(self, index) -> None:
+super().__init__()
+self.index = index
+
+@property
+def name(self) -> str:
+return 'BigQuery Console #{index}'.format(index=self.index + 1)
+
+def get_link(self, operator, dttm):
+ti = TaskInstance(task=operator, execution_date=dttm)
+job_ids = ti.xcom_pull(task_ids=operator.task_id, key='job_id')
+if not job_ids:
+return None
+if len(job_ids) < self.index:
+return None
+job_id = job_ids[self.index]
+return
'https://console.cloud.google.com/bigquery?j={job_id}'.format(job_id=job_id)
Review comment:
nit: Can we extract this string to a template and reuse it in line 44?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] milton0825 commented on a change in pull request #5906: [AIRFLOW-5304][depends on AIRFLOW-5306] Fix extra links in BigQueryOperator for task with muliple queriess
milton0825 commented on a change in pull request #5906: [AIRFLOW-5304][depends
on AIRFLOW-5306] Fix extra links in BigQueryOperator for task with muliple
queriess
URL: https://github.com/apache/airflow/pull/5906#discussion_r317419518
##
File path: airflow/contrib/operators/bigquery_operator.py
##
@@ -45,6 +45,30 @@ def get_link(self, operator, dttm):
job_id=job_id) if job_id else ''
+class BigQueryConsoleIndexableLink(BaseOperatorLink):
+"""
+Helper class for constructing BigQuery link.
+"""
+
+def __init__(self, index) -> None:
+super().__init__()
+self.index = index
+
+@property
+def name(self) -> str:
+return 'BigQuery Console #{index}'.format(index=self.index + 1)
+
+def get_link(self, operator, dttm):
+ti = TaskInstance(task=operator, execution_date=dttm)
+job_ids = ti.xcom_pull(task_ids=operator.task_id, key='job_id')
+if not job_ids:
+return None
+if len(job_ids) < self.index:
+return None
+job_id = job_ids[self.index]
+return
'https://console.cloud.google.com/bigquery?j={job_id}'.format(job_id=job_id)
Review comment:
Can we extract this string to a template and reuse it in line 44?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-5305) Sort extra links by name
[ https://issues.apache.org/jira/browse/AIRFLOW-5305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915388#comment-16915388 ] ASF subversion and git services commented on AIRFLOW-5305: -- Commit 337cc5f91b0dea4b8abc9c251cae533240585bbf in airflow's branch refs/heads/master from Kamil Breguła [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=337cc5f ] [AIRFLOW-5305] Sort extra links by name (#5905) > Sort extra links by name > > > Key: AIRFLOW-5305 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5305 > Project: Apache Airflow > Issue Type: Improvement > Components: operators >Affects Versions: 1.10.4 >Reporter: Kamil Bregula >Assignee: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (AIRFLOW-5305) Sort extra links by name
[ https://issues.apache.org/jira/browse/AIRFLOW-5305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915387#comment-16915387 ] ASF GitHub Bot commented on AIRFLOW-5305: - milton0825 commented on pull request #5905: [AIRFLOW-5305] Sort extra links by name URL: https://github.com/apache/airflow/pull/5905 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Sort extra links by name > > > Key: AIRFLOW-5305 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5305 > Project: Apache Airflow > Issue Type: Improvement > Components: operators >Affects Versions: 1.10.4 >Reporter: Kamil Bregula >Assignee: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [airflow] milton0825 merged pull request #5905: [AIRFLOW-5305] Sort extra links by name
milton0825 merged pull request #5905: [AIRFLOW-5305] Sort extra links by name URL: https://github.com/apache/airflow/pull/5905 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ToxaZ opened a new pull request #5913: Change SQLSensor to treat NULL as keep poking.
ToxaZ opened a new pull request #5913: Change SQLSensor to treat NULL as keep
poking.
URL: https://github.com/apache/airflow/pull/5913
This PR is targeting Airflow 2.0+ as discussed
[before](https://github.com/apache/airflow/pull/5688).
### Jira
- [x] My PR addresses the following:
- https://issues.apache.org/jira/browse/AIRFLOW-5073
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
- switches NULL response to keep poking instead of 'success'
- removes `allow_true` parameter to decrease params clutter.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
- tests if Sensor returns False on `None` response from DB
- removes testing of `allow_null` parameter
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
### Code Quality
- [X] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (AIRFLOW-5310) Add PrestoToGoogleStorageOperator
lovk korm created AIRFLOW-5310: -- Summary: Add PrestoToGoogleStorageOperator Key: AIRFLOW-5310 URL: https://issues.apache.org/jira/browse/AIRFLOW-5310 Project: Apache Airflow Issue Type: Wish Components: contrib, gcp, operators Affects Versions: 1.10.4 Reporter: lovk korm Please add PrestoToGoogleStorageOperator -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (AIRFLOW-5154) Add ability to generate graph alerts to email
[ https://issues.apache.org/jira/browse/AIRFLOW-5154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915295#comment-16915295 ] lovk korm commented on AIRFLOW-5154: Such docs would be great > Add ability to generate graph alerts to email > - > > Key: AIRFLOW-5154 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5154 > Project: Apache Airflow > Issue Type: Wish > Components: contrib >Affects Versions: 1.10.4 >Reporter: lovk korm >Priority: Major > > I'm not sure how this is doable but one of the key components that is missing > in airflow is the ability to notify about detecting anomalies something like > graphana [https://grafana.com/] > It would be great if airflow can add support for such tools > > I'm talking here about +*airflow itself*+. For example: if DAG run normally > takes 5 minutes but now for any reason it's running over 30 minutes than we > want an alert to be sent with graph that shows that anomaly. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (AIRFLOW-4873) Improve BROKEN DAG error message by specifying dag_id
[
https://issues.apache.org/jira/browse/AIRFLOW-4873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915291#comment-16915291
]
jack commented on AIRFLOW-4873:
---
[~ash] No. there is no link between dag_id and file names.
I'm not sure I understand what you mean both dag_id and file name are static
data. they don't change nor calculated on run time. So why there isn't dag_id?
This can be taken from dag_bag isn't it? it contains a mapping of files to the
dags?
> Improve BROKEN DAG error message by specifying dag_id
> --
>
> Key: AIRFLOW-4873
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4873
> Project: Apache Airflow
> Issue Type: Improvement
> Components: ui
>Affects Versions: 1.10.3
>Reporter: jack
>Assignee: Omkar
>Priority: Major
> Labels: easy-fix
>
> In some cases there is a broken DAG message appear in the UI:
>
> {code:java}
> Broken DAG: [/home/airflow/dags/my_dag.py] No module named 'HelperClass'
> {code}
>
> The problem here is that the error indicates only the file name but it
> doesn't specify the dag_id.
> The error could be improved to (or similar):
>
> {code:java}
> Broken DAG: DAG_ID : [/home/airflow/dags/my_dag.py] No module named
> 'HelperClass' {code}
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (AIRFLOW-5309) Use assert_called_once or has_calls in tests
[
https://issues.apache.org/jira/browse/AIRFLOW-5309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915286#comment-16915286
]
ASF GitHub Bot commented on AIRFLOW-5309:
-
nuclearpinguin commented on pull request #5912: [AIRFLOW-5309] Use
assert_called_once or has_calls in tests
URL: https://github.com/apache/airflow/pull/5912
Using mock.assert_call_with method can result in flaky tests (ex. iterating
through dict in python 3.5 which does not store order of elements). That's why
it's better to use assert_called_once_with or has_calls methods.
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title.
- https://issues.apache.org/jira/browse/AIRFLOW-5309
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
### Code Quality
- [ ] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Use assert_called_once or has_calls in tests
>
>
> Key: AIRFLOW-5309
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5309
> Project: Apache Airflow
> Issue Type: Bug
> Components: tests
>Affects Versions: 1.10.4
>Reporter: Tomasz Urbaszek
>Assignee: Tomasz Urbaszek
>Priority: Major
>
> Using mock.assert_call_with method can result in flaky tests (ex. iterating
> through dict in python 3.5 which does not store order of elements). That's
> why it's better to use assert_called_once_with or has_calls methods.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[GitHub] [airflow] nuclearpinguin opened a new pull request #5912: [AIRFLOW-5309] Use assert_called_once or has_calls in tests
nuclearpinguin opened a new pull request #5912: [AIRFLOW-5309] Use
assert_called_once or has_calls in tests
URL: https://github.com/apache/airflow/pull/5912
Using mock.assert_call_with method can result in flaky tests (ex. iterating
through dict in python 3.5 which does not store order of elements). That's why
it's better to use assert_called_once_with or has_calls methods.
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title.
- https://issues.apache.org/jira/browse/AIRFLOW-5309
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
### Code Quality
- [ ] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] haoliang7 commented on issue #5495: [AIRFLOW-4858] Deprecate "Historical convenience functions" in conf
haoliang7 commented on issue #5495: [AIRFLOW-4858] Deprecate "Historical convenience functions" in conf URL: https://github.com/apache/airflow/pull/5495#issuecomment-524637187 > The change was a little bit bigger than needed (`configuration.conf.getboolean` was fine) but making it all consistent is no bad thing. > > One small comment (on the warnings filter) then this looks good to me, thanks. > > (Sorry, it's conflicted again, I didn't see the ping because my inbox is currently 1k deep in unread airflow messages ) @ashb The conflicts have been resolved. The warnings filter was a mistake that has been corrected. Please kindly review again. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-5309) Use assert_called_once or has_calls in tests
Tomasz Urbaszek created AIRFLOW-5309: Summary: Use assert_called_once or has_calls in tests Key: AIRFLOW-5309 URL: https://issues.apache.org/jira/browse/AIRFLOW-5309 Project: Apache Airflow Issue Type: Bug Components: tests Affects Versions: 1.10.4 Reporter: Tomasz Urbaszek Assignee: Tomasz Urbaszek Using mock.assert_call_with method can result in flaky tests (ex. iterating through dict in python 3.5 which does not store order of elements). That's why it's better to use assert_called_once_with or has_calls methods. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [airflow] serkef commented on issue #5557: [AIRFLOW-4928] Move all config parses to class properties
serkef commented on issue #5557: [AIRFLOW-4928] Move all config parses to class properties URL: https://github.com/apache/airflow/pull/5557#issuecomment-524633452 @ashb Fixed your comments and rebased. Please take another look. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] serkef commented on a change in pull request #5557: [AIRFLOW-4928] Move all config parses to class properties
serkef commented on a change in pull request #5557: [AIRFLOW-4928] Move all
config parses to class properties
URL: https://github.com/apache/airflow/pull/5557#discussion_r317399593
##
File path: airflow/models/dagbag.py
##
@@ -194,7 +200,7 @@ def process_file(self, filepath, only_if_updated=True,
safe_mode=True):
if mod_name in sys.modules:
del sys.modules[mod_name]
-with timeout(configuration.conf.getint('core',
"DAGBAG_IMPORT_TIMEOUT")):
+with timeout(self.dagbag_import_timeout):
Review comment:
good point
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #5907: [AIRFLOW-5303] Return project_id for other authorization methods
mik-laj commented on a change in pull request #5907: [AIRFLOW-5303] Return
project_id for other authorization methods
URL: https://github.com/apache/airflow/pull/5907#discussion_r317391600
##
File path: airflow/contrib/hooks/gcp_api_base_hook.py
##
@@ -169,7 +179,11 @@ def project_id(self) -> Optional[str]:
:return: id of the project
:rtype: str
"""
-return self._get_field('project')
+overridden_project_id = self._get_field('project')
Review comment:
I wonder if this piece of code should not be transferred to the
_get_credentials_and_project_id method. Then this method will never return an
invalid value and it will be easier to use it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-5308) Pass credentials object to pandas_gbq
[
https://issues.apache.org/jira/browse/AIRFLOW-5308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915213#comment-16915213
]
ASF GitHub Bot commented on AIRFLOW-5308:
-
mik-laj commented on pull request #5911:
[AIRFLOW-5308][depedens-on-AIRFLOW-5303] Pass credentials object to pandas_gbq
URL: https://github.com/apache/airflow/pull/5911
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
### Code Quality
- [ ] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Pass credentials object to pandas_gbq
> -
>
> Key: AIRFLOW-5308
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5308
> Project: Apache Airflow
> Issue Type: Bug
> Components: gcp
>Affects Versions: 1.10.4
>Reporter: Kamil Bregula
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[GitHub] [airflow] mik-laj opened a new pull request #5911: [AIRFLOW-5308][depedens-on-AIRFLOW-5303] Pass credentials object to pandas_gbq
mik-laj opened a new pull request #5911:
[AIRFLOW-5308][depedens-on-AIRFLOW-5303] Pass credentials object to pandas_gbq
URL: https://github.com/apache/airflow/pull/5911
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
### Code Quality
- [ ] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (AIRFLOW-5308) Pass credentials object to pandas_gbq
Kamil Bregula created AIRFLOW-5308: -- Summary: Pass credentials object to pandas_gbq Key: AIRFLOW-5308 URL: https://issues.apache.org/jira/browse/AIRFLOW-5308 Project: Apache Airflow Issue Type: Bug Components: gcp Affects Versions: 1.10.4 Reporter: Kamil Bregula -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Assigned] (AIRFLOW-5303) Return project_id for other authorization methods
[ https://issues.apache.org/jira/browse/AIRFLOW-5303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula reassigned AIRFLOW-5303: -- Assignee: Kamil Bregula > Return project_id for other authorization methods > - > > Key: AIRFLOW-5303 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5303 > Project: Apache Airflow > Issue Type: Bug > Components: gcp >Affects Versions: 1.10.4 >Reporter: Kamil Bregula >Assignee: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Assigned] (AIRFLOW-5304) Fix extra links in BigQueryOperator for task with multiple queries
[ https://issues.apache.org/jira/browse/AIRFLOW-5304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula reassigned AIRFLOW-5304: -- Assignee: Kamil Bregula > Fix extra links in BigQueryOperator for task with multiple queries > -- > > Key: AIRFLOW-5304 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5304 > Project: Apache Airflow > Issue Type: Bug > Components: gcp >Affects Versions: 1.10.4 >Reporter: Kamil Bregula >Assignee: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Assigned] (AIRFLOW-5305) Sort extra links by name
[ https://issues.apache.org/jira/browse/AIRFLOW-5305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula reassigned AIRFLOW-5305: -- Assignee: Kamil Bregula > Sort extra links by name > > > Key: AIRFLOW-5305 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5305 > Project: Apache Airflow > Issue Type: Improvement > Components: operators >Affects Versions: 1.10.4 >Reporter: Kamil Bregula >Assignee: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Assigned] (AIRFLOW-5306) Fix the display of links when they contain special characters
[ https://issues.apache.org/jira/browse/AIRFLOW-5306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula reassigned AIRFLOW-5306: -- Assignee: Kamil Bregula > Fix the display of links when they contain special characters > - > > Key: AIRFLOW-5306 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5306 > Project: Apache Airflow > Issue Type: Bug > Components: operators >Affects Versions: 1.10.4 >Reporter: Kamil Bregula >Assignee: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (AIRFLOW-5307) Move the BaseOperator to the operators package
[
https://issues.apache.org/jira/browse/AIRFLOW-5307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915182#comment-16915182
]
ASF GitHub Bot commented on AIRFLOW-5307:
-
BasPH commented on pull request #5910: [WIP][AIRFLOW-5307] Move BaseOperator to
airflow.operators.base_operator
URL: https://github.com/apache/airflow/pull/5910
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-5307
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
The BaseOperator currently resides in /airflow/models but has never been a
database model, i.e. it is not stored in a database table. I suggest to move it
to its logical place, the operators package.
To preserve backwards compatibility I import the BaseOperator in
/airflow/models/\_\_init\_\_.py and raise a DeprecationWarning when imported
from there. All references to airflow.models.BaseOperator have been removed and
I suggest to removed backward compatibility in Airflow 2.0.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
No test logic altered, only moved package names.
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
### Code Quality
- [x] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Move the BaseOperator to the operators package
> --
>
> Key: AIRFLOW-5307
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5307
> Project: Apache Airflow
> Issue Type: Improvement
> Components: core
>Affects Versions: 2.0.0
>Reporter: Bas Harenslak
>Priority: Major
>
> The BaseOperator currently resides in /airflow/models but has never been a
> database model, i.e. it is not stored in a database table. I suggest to move
> it to its logical place, the operators package.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[GitHub] [airflow] BasPH opened a new pull request #5910: [WIP][AIRFLOW-5307] Move BaseOperator to airflow.operators.base_operator
BasPH opened a new pull request #5910: [WIP][AIRFLOW-5307] Move BaseOperator to
airflow.operators.base_operator
URL: https://github.com/apache/airflow/pull/5910
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-5307
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
The BaseOperator currently resides in /airflow/models but has never been a
database model, i.e. it is not stored in a database table. I suggest to move it
to its logical place, the operators package.
To preserve backwards compatibility I import the BaseOperator in
/airflow/models/\_\_init\_\_.py and raise a DeprecationWarning when imported
from there. All references to airflow.models.BaseOperator have been removed and
I suggest to removed backward compatibility in Airflow 2.0.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
No test logic altered, only moved package names.
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
### Code Quality
- [x] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (AIRFLOW-5307) Move the BaseOperator to the operators package
[ https://issues.apache.org/jira/browse/AIRFLOW-5307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bas Harenslak updated AIRFLOW-5307: --- Description: The BaseOperator currently resides in /airflow/models but has never been a database model, i.e. it is not stored in a database table. I suggest to move it to its logical place, the operators package. > Move the BaseOperator to the operators package > -- > > Key: AIRFLOW-5307 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5307 > Project: Apache Airflow > Issue Type: Improvement > Components: core >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Priority: Major > > The BaseOperator currently resides in /airflow/models but has never been a > database model, i.e. it is not stored in a database table. I suggest to move > it to its logical place, the operators package. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (AIRFLOW-5307) Move the BaseOperator to the operators package
Bas Harenslak created AIRFLOW-5307: -- Summary: Move the BaseOperator to the operators package Key: AIRFLOW-5307 URL: https://issues.apache.org/jira/browse/AIRFLOW-5307 Project: Apache Airflow Issue Type: Improvement Components: core Affects Versions: 2.0.0 Reporter: Bas Harenslak -- This message was sent by Atlassian Jira (v8.3.2#803003)
