[GitHub] [airflow] ashb opened a new pull request #7534: [AIRFLOW-6867] Fix bug in kill zombies
ashb opened a new pull request #7534: [AIRFLOW-6867] Fix bug in kill zombies URL: https://github.com/apache/airflow/pull/7534 The refactor in #7488 introduced a scheduler-breaking bug. I am not happy about merging this without a unit test to stop it breaking again, but given how much process_file does it's hard to test. Thoughts? --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [ ] Description above provides context of the change - [ ] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [ ] Unit tests coverage for changes (not needed for documentation changes) - [ ] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [ ] Relevant documentation is updated including usage instructions. - [ ] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7527: [AIRFLOW-6907] Simplify SchedulerJob
codecov-io edited a comment on issue #7527: [AIRFLOW-6907] Simplify SchedulerJob URL: https://github.com/apache/airflow/pull/7527#issuecomment-590717059 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7527?src=pr&el=h1) Report > :exclamation: No coverage uploaded for pull request base (`master@d1a3424`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit). > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/7527?src=pr&el=tree) ```diff @@Coverage Diff@@ ## master#7527 +/- ## = Coverage ? 86.57% = Files ? 896 Lines ?42634 Branches ?0 = Hits ?36910 Misses? 5724 Partials ?0 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7527?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7527/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `90.28% <100%> (ø)` | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7527?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7527?src=pr&el=footer). Last update [d1a3424...37c630d](https://codecov.io/gh/apache/airflow/pull/7527?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] yustoris commented on issue #6337: [AIRFLOW-5659] - Add support for ephemeral storage on KubernetesPodOp…
yustoris commented on issue #6337: [AIRFLOW-5659] - Add support for ephemeral storage on KubernetesPodOp… URL: https://github.com/apache/airflow/pull/6337#issuecomment-590856588 Is any progress? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] stale[bot] commented on issue #6375: [AIRFLOW-5706] Add KubernetesPodYamlOperator
stale[bot] commented on issue #6375: [AIRFLOW-5706] Add KubernetesPodYamlOperator URL: https://github.com/apache/airflow/pull/6375#issuecomment-590854881 This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #7527: [AIRFLOW-6907] Simplify SchedulerJob
ashb commented on a change in pull request #7527: [AIRFLOW-6907] Simplify SchedulerJob URL: https://github.com/apache/airflow/pull/7527#discussion_r383838670 ## File path: airflow/jobs/scheduler_job.py ## @@ -1098,37 +1094,20 @@ def _find_executable_task_instances(self, simple_dag_bag, states, session=None): TI = models.TaskInstance DR = models.DagRun DM = models.DagModel -ti_query = ( +task_instances_to_examine = ( session .query(TI) .filter(TI.dag_id.in_(simple_dag_bag.dag_ids)) .outerjoin( -DR, -and_(DR.dag_id == TI.dag_id, DR.execution_date == TI.execution_date) +DR, and_(DR.dag_id == TI.dag_id, DR.execution_date == TI.execution_date) ) -.filter(or_(DR.run_id == None, # noqa: E711 pylint: disable=singleton-comparison -not_(DR.run_id.like(BackfillJob.ID_PREFIX + '%' +.filter(or_(DR.run_id.is_(None), not_(DR.run_id.like(BackfillJob.ID_PREFIX + '%' .outerjoin(DM, DM.dag_id == TI.dag_id) -.filter(or_(DM.dag_id == None, # noqa: E711 pylint: disable=singleton-comparison -not_(DM.is_paused))) +.filter(or_(DM.dag_id.is_(None), not_(DM.is_paused))) Review comment: Yeah I'm okay with this. My question is why in #7484 does he have ``` models.DagRun.state.is_(None))) # pylint: disable=no-member ``` but we don't need that here? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on issue #7484: [AIRFLOW-6864] Make airflow/jobs pylint compatible
ashb commented on issue #7484: [AIRFLOW-6864] Make airflow/jobs pylint compatible URL: https://github.com/apache/airflow/pull/7484#issuecomment-590834990 @nuclearpinguin See https://github.com/apache/airflow/pull/7527/files#r383812108 -- this is my only outstanding query about this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #7527: [AIRFLOW-6907] Simplify SchedulerJob
mik-laj commented on a change in pull request #7527: [AIRFLOW-6907] Simplify SchedulerJob URL: https://github.com/apache/airflow/pull/7527#discussion_r383830531 ## File path: airflow/providers/google/cloud/hooks/stackdriver.py ## @@ -29,7 +29,7 @@ from google.protobuf.json_format import MessageToDict, MessageToJson, Parse from googleapiclient.errors import HttpError -from airflow import AirflowException +from airflow.exceptions import AirflowException Review comment: This change was already in the master, but the PR preview has not been updated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-6913) CLONE - Allow filtering by all columns in Browse Logs view
Ebrima Jallow created AIRFLOW-6913: -- Summary: CLONE - Allow filtering by all columns in Browse Logs view Key: AIRFLOW-6913 URL: https://issues.apache.org/jira/browse/AIRFLOW-6913 Project: Apache Airflow Issue Type: Improvement Components: logging, ui Affects Versions: 1.10.2 Reporter: Ebrima Jallow The "Browse Logs" UI currently allows filtering by "DAG ID", "Task ID", "Execution Date", and "Extra". !Screenshot from 2019-03-13 11-41-20.png! For consistency and flexibility, it would be good to allow filtering by any of the available columns, specifically "Datetime", "Event", "Execution Date", and "Owner". !Screenshot from 2019-03-13 11-44-26.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] mik-laj commented on a change in pull request #7527: [AIRFLOW-6907] Simplify SchedulerJob
mik-laj commented on a change in pull request #7527: [AIRFLOW-6907] Simplify SchedulerJob URL: https://github.com/apache/airflow/pull/7527#discussion_r383821345 ## File path: airflow/jobs/scheduler_job.py ## @@ -1098,37 +1094,20 @@ def _find_executable_task_instances(self, simple_dag_bag, states, session=None): TI = models.TaskInstance DR = models.DagRun DM = models.DagModel -ti_query = ( +task_instances_to_examine = ( session .query(TI) .filter(TI.dag_id.in_(simple_dag_bag.dag_ids)) .outerjoin( -DR, -and_(DR.dag_id == TI.dag_id, DR.execution_date == TI.execution_date) +DR, and_(DR.dag_id == TI.dag_id, DR.execution_date == TI.execution_date) ) -.filter(or_(DR.run_id == None, # noqa: E711 pylint: disable=singleton-comparison -not_(DR.run_id.like(BackfillJob.ID_PREFIX + '%' +.filter(or_(DR.run_id.is_(None), not_(DR.run_id.like(BackfillJob.ID_PREFIX + '%' .outerjoin(DM, DM.dag_id == TI.dag_id) -.filter(or_(DM.dag_id == None, # noqa: E711 pylint: disable=singleton-comparison -not_(DM.is_paused))) +.filter(or_(DM.dag_id.is_(None), not_(DM.is_paused))) Review comment: `DM.dag_id == None` => disable=singleton-comparison `DM.dag_id.is_(None)` is correct syntax. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #7527: [AIRFLOW-6907] Simplify SchedulerJob
mik-laj commented on a change in pull request #7527: [AIRFLOW-6907] Simplify SchedulerJob URL: https://github.com/apache/airflow/pull/7527#discussion_r383821345 ## File path: airflow/jobs/scheduler_job.py ## @@ -1098,37 +1094,20 @@ def _find_executable_task_instances(self, simple_dag_bag, states, session=None): TI = models.TaskInstance DR = models.DagRun DM = models.DagModel -ti_query = ( +task_instances_to_examine = ( session .query(TI) .filter(TI.dag_id.in_(simple_dag_bag.dag_ids)) .outerjoin( -DR, -and_(DR.dag_id == TI.dag_id, DR.execution_date == TI.execution_date) +DR, and_(DR.dag_id == TI.dag_id, DR.execution_date == TI.execution_date) ) -.filter(or_(DR.run_id == None, # noqa: E711 pylint: disable=singleton-comparison -not_(DR.run_id.like(BackfillJob.ID_PREFIX + '%' +.filter(or_(DR.run_id.is_(None), not_(DR.run_id.like(BackfillJob.ID_PREFIX + '%' .outerjoin(DM, DM.dag_id == TI.dag_id) -.filter(or_(DM.dag_id == None, # noqa: E711 pylint: disable=singleton-comparison -not_(DM.is_paused))) +.filter(or_(DM.dag_id.is_(None), not_(DM.is_paused))) Review comment: `DM.dag_id == None` => disable=singleton-comparison 'DM.dag_id.is_(None)` is correct syntax. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #7489: [AIRFLOW-6869][WIP] Bulk fetch DAGRuns for _process_task_instances
mik-laj commented on a change in pull request #7489: [AIRFLOW-6869][WIP] Bulk fetch DAGRuns for _process_task_instances URL: https://github.com/apache/airflow/pull/7489#discussion_r383814846 ## File path: airflow/jobs/scheduler_job.py ## @@ -631,16 +632,16 @@ def create_dag_run(self, dag, session=None): return next_run @provide_session -def _process_task_instances(self, dag, task_instances_list, session=None): +def _process_task_instances(self, dag, dag_runs, session=None): Review comment: In this PR, I want to focus only on necessary changes. I want to add type-hint in separate PR, because type-hint in the core are problematic. They cause cyclical imports. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #7489: [AIRFLOW-6869][WIP] Bulk fetch DAGRuns for _process_task_instances
mik-laj commented on a change in pull request #7489: [AIRFLOW-6869][WIP] Bulk fetch DAGRuns for _process_task_instances URL: https://github.com/apache/airflow/pull/7489#discussion_r383814563 ## File path: airflow/models/dagrun.py ## @@ -125,14 +125,16 @@ def refresh_from_db(self, session=None): @staticmethod @provide_session -def find(dag_id=None, run_id=None, execution_date=None, +def find(dag_id=None, dag_ids=None, run_id=None, execution_date=None, Review comment: In this PR, I want to focus only on necessary changes. I want to add type-hint in separate PR, but type-hint in core are problematic. They cause cyclical imports. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #7527: [AIRFLOW-6907] Simplify SchedulerJob
ashb commented on a change in pull request #7527: [AIRFLOW-6907] Simplify SchedulerJob URL: https://github.com/apache/airflow/pull/7527#discussion_r383813867 ## File path: airflow/providers/google/cloud/hooks/stackdriver.py ## @@ -29,7 +29,7 @@ from google.protobuf.json_format import MessageToDict, MessageToJson, Parse from googleapiclient.errors import HttpError -from airflow import AirflowException +from airflow.exceptions import AirflowException Review comment: Other than this, :+1: to the change. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #7481: [AIRFLOW-6862][WIP] Do not check the freshness of fresh DAG
mik-laj commented on a change in pull request #7481: [AIRFLOW-6862][WIP] Do not check the freshness of fresh DAG URL: https://github.com/apache/airflow/pull/7481#discussion_r383813761 ## File path: tests/jobs/test_scheduler_job.py ## @@ -1010,7 +1010,7 @@ def test_process_dags_not_create_dagrun_for_subdags(self): scheduler._process_task_instances = mock.MagicMock() scheduler.manage_slas = mock.MagicMock() -scheduler._process_dags(self.dagbag, [dag] + dag.subdags, []) +scheduler._process_dags([dag] + dag.subdags, []) Review comment: In another PR I deletes the tis_out parameter completely. This parameter is also required because this method returns values using this list. Unfortunately, we do not have an assertion in the tests. In real code is always required. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #7527: [AIRFLOW-6907] Simplify SchedulerJob
ashb commented on a change in pull request #7527: [AIRFLOW-6907] Simplify SchedulerJob URL: https://github.com/apache/airflow/pull/7527#discussion_r383813377 ## File path: airflow/providers/google/cloud/hooks/stackdriver.py ## @@ -29,7 +29,7 @@ from google.protobuf.json_format import MessageToDict, MessageToJson, Parse from googleapiclient.errors import HttpError -from airflow import AirflowException +from airflow.exceptions import AirflowException Review comment: Unrelated change here? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #7527: [AIRFLOW-6907] Simplify SchedulerJob
ashb commented on a change in pull request #7527: [AIRFLOW-6907] Simplify SchedulerJob URL: https://github.com/apache/airflow/pull/7527#discussion_r383812108 ## File path: airflow/jobs/scheduler_job.py ## @@ -1098,37 +1094,20 @@ def _find_executable_task_instances(self, simple_dag_bag, states, session=None): TI = models.TaskInstance DR = models.DagRun DM = models.DagModel -ti_query = ( +task_instances_to_examine = ( session .query(TI) .filter(TI.dag_id.in_(simple_dag_bag.dag_ids)) .outerjoin( -DR, -and_(DR.dag_id == TI.dag_id, DR.execution_date == TI.execution_date) +DR, and_(DR.dag_id == TI.dag_id, DR.execution_date == TI.execution_date) ) -.filter(or_(DR.run_id == None, # noqa: E711 pylint: disable=singleton-comparison -not_(DR.run_id.like(BackfillJob.ID_PREFIX + '%' +.filter(or_(DR.run_id.is_(None), not_(DR.run_id.like(BackfillJob.ID_PREFIX + '%' .outerjoin(DM, DM.dag_id == TI.dag_id) -.filter(or_(DM.dag_id == None, # noqa: E711 pylint: disable=singleton-comparison -not_(DM.is_paused))) +.filter(or_(DM.dag_id.is_(None), not_(DM.is_paused))) Review comment: @nuclearpinguin This was an example of where we don't need a noqa anymore -- do you know why you did on your pylint fixes to jobs? Is it because we are ignoring this whole file? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on issue #7533: [AIRFLOW-5485] - scheduler_job: replace == None with is
mik-laj commented on issue #7533: [AIRFLOW-5485] - scheduler_job: replace == None with is URL: https://github.com/apache/airflow/pull/7533#issuecomment-590812592 I did it earlier in this PR: https://github.com/apache/airflow/pull/7527 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (AIRFLOW-5924) Automatically unify bucket name and key in S3Hook
[ https://issues.apache.org/jira/browse/AIRFLOW-5924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall resolved AIRFLOW-5924. --- Resolution: Fixed > Automatically unify bucket name and key in S3Hook > - > > Key: AIRFLOW-5924 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5924 > Project: Apache Airflow > Issue Type: New Feature > Components: aws, hooks >Affects Versions: 2.0.0, 1.10.6 >Reporter: Felix Uellendall >Assignee: Felix Uellendall >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5924) Automatically unify bucket name and key in S3Hook
[ https://issues.apache.org/jira/browse/AIRFLOW-5924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17044316#comment-17044316 ] ASF subversion and git services commented on AIRFLOW-5924: -- Commit 6eaa7e3b1845644d5ec65a00a997f4029bec9628 in airflow's branch refs/heads/master from Felix Uellendall [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=6eaa7e3 ] [AIRFLOW-5924] Automatically unify bucket name and key in S3Hook (#6574) - change provide_bucket_name to provide bucket name also for function with keys - refactoring > Automatically unify bucket name and key in S3Hook > - > > Key: AIRFLOW-5924 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5924 > Project: Apache Airflow > Issue Type: New Feature > Components: aws, hooks >Affects Versions: 2.0.0, 1.10.6 >Reporter: Felix Uellendall >Assignee: Felix Uellendall >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5924) Automatically unify bucket name and key in S3Hook
[ https://issues.apache.org/jira/browse/AIRFLOW-5924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17044315#comment-17044315 ] ASF GitHub Bot commented on AIRFLOW-5924: - feluelle commented on pull request #6574: [AIRFLOW-5924] Automatically unify bucket name and key in S3Hook URL: https://github.com/apache/airflow/pull/6574 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Automatically unify bucket name and key in S3Hook > - > > Key: AIRFLOW-5924 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5924 > Project: Apache Airflow > Issue Type: New Feature > Components: aws, hooks >Affects Versions: 2.0.0, 1.10.6 >Reporter: Felix Uellendall >Assignee: Felix Uellendall >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] feluelle merged pull request #6574: [AIRFLOW-5924] Automatically unify bucket name and key in S3Hook
feluelle merged pull request #6574: [AIRFLOW-5924] Automatically unify bucket name and key in S3Hook URL: https://github.com/apache/airflow/pull/6574 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on issue #7269: [AIRFLOW-6651] Add Redis Heartbeat option
ashb commented on issue #7269: [AIRFLOW-6651] Add Redis Heartbeat option URL: https://github.com/apache/airflow/pull/7269#issuecomment-590800040 A question: Have you looked to see if this feature is built in to celery? Is it possible we could rely on celery more for this? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on issue #7269: [AIRFLOW-6651] Add Redis Heartbeat option
ashb commented on issue #7269: [AIRFLOW-6651] Add Redis Heartbeat option URL: https://github.com/apache/airflow/pull/7269#issuecomment-590799186 I can see this being useful so long as it does actually reduce DB load/active connections :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6574: [AIRFLOW-5924] Automatically unify bucket name and key in S3Hook
codecov-io edited a comment on issue #6574: [AIRFLOW-5924] Automatically unify bucket name and key in S3Hook URL: https://github.com/apache/airflow/pull/6574#issuecomment-553548307 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6574?src=pr&el=h1) Report > Merging [#6574](https://codecov.io/gh/apache/airflow/pull/6574?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/3320e432a129476dbc1c55be3b3faa3326a635bc?src=pr&el=desc) will **decrease** coverage by `0.09%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/6574?src=pr&el=tree) ```diff @@Coverage Diff@@ ## master#6574 +/- ## = - Coverage86.8% 86.71% -0.1% = Files 893 896 +3 Lines 4224143283 +1042 = + Hits3666937534+865 - Misses 5572 5749+177 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6574?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/providers/amazon/aws/hooks/s3.py](https://codecov.io/gh/apache/airflow/pull/6574/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYW1hem9uL2F3cy9ob29rcy9zMy5weQ==) | `96.6% <100%> (+0.74%)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6574/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6574/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6574/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `47.18% <0%> (-45.08%)` | :arrow_down: | | [...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/6574/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==) | `69.69% <0%> (-25.26%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6574/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [airflow/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/6574/diff?src=pr&el=tree#diff-YWlyZmxvdy9fX2luaXRfXy5weQ==) | `93.33% <0%> (-6.67%)` | :arrow_down: | | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/6574/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `90.98% <0%> (-1.13%)` | :arrow_down: | | [...providers/microsoft/azure/hooks/azure\_fileshare.py](https://codecov.io/gh/apache/airflow/pull/6574/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbWljcm9zb2Z0L2F6dXJlL2hvb2tzL2F6dXJlX2ZpbGVzaGFyZS5weQ==) | `100% <0%> (ø)` | :arrow_up: | | [...ow/providers/google/cloud/operators/stackdriver.py](https://codecov.io/gh/apache/airflow/pull/6574/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL29wZXJhdG9ycy9zdGFja2RyaXZlci5weQ==) | `100% <0%> (ø)` | | | ... and [10 more](https://codecov.io/gh/apache/airflow/pull/6574/diff?src=pr&el=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6574?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6574?src=pr&el=footer). Last update [3320e43...574aac6](https://codecov.io/gh/apache/airflow/pull/6574?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feluelle commented on issue #6576: [AIRFLOW-5922] Add option to specify the mysql client library used in MySqlHook
feluelle commented on issue #6576: [AIRFLOW-5922] Add option to specify the mysql client library used in MySqlHook URL: https://github.com/apache/airflow/pull/6576#issuecomment-590781661 I did rebase. I will investigate the issue further when I have more time for it :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on issue #7484: [AIRFLOW-6864] Make airflow/jobs pylint compatible
nuclearpinguin commented on issue #7484: [AIRFLOW-6864] Make airflow/jobs pylint compatible URL: https://github.com/apache/airflow/pull/7484#issuecomment-590767204 @kaxil @potiuk @mik-laj @ashb are we good to go? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-659) Automatic Refresh on DAG Graph View
[ https://issues.apache.org/jira/browse/AIRFLOW-659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17044279#comment-17044279 ] Ashley Collins commented on AIRFLOW-659: I would also like this feature. I was thinking of something where state changes are pushed to the browser from the webserver via websockets. > Automatic Refresh on DAG Graph View > --- > > Key: AIRFLOW-659 > URL: https://issues.apache.org/jira/browse/AIRFLOW-659 > Project: Apache Airflow > Issue Type: Improvement > Components: ui, webserver >Reporter: Robin Miller >Assignee: Robin Miller >Priority: Minor > > It would be nice not to have to press a button to refresh the graph view all > the time. There's no reason not to have the option to have this refresh > itself regularly so you can have it on a screen in the background and know > that it's reasonably up to date when you glance across at it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] nuclearpinguin commented on issue #6371: [AIRFLOW-5691] Rewrite Dataproc operators to use python library
nuclearpinguin commented on issue #6371: [AIRFLOW-5691] Rewrite Dataproc operators to use python library URL: https://github.com/apache/airflow/pull/6371#issuecomment-590757917 There's a JIRA: https://issues.apache.org/jira/browse/AIRFLOW-5691 No change was intentional. Also, this change was not backported to 1.10.x . So, from what I understand the submit job operator should: - check if job with provided id already exists - if yes then the existing job should be used - if no create new job Doesn't `request_id` work like that? > If the server receives two SubmitJobRequest requests with the same id, then the second request will be ignored and the first Job created and stored in the backend is returned. https://googleapis.dev/python/dataproc/latest/gapic/v1/api.html#google.cloud.dataproc_v1.JobControllerClient.submit_job @dossett @digger This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] blcksrx commented on issue #7422: [AIRFLOW-6809] Test for presto operators
blcksrx commented on issue #7422: [AIRFLOW-6809] Test for presto operators URL: https://github.com/apache/airflow/pull/7422#issuecomment-590757616 > Flake8 errors: > > ``` > tests/providers/apache/hive/operators/test_hive.py:156:14: F821 undefined name 'HdfsSensor' > tests/providers/apache/hive/operators/test_hive.py:164:14: F821 undefined name 'WebHdfsSensor' > tests/providers/apache/hive/operators/test_hive.py:185:30: F821 undefined name 'DEFAULT_DATE_DS' > tests/providers/apache/hive/operators/test_hive.py:219:32: F821 undefined name 'AirflowSensorTimeout' > tests/providers/apache/hive/operators/test_hive.py:244:43: F821 undefined name 'DEFAULT_DATE_DS' > There were some flake8 errors. Exiting > ``` I know! i will fix it as soon as possible This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7484: [AIRFLOW-6864] Make airflow/jobs pylint compatible
codecov-io edited a comment on issue #7484: [AIRFLOW-6864] Make airflow/jobs pylint compatible URL: https://github.com/apache/airflow/pull/7484#issuecomment-590051665 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7484?src=pr&el=h1) Report > :exclamation: No coverage uploaded for pull request base (`master@20b6b34`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit). > The diff coverage is `90.99%`. [](https://codecov.io/gh/apache/airflow/pull/7484?src=pr&el=tree) ```diff @@Coverage Diff@@ ## master#7484 +/- ## = Coverage ? 86.85% = Files ? 896 Lines ?42658 Branches ?0 = Hits ?37051 Misses? 5607 Partials ?0 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7484?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/task/task\_runner/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/7484/diff?src=pr&el=tree#diff-YWlyZmxvdy90YXNrL3Rhc2tfcnVubmVyL19faW5pdF9fLnB5) | `63.63% <ø> (ø)` | | | [airflow/task/task\_runner/base\_task\_runner.py](https://codecov.io/gh/apache/airflow/pull/7484/diff?src=pr&el=tree#diff-YWlyZmxvdy90YXNrL3Rhc2tfcnVubmVyL2Jhc2VfdGFza19ydW5uZXIucHk=) | `88.88% <ø> (ø)` | | | [airflow/cli/commands/scheduler\_command.py](https://codecov.io/gh/apache/airflow/pull/7484/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvc2NoZWR1bGVyX2NvbW1hbmQucHk=) | `0% <0%> (ø)` | | | [airflow/utils/types.py](https://codecov.io/gh/apache/airflow/pull/7484/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy90eXBlcy5weQ==) | `100% <100%> (ø)` | | | [airflow/cli/commands/task\_command.py](https://codecov.io/gh/apache/airflow/pull/7484/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvdGFza19jb21tYW5kLnB5) | `72.94% <100%> (ø)` | | | [airflow/models/dag.py](https://codecov.io/gh/apache/airflow/pull/7484/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFnLnB5) | `91.37% <100%> (ø)` | | | [airflow/ti\_deps/deps/dagrun\_id\_dep.py](https://codecov.io/gh/apache/airflow/pull/7484/diff?src=pr&el=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvZGFncnVuX2lkX2RlcC5weQ==) | `100% <100%> (ø)` | | | [airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/7484/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5) | `90.47% <100%> (ø)` | | | [airflow/models/dagrun.py](https://codecov.io/gh/apache/airflow/pull/7484/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFncnVuLnB5) | `95.74% <100%> (ø)` | | | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/7484/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `92.15% <100%> (ø)` | | | ... and [10 more](https://codecov.io/gh/apache/airflow/pull/7484/diff?src=pr&el=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7484?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7484?src=pr&el=footer). Last update [20b6b34...0027102](https://codecov.io/gh/apache/airflow/pull/7484?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] Fokko commented on issue #7533: [AIRFLOW-5485] - scheduler_job: replace == None with is
Fokko commented on issue #7533: [AIRFLOW-5485] - scheduler_job: replace == None with is URL: https://github.com/apache/airflow/pull/7533#issuecomment-590755676 Thanks @saurabhdhupar for the contribution. Let's wait for the CI. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] Fokko commented on a change in pull request #7533: [AIRFLOW-5485] - scheduler_job: replace == None with is
Fokko commented on a change in pull request #7533: [AIRFLOW-5485] -
scheduler_job: replace == None with is
URL: https://github.com/apache/airflow/pull/7533#discussion_r383736315
##
File path: airflow/jobs/scheduler_job.py
##
@@ -915,7 +915,7 @@ class SchedulerJob(BaseJob):
}
heartrate = conf.getint('scheduler', 'SCHEDULER_HEARTBEAT_SEC')
-def __init__(
+def __init__(
Review comment:
I think this will cause a lint error.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] Fokko commented on a change in pull request #7533: [AIRFLOW-5485] - scheduler_job: replace == None with is
Fokko commented on a change in pull request #7533: [AIRFLOW-5485] -
scheduler_job: replace == None with is
URL: https://github.com/apache/airflow/pull/7533#discussion_r383736315
##
File path: airflow/jobs/scheduler_job.py
##
@@ -915,7 +915,7 @@ class SchedulerJob(BaseJob):
}
heartrate = conf.getint('scheduler', 'SCHEDULER_HEARTBEAT_SEC')
-def __init__(
+def __init__(
Review comment:
I think this additional space will cause a lint error. Please revert this
line.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-6815) Dag Graph View Javascript Error
[
https://issues.apache.org/jira/browse/AIRFLOW-6815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17044246#comment-17044246
]
Ashley Collins commented on AIRFLOW-6815:
-
We saw the same thing when Chrome was upgraded to v80. The latest point release
of Chrome seems to have fixed it for us (80.0.3987.87 to 80.0.3987.106)
> Dag Graph View Javascript Error
> ---
>
> Key: AIRFLOW-6815
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6815
> Project: Apache Airflow
> Issue Type: Bug
> Components: DAG
>Affects Versions: 1.10.9
> Environment: ubuntu
>Reporter: ceyhun kerti
>Assignee: Andrew
>Priority: Blocker
> Attachments: Screenshot at 2020-02-15 13-52-20.png, my_dag.py
>
>
> airflow does not display graph view after certain amount of parallel tasks,
> and gives javascript error in console.
>
> [fresh installed airflow|[https://airflow.apache.org/docs/stable/start.html#]]
> and added the dag in attachment [^my_dag.py]
>
>
>
> {code:java}
> dagre-d3.js:3704 Uncaught Error: Edge 'undefined' is not in graph
> at Constructor.BaseGraph._strictGetEdge (dagre-d3.js:3704)
> at Constructor.BaseGraph._delEdge (dagre-d3.js:3662)
> at Constructor.Digraph.delEdge (dagre-d3.js:4048)
> at Constructor. (dagre-d3.js:3633)
> at Array.forEach ()
> at Constructor.BaseGraph.delNode (dagre-d3.js:3633)
> at Constructor.Digraph.delNode (dagre-d3.js:4016)
> at Constructor.delNode (dagre-d3.js:4809)
> at dagre-d3.js:1551
> at Constructor.BaseGraph.eachNode (dagre-d3.js:3585)
> BaseGraph._strictGetEdge @ dagre-d3.js:3704
> BaseGraph._delEdge @ dagre-d3.js:3662
> Digraph.delEdge @ dagre-d3.js:4048
> (anonymous) @ dagre-d3.js:3633
> BaseGraph.delNode @ dagre-d3.js:3633
> Digraph.delNode @ dagre-d3.js:4016
> Constructor.delNode @ dagre-d3.js:4809
> (anonymous) @ dagre-d3.js:1551
> BaseGraph.eachNode @ dagre-d3.js:3585
> undoNormalize @ dagre-d3.js:1541
> (anonymous) @ dagre-d3.js:3466
> run @ dagre-d3.js:1477
> (anonymous) @ dagre-d3.js:3466
> runLayout @ dagre-d3.js:268
> Renderer.run @ dagre-d3.js:202
> (anonymous) @ graph?dag_id=my_dag&execution_date=:949{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6909) Prepare backport packages on post-test stage
[ https://issues.apache.org/jira/browse/AIRFLOW-6909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Urbaszek resolved AIRFLOW-6909. -- Fix Version/s: 1.10.10 Resolution: Done > Prepare backport packages on post-test stage > > > Key: AIRFLOW-6909 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6909 > Project: Apache Airflow > Issue Type: New Feature > Components: ci >Affects Versions: 1.10.9 >Reporter: Kamil Bregula >Priority: Major > Fix For: 1.10.10 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6909) Prepare backport packages on post-test stage
[ https://issues.apache.org/jira/browse/AIRFLOW-6909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17044237#comment-17044237 ] ASF subversion and git services commented on AIRFLOW-6909: -- Commit 212c02b1b00c21af20a5798d9f72b2961c59f15e in airflow's branch refs/heads/master from Kamil Breguła [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=212c02b ] [AIRFLOW-6909] Prepare backport packages on post-test stage (#7529) > Prepare backport packages on post-test stage > > > Key: AIRFLOW-6909 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6909 > Project: Apache Airflow > Issue Type: New Feature > Components: ci >Affects Versions: 1.10.9 >Reporter: Kamil Bregula >Priority: Major > Fix For: 1.10.10 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6909) Prepare backport packages on post-test stage
[ https://issues.apache.org/jira/browse/AIRFLOW-6909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17044236#comment-17044236 ] ASF GitHub Bot commented on AIRFLOW-6909: - nuclearpinguin commented on pull request #7529: [AIRFLOW-6909] Prepare backport packages on post-test stage URL: https://github.com/apache/airflow/pull/7529 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Prepare backport packages on post-test stage > > > Key: AIRFLOW-6909 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6909 > Project: Apache Airflow > Issue Type: New Feature > Components: ci >Affects Versions: 1.10.9 >Reporter: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] nuclearpinguin merged pull request #7529: [AIRFLOW-6909] Prepare backport packages on post-test stage
nuclearpinguin merged pull request #7529: [AIRFLOW-6909] Prepare backport packages on post-test stage URL: https://github.com/apache/airflow/pull/7529 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Comment Edited] (AIRFLOW-5071) Thousand os Executor reports task instance X finished (success) although the task says its queued. Was the task killed externally?
[
https://issues.apache.org/jira/browse/AIRFLOW-5071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17044214#comment-17044214
]
Adam Angeli edited comment on AIRFLOW-5071 at 2/25/20 8:24 AM:
---
Another variant of this involves {{_executeHelper_}} _from {{SchedulerJob}}._
It changes the state of tasks to _queued_ and adds them to the executor's
{{queued_tasks}}. It then proceeds to heartbeat the executor, which may not
process all the {{queued_tasks}} if there aren't open slots. It follows up by
calling {{_change_state_for_tasks_failed_to_execute_}}_,_ which resets the task
state to _scheduled_ for anything that couldn't be processed. But it leaves
them in {{queued_tasks}} and they will eventually make their way to a celery
worker, still in the _scheduled_ state. It looks like this may be fixed in
{{1.10.7}}+, as the task instance is removed from {{queued_tasks}} after the
state is set back to _scheduled_. As a stopgap, you could try increasing your
{{core.parallelism}} setting to avoid the scenario where you don't have any
open slots.
was (Author: adam.angeli):
Another variant of this involves {{_executeHelper_}} _from {{SchedulerJob}}._
It changes the state of tasks to _queued_ and adds them to the executor's
{{queued_tasks}}. It then proceeds to heartbeat the executor, which may not
process all the {{queued_tasks}} if there aren't open slots. It follows up by
calling {{_change_state_for_tasks_failed_to_execute_}}_, which resets the task
state to _scheduled_ for anything that couldn't be processed. But it leaves
them in {{queued_tasks}} and they will eventually make their way to a celery
worker, still in the _scheduled_ state. It looks like this may be fixed in
{{1.10.7}}+, as the task instance is removed from {{queued_tasks}} after the
state is set back to _scheduled_. As a stopgap, you could try increasing your
{{core.parallelism}} setting to avoid the scenario where you don't have any
open slots.
> Thousand os Executor reports task instance X finished (success) although the
> task says its queued. Was the task killed externally?
> --
>
> Key: AIRFLOW-5071
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5071
> Project: Apache Airflow
> Issue Type: Bug
> Components: DAG, scheduler
>Affects Versions: 1.10.3
>Reporter: msempere
>Priority: Critical
> Attachments: image-2020-01-27-18-10-29-124.png
>
>
> I'm opening this issue because since I update to 1.10.3 I'm seeing thousands
> of daily messages like the following in the logs:
>
> ```
> {{__init__.py:1580}} ERROR - Executor reports task instance 2019-07-29 00:00:00+00:00 [queued]> finished (success) although the task says
> its queued. Was the task killed externally?
> {{jobs.py:1484}} ERROR - Executor reports task instance 2019-07-29 00:00:00+00:00 [queued]> finished (success) although the task says
> its queued. Was the task killed externally?
> ```
> -And looks like this is triggering also thousand of daily emails because the
> flag to send email in case of failure is set to True.-
> I have Airflow setup to use Celery and Redis as a backend queue service.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (AIRFLOW-5071) Thousand os Executor reports task instance X finished (success) although the task says its queued. Was the task killed externally?
[
https://issues.apache.org/jira/browse/AIRFLOW-5071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17044214#comment-17044214
]
Adam Angeli edited comment on AIRFLOW-5071 at 2/25/20 8:20 AM:
---
Another variant of this involves {{_executeHelper_}} _from {{SchedulerJob}}._
It changes the state of tasks to _queued_ and adds them to the executor's
{{queued_tasks}}. It then proceeds to heartbeat the executor, which may not
process all the {{queued_tasks}} if there aren't open slots. It follows up by
calling {{_change_state_for_tasks_failed_to_execute_}}_, which resets the task
state to _scheduled_ for anything that couldn't be processed. But it leaves
them in {{queued_tasks}} and they will eventually make their way to a celery
worker, still in the _scheduled_ state. It looks like this may be fixed in
{{1.10.7}}+, as the task instance is removed from {{queued_tasks}} after the
state is set back to _scheduled_. As a stopgap, you could try increasing your
{{core.parallelism}} setting to avoid the scenario where you don't have any
open slots.
was (Author: adam.angeli):
Another variant of this involves {{_executeHelper_}} _from {{SchedulerJob}}._
It changes the state of tasks to _queued_ and adds them to the executor's
{{queued_tasks}}. It then proceeds to heartbeat the executor, which may not
process all the {{queued_tasks}} if there aren't open slots. It follows up by
calling {{_change_state_for_tasks_failed_to_execute_}}_, which resets the task
state to _scheduled_ for anything that couldn't be processed. But it leaves
them in {{queued_tasks}} and they will eventually make their way to a celery
worker, still in the _scheduled_ state. It looks like this may be fixed in
{{1.7.0}}+, as the task instance is removed from {{queued_tasks}} after the
state is set back to _scheduled_. As a stopgap, you could try increasing your
{{core.parallelism}} setting to avoid the scenario where you don't have any
open slots.
> Thousand os Executor reports task instance X finished (success) although the
> task says its queued. Was the task killed externally?
> --
>
> Key: AIRFLOW-5071
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5071
> Project: Apache Airflow
> Issue Type: Bug
> Components: DAG, scheduler
>Affects Versions: 1.10.3
>Reporter: msempere
>Priority: Critical
> Attachments: image-2020-01-27-18-10-29-124.png
>
>
> I'm opening this issue because since I update to 1.10.3 I'm seeing thousands
> of daily messages like the following in the logs:
>
> ```
> {{__init__.py:1580}} ERROR - Executor reports task instance 2019-07-29 00:00:00+00:00 [queued]> finished (success) although the task says
> its queued. Was the task killed externally?
> {{jobs.py:1484}} ERROR - Executor reports task instance 2019-07-29 00:00:00+00:00 [queued]> finished (success) although the task says
> its queued. Was the task killed externally?
> ```
> -And looks like this is triggering also thousand of daily emails because the
> flag to send email in case of failure is set to True.-
> I have Airflow setup to use Celery and Redis as a backend queue service.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] saurabhdhupar opened a new pull request #7533: Fix for Airflow 5485
saurabhdhupar opened a new pull request #7533: Fix for Airflow 5485 URL: https://github.com/apache/airflow/pull/7533 --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [x ] Description above provides context of the change - [ x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [x ] Unit tests coverage for changes (not needed for documentation changes) - [ x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x ] Relevant documentation is updated including usage instructions. - [x ] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Comment Edited] (AIRFLOW-5071) Thousand os Executor reports task instance X finished (success) although the task says its queued. Was the task killed externally?
[
https://issues.apache.org/jira/browse/AIRFLOW-5071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17044214#comment-17044214
]
Adam Angeli edited comment on AIRFLOW-5071 at 2/25/20 8:16 AM:
---
Another variant of this involves {{_executeHelper_}} _from {{SchedulerJob}}._
It changes the state of tasks to _queued_ and adds them to the executor's
{{queued_tasks}}. It then proceeds to heartbeat the executor, which may not
process all the {{queued_tasks}} if there aren't open slots. It follows up by
calling {{_change_state_for_tasks_failed_to_execute_}}_, which resets the task
state to _scheduled_ for anything that couldn't be processed. But it leaves
them in {{queued_tasks}} and they will eventually make their way to a celery
worker, still in the _scheduled_ state. It looks like this may be fixed in
{{1.7.0}}+, as the task instance is removed from {{queued_tasks}} after the
state is set back to _scheduled_. As a stopgap, you could try increasing your
{{core.parallelism}} setting to avoid the scenario where you don't have any
open slots.
was (Author: adam.angeli):
Another variant of this involves {{_executeHelper}} from {{SchedulerJob}}. It
changes the state of tasks to _queued_ and adds them to the executor's
{{queued_tasks}}. It then proceeds to heartbeat the executor, which may not
process all the {{queued_tasks}} if there aren't open slots. It follows up by
calling {{_change_state_for_tasks_failed_to_execute}}, which resets the task
state to _scheduled_ for anything that couldn't be processed. But it leaves
them in {{queued_tasks}} and they will eventually make their way to a celery
worker, still in the _scheduled_ state. It looks like this may be fixed in
{{1.7.0}}+, as the task instance is removed from {{queued_tasks}} after the
state is set back to _scheduled_. As a stopgap, you could try increasing your
{{core.parallelism}} setting to avoid the scenario where you don't have any
open slots.
> Thousand os Executor reports task instance X finished (success) although the
> task says its queued. Was the task killed externally?
> --
>
> Key: AIRFLOW-5071
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5071
> Project: Apache Airflow
> Issue Type: Bug
> Components: DAG, scheduler
>Affects Versions: 1.10.3
>Reporter: msempere
>Priority: Critical
> Attachments: image-2020-01-27-18-10-29-124.png
>
>
> I'm opening this issue because since I update to 1.10.3 I'm seeing thousands
> of daily messages like the following in the logs:
>
> ```
> {{__init__.py:1580}} ERROR - Executor reports task instance 2019-07-29 00:00:00+00:00 [queued]> finished (success) although the task says
> its queued. Was the task killed externally?
> {{jobs.py:1484}} ERROR - Executor reports task instance 2019-07-29 00:00:00+00:00 [queued]> finished (success) although the task says
> its queued. Was the task killed externally?
> ```
> -And looks like this is triggering also thousand of daily emails because the
> flag to send email in case of failure is set to True.-
> I have Airflow setup to use Celery and Redis as a backend queue service.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5071) Thousand os Executor reports task instance X finished (success) although the task says its queued. Was the task killed externally?
[
https://issues.apache.org/jira/browse/AIRFLOW-5071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17044214#comment-17044214
]
Adam Angeli commented on AIRFLOW-5071:
--
Another variant of this involves {{_executeHelper}} from {{SchedulerJob}}. It
changes the state of tasks to _queued_ and adds them to the executor's
{{queued_tasks}}. It then proceeds to heartbeat the executor, which may not
process all the {{queued_tasks}} if there aren't open slots. It follows up by
calling {{_change_state_for_tasks_failed_to_execute}}, which resets the task
state to _scheduled_ for anything that couldn't be processed. But it leaves
them in {{queued_tasks}} and they will eventually make their way to a celery
worker, still in the _scheduled_ state. It looks like this may be fixed in
{{1.7.0}}+, as the task instance is removed from {{queued_tasks}} after the
state is set back to _scheduled_. As a stopgap, you could try increasing your
{{core.parallelism}} setting to avoid the scenario where you don't have any
open slots.
> Thousand os Executor reports task instance X finished (success) although the
> task says its queued. Was the task killed externally?
> --
>
> Key: AIRFLOW-5071
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5071
> Project: Apache Airflow
> Issue Type: Bug
> Components: DAG, scheduler
>Affects Versions: 1.10.3
>Reporter: msempere
>Priority: Critical
> Attachments: image-2020-01-27-18-10-29-124.png
>
>
> I'm opening this issue because since I update to 1.10.3 I'm seeing thousands
> of daily messages like the following in the logs:
>
> ```
> {{__init__.py:1580}} ERROR - Executor reports task instance 2019-07-29 00:00:00+00:00 [queued]> finished (success) although the task says
> its queued. Was the task killed externally?
> {{jobs.py:1484}} ERROR - Executor reports task instance 2019-07-29 00:00:00+00:00 [queued]> finished (success) although the task says
> its queued. Was the task killed externally?
> ```
> -And looks like this is triggering also thousand of daily emails because the
> flag to send email in case of failure is set to True.-
> I have Airflow setup to use Celery and Redis as a backend queue service.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] kaxil merged pull request #7530: [AIRFLOW-XXXX] Update docs on starting Kubernetes tests
kaxil merged pull request #7530: [AIRFLOW-] Update docs on starting Kubernetes tests URL: https://github.com/apache/airflow/pull/7530 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
