[jira] [Created] (AIRFLOW-2447) Fix TestHiveMetastoreHook to run all cases

2018-05-09 Thread Kengo Seki (JIRA) 
Kengo Seki created AIRFLOW-2447:
---

 Summary: Fix TestHiveMetastoreHook to run all cases
 Key: AIRFLOW-2447
 URL: https://issues.apache.org/jira/browse/AIRFLOW-2447
 Project: Apache Airflow
  Issue Type: Bug
  Components: hive_hooks, tests
Reporter: Kengo Seki
Assignee: Kengo Seki


TestHiveMetastoreHook has a method called {{get_databases}} but it isn't 
executed since its name doesn't start with {{test_}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (AIRFLOW-2429) Make Airflow Flake8 compliant

2018-05-09 Thread Tao Feng (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tao Feng reassigned AIRFLOW-2429:
-

Assignee: Tao Feng

> Make Airflow Flake8 compliant 
> --
>
> Key: AIRFLOW-2429
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2429
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Fokko Driesprong
>Assignee: Tao Feng
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-2446) Add S3ToRedshiftTransfer to the "Integration" doc

2018-05-09 Thread Kengo Seki (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kengo Seki updated AIRFLOW-2446:

Summary: Add S3ToRedshiftTransfer to the "Integration" doc  (was: Add 
S3ToRedshiftTransfer into the "Integration" doc)

> Add S3ToRedshiftTransfer to the "Integration" doc
> -
>
> Key: AIRFLOW-2446
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2446
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: aws, docs, Documentation, redshift
>Reporter: Kengo Seki
>Assignee: Kengo Seki
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (AIRFLOW-2402) Airflow 1.10 Logs UI throws oops error

2018-05-09 Thread Kevin Yang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kevin Yang reassigned AIRFLOW-2402:
---

Assignee: Kevin Yang  (was: Ramki Subramanian)

> Airflow 1.10 Logs UI throws oops error
> --
>
> Key: AIRFLOW-2402
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2402
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: authentication, ui
>Affects Versions: 1.10.0
>Reporter: Ramki Subramanian
>Assignee: Kevin Yang
>Priority: Major
> Fix For: 1.10.0
>
>
> Hi,
> I am getting an error at
> [incubator-airflow/airflow/www_rbac/views.py|https://github.com/apache/incubator-airflow/blob/4d64ad4928f0188f7532936e8da6612f5ec7170d/airflow/www_rbac/views.py#L454]
> Line 454 in 
> [4d64ad4|https://github.com/apache/incubator-airflow/commit/4d64ad4928f0188f7532936e8da6612f5ec7170d]
> | |logs[i] = log.decode('utf-8')|
>  
> {{/home/user/lib/python2.7/site-packages/airflow/www_rbac/views.py", line 
> 454, in log logs[i] = log.decode('utf-8') AttributeError: 'list' object has 
> no attribute 'decode' }}
> Not sure if someone is already looking into this, or I am missing some config?
> Branch : 1.10_test
> More Info here:
> [https://github.com/apache/incubator-airflow/commit/05e1861e24de42f9a2c649cd93041c5c744504e1#diff-77df5adb32d964f37748c4557ffb3c4c]
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AIRFLOW-2446) Add S3ToRedshiftTransfer into the "Integration" doc

2018-05-09 Thread Kengo Seki (JIRA) 
Kengo Seki created AIRFLOW-2446:
---

 Summary: Add S3ToRedshiftTransfer into the "Integration" doc
 Key: AIRFLOW-2446
 URL: https://issues.apache.org/jira/browse/AIRFLOW-2446
 Project: Apache Airflow
  Issue Type: Improvement
  Components: aws, docs, Documentation, redshift
Reporter: Kengo Seki
Assignee: Kengo Seki






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

incubator-airflow git commit: closes apache/incubator-airflow#3325 *Need to be reopened and follow contributor guidelines*

2018-05-09 Thread sanand 
Repository: incubator-airflow
Updated Branches:
  refs/heads/master 2a55ffe0c -> 06017f66e


closes apache/incubator-airflow#3325 *Need to be reopened and follow 
contributor guidelines*


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/06017f66
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/06017f66
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/06017f66

Branch: refs/heads/master
Commit: 06017f66eed0aef2c9c441bbd0ae5869059d011e
Parents: 2a55ffe
Author: r39132 
Authored: Wed May 9 14:14:04 2018 -0700
Committer: r39132 
Committed: Wed May 9 14:14:04 2018 -0700

--

--

[jira] [Created] (AIRFLOW-2445) Allow templating for kubernetes operator

2018-05-09 Thread Sergio B (JIRA) 
Sergio B created AIRFLOW-2445:
-

 Summary: Allow templating for kubernetes operator
 Key: AIRFLOW-2445
 URL: https://issues.apache.org/jira/browse/AIRFLOW-2445
 Project: Apache Airflow
  Issue Type: Bug
Reporter: Sergio B


Allow templating in command, arguments and environment variables



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-2445) Allow templating for kubernetes operator

2018-05-09 Thread Sergio B (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergio B updated AIRFLOW-2445:
--
Issue Type: Improvement  (was: Bug)

> Allow templating for kubernetes operator
> 
>
> Key: AIRFLOW-2445
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2445
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Sergio B
>Priority: Major
>
> Allow templating in command, arguments and environment variables



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AIRFLOW-2444) Remove unused option(include_adhoc) in cli backfill command

2018-05-09 Thread Tao Feng (JIRA) 
Tao Feng created AIRFLOW-2444:
-

 Summary: Remove unused option(include_adhoc) in cli backfill 
command
 Key: AIRFLOW-2444
 URL: https://issues.apache.org/jira/browse/AIRFLOW-2444
 Project: Apache Airflow
  Issue Type: Bug
Reporter: Tao Feng
Assignee: Tao Feng


Include_adhoc is removed in this 
pr([https://github.com/apache/incubator-airflow/pull/1667/files).] This option 
currently doesn't do anything. Just remove it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-2253) Adding Airflow CLI instrumentation

2018-05-09 Thread Jin Hyuk Chang (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-2253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469396#comment-16469396
 ] 

Jin Hyuk Chang commented on AIRFLOW-2253:
-

[~kevcampb],

I could be wrong, but I believe it's a correct behavior of initdb to just 
create table but not adding any record in it. The error message is just for 
information but it does not actually make "initdb" operation fail. I've posted 
a PR to remove cli_logger on "initdb".

Let me know if you have any issue.

 

Thanks,

Jin

> Adding Airflow CLI instrumentation
> --
>
> Key: AIRFLOW-2253
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2253
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Jin Hyuk Chang
>Assignee: Jin Hyuk Chang
>Priority: Major
>
> We are trying to improve user experience on Airflow particularly on Airflow 
> CLI where our user does most of testing in CLI. As a first step, we'd like to 
> instrument user's activity on CLI.
> Currently, Airflow instruments user activity from web UI via 
> www.utils.action_logger, but not on CLI and we are trying to instrument user 
> activity from Airflow CLI as well. By default, we'd like to instrument same 
> as what current behavior from Web UI.
> Should it need different instrumentation, we will provide a registry method 
> to register callback so that customized instrument can be easily plugged in. 
> (e.g: We have our customized event topic that already wired with ingestion 
> pipeline. Having pluggable callback, making integration with customized 
> instrumentation a breeze.)
> Thanks!
> Jin



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-2442) Airflow run command leaves database connections open

2018-05-09 Thread Alejandro Fernandez (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alejandro Fernandez updated AIRFLOW-2442:
-
Summary: Airflow run command leaves database connections open  (was: 
Airflow run command leaves database connections open,)

> Airflow run command leaves database connections open
> 
>
> Key: AIRFLOW-2442
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2442
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: cli
>Affects Versions: Airflow 1.8, 1.8.0
>Reporter: Alejandro Fernandez
>Assignee: Alejandro Fernandez
>Priority: Major
> Fix For: Airflow 2.0
>
> Attachments: connection_duration_1_hour.png, db_connections.png, 
> fixed_before_and_after.jpg, monthly_db_connections.png, running_tasks.png
>
>
> *Summary*
> The "airflow run" command creates a connection to the database and leaves it 
> open (until killed by SQLALchemy later). The number of these connections can 
> skyrocket whenever hundreds/thousands of tasks are launched simultaneously, 
> and potentially hit the database connection limit.
> The problem is that in cli.py, the run() method first calls 
> {code:java}
> settings.configure_orm(disable_connection_pool=True){code}
> correctly
>  to use a NullPool, but then parses any custom configs and again calls
> {code:java}
> settings.configure_orm(){code}
> , thereby overriding the desired behavior by instead using a QueuePool.
>  The QueuePool uses the default configs for SQL_ALCHEMY_POOL_SIZE and 
> SQL_ALCHEMY_POOL_RECYCLE. This means that while the task is running and the 
> executor is sending heartbeats, the sleeping connection is idle until it is 
> killed by SQLAlchemy.
> This fixes a bug introduced by 
> [https://github.com/apache/incubator-airflow/pull/1934] in 
> [https://github.com/apache/incubator-airflow/pull/1934/commits/b380013634b02bb4c1b9d1cc587ccd12383820b6#diff-1c2404a3a60f829127232842250ff406R344]
>   
> which is present in branches 1-8-stable, 1-9-stable, and 1-10-test
> NOTE: Will create a PR once I've done more testing since I'm on an older 
> branch. For now, attaching a patch file [^AIRFLOW-2442.patch]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-2442) Airflow run command leaves database connections open,

2018-05-09 Thread Alejandro Fernandez (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alejandro Fernandez updated AIRFLOW-2442:
-
Summary: Airflow run command leaves database connections open,  (was: 
Airflow run command leaves database connections open, which can hit the 
database limit)

> Airflow run command leaves database connections open,
> -
>
> Key: AIRFLOW-2442
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2442
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: cli
>Affects Versions: Airflow 1.8, 1.8.0
>Reporter: Alejandro Fernandez
>Assignee: Alejandro Fernandez
>Priority: Major
> Fix For: Airflow 2.0
>
> Attachments: connection_duration_1_hour.png, db_connections.png, 
> fixed_before_and_after.jpg, monthly_db_connections.png, running_tasks.png
>
>
> *Summary*
> The "airflow run" command creates a connection to the database and leaves it 
> open (until killed by SQLALchemy later). The number of these connections can 
> skyrocket whenever hundreds/thousands of tasks are launched simultaneously, 
> and potentially hit the database connection limit.
> The problem is that in cli.py, the run() method first calls 
> {code:java}
> settings.configure_orm(disable_connection_pool=True){code}
> correctly
>  to use a NullPool, but then parses any custom configs and again calls
> {code:java}
> settings.configure_orm(){code}
> , thereby overriding the desired behavior by instead using a QueuePool.
>  The QueuePool uses the default configs for SQL_ALCHEMY_POOL_SIZE and 
> SQL_ALCHEMY_POOL_RECYCLE. This means that while the task is running and the 
> executor is sending heartbeats, the sleeping connection is idle until it is 
> killed by SQLAlchemy.
> This fixes a bug introduced by 
> [https://github.com/apache/incubator-airflow/pull/1934] in 
> [https://github.com/apache/incubator-airflow/pull/1934/commits/b380013634b02bb4c1b9d1cc587ccd12383820b6#diff-1c2404a3a60f829127232842250ff406R344]
>   
> which is present in branches 1-8-stable, 1-9-stable, and 1-10-test
> NOTE: Will create a PR once I've done more testing since I'm on an older 
> branch. For now, attaching a patch file [^AIRFLOW-2442.patch]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-2442) Airflow run command leaves database connections open, which can hit the database limit

2018-05-09 Thread Alejandro Fernandez (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alejandro Fernandez updated AIRFLOW-2442:
-
Attachment: (was: AIRFLOW-2442.patch)

> Airflow run command leaves database connections open, which can hit the 
> database limit
> --
>
> Key: AIRFLOW-2442
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2442
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: cli
>Affects Versions: Airflow 1.8, 1.8.0
>Reporter: Alejandro Fernandez
>Assignee: Alejandro Fernandez
>Priority: Major
> Fix For: Airflow 2.0
>
> Attachments: connection_duration_1_hour.png, db_connections.png, 
> fixed_before_and_after.jpg, monthly_db_connections.png, running_tasks.png
>
>
> *Summary*
> The "airflow run" command creates a connection to the database and leaves it 
> open (until killed by SQLALchemy later). The number of these connections can 
> skyrocket whenever hundreds/thousands of tasks are launched simultaneously, 
> and potentially hit the database connection limit.
> The problem is that in cli.py, the run() method first calls 
> {code:java}
> settings.configure_orm(disable_connection_pool=True){code}
> correctly
>  to use a NullPool, but then parses any custom configs and again calls
> {code:java}
> settings.configure_orm(){code}
> , thereby overriding the desired behavior by instead using a QueuePool.
>  The QueuePool uses the default configs for SQL_ALCHEMY_POOL_SIZE and 
> SQL_ALCHEMY_POOL_RECYCLE. This means that while the task is running and the 
> executor is sending heartbeats, the sleeping connection is idle until it is 
> killed by SQLAlchemy.
> This fixes a bug introduced by 
> [https://github.com/apache/incubator-airflow/pull/1934] in 
> [https://github.com/apache/incubator-airflow/pull/1934/commits/b380013634b02bb4c1b9d1cc587ccd12383820b6#diff-1c2404a3a60f829127232842250ff406R344]
>   
> which is present in branches 1-8-stable, 1-9-stable, and 1-10-test
> NOTE: Will create a PR once I've done more testing since I'm on an older 
> branch. For now, attaching a patch file [^AIRFLOW-2442.patch]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (AIRFLOW-2393) UI tree view struggles with large dags (60 tasks)x25 dag histories

2018-05-09 Thread Arthur Wiedmer (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arthur Wiedmer resolved AIRFLOW-2393.
-
   Resolution: Fixed
Fix Version/s: 2.0.0

Issue resolved by pull request #3279
[https://github.com/apache/incubator-airflow/pull/3279]

> UI tree view struggles with large dags (60 tasks)x25 dag histories
> --
>
> Key: AIRFLOW-2393
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2393
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: ui
>Affects Versions: 1.9.0
>Reporter: Badger
>Priority: Major
> Fix For: 2.0.0
>
>
> Hi, 
> We are noticing the tree view is taking a long time to render as our DAG has 
> become more complex. We will need to start breaking our dag apart in order to 
> continue to use the user interface. 
> The basic problem is that a reasonably complex DAG (60 operators) x the 
> standard 25 dag run histories on the tree view causes a 350MB json response 
> (compressed to 8MB) to be downloaded, this then needs the browser to render 
> it.
> On quick observation this appears to be because, the response appears to 
> contain all meta-data for each task.
> Is this something others think is a problem. We occasionally have to refresh 
> due to memory errors and have already increased the RAM allocated to the box.
> A suggestion might be to load specific instance history when a user hovers 
> over the task, rather than exporting all of the history on page load. I'd 
> look at contributing a PR but haven't had chance to take a look at this area 
> of the code base.
> Thanks



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-2393) UI tree view struggles with large dags (60 tasks)x25 dag histories

2018-05-09 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-2393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468984#comment-16468984
 ] 

ASF subversion and git services commented on AIRFLOW-2393:
--

Commit 2a55ffe0cd2c04fde2eecdbeff2fa727067069e1 in incubator-airflow's branch 
refs/heads/master from Tao feng
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=2a55ffe ]

[AIRFLOW-2086][AIRFLOW-2393] Customize default dagrun number in tree view

Closes #3279 from feng-tao/reduce-tree-view

This introduces a new configuration variable to set the default
number of dag runs displayed in the tree view. For large DAGs, this
could cause timeouts in the webserver.


> UI tree view struggles with large dags (60 tasks)x25 dag histories
> --
>
> Key: AIRFLOW-2393
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2393
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: ui
>Affects Versions: 1.9.0
>Reporter: Badger
>Priority: Major
>
> Hi, 
> We are noticing the tree view is taking a long time to render as our DAG has 
> become more complex. We will need to start breaking our dag apart in order to 
> continue to use the user interface. 
> The basic problem is that a reasonably complex DAG (60 operators) x the 
> standard 25 dag run histories on the tree view causes a 350MB json response 
> (compressed to 8MB) to be downloaded, this then needs the browser to render 
> it.
> On quick observation this appears to be because, the response appears to 
> contain all meta-data for each task.
> Is this something others think is a problem. We occasionally have to refresh 
> due to memory errors and have already increased the RAM allocated to the box.
> A suggestion might be to load specific instance history when a user hovers 
> over the task, rather than exporting all of the history on page load. I'd 
> look at contributing a PR but haven't had chance to take a look at this area 
> of the code base.
> Thanks



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (AIRFLOW-2086) The tree view page is too slow when display big dag.

2018-05-09 Thread Arthur Wiedmer (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arthur Wiedmer resolved AIRFLOW-2086.
-
   Resolution: Fixed
Fix Version/s: 2.0.0

Issue resolved by pull request #3279
[https://github.com/apache/incubator-airflow/pull/3279]

> The tree view page is too slow when display big dag.
> 
>
> Key: AIRFLOW-2086
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2086
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: webserver
>Reporter: Lintao LI
>Priority: Major
> Fix For: 2.0.0
>
>
> The tree view page is too slow for big(actually not too big) dag. 
> The page size will increase dramatically to hundreds of MB.
> please refer to 
> [here|https://stackoverflow.com/questions/48656221/apache-airflow-webui-tree-view-is-too-slow]
>  for details.
> I think the page contains a lot of redundant data. it's a bug or a flaw of 
> design.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-2086) The tree view page is too slow when display big dag.

2018-05-09 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-2086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468983#comment-16468983
 ] 

ASF subversion and git services commented on AIRFLOW-2086:
--

Commit 2a55ffe0cd2c04fde2eecdbeff2fa727067069e1 in incubator-airflow's branch 
refs/heads/master from Tao feng
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=2a55ffe ]

[AIRFLOW-2086][AIRFLOW-2393] Customize default dagrun number in tree view

Closes #3279 from feng-tao/reduce-tree-view

This introduces a new configuration variable to set the default
number of dag runs displayed in the tree view. For large DAGs, this
could cause timeouts in the webserver.


> The tree view page is too slow when display big dag.
> 
>
> Key: AIRFLOW-2086
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2086
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: webserver
>Reporter: Lintao LI
>Priority: Major
> Fix For: 2.0.0
>
>
> The tree view page is too slow for big(actually not too big) dag. 
> The page size will increase dramatically to hundreds of MB.
> please refer to 
> [here|https://stackoverflow.com/questions/48656221/apache-airflow-webui-tree-view-is-too-slow]
>  for details.
> I think the page contains a lot of redundant data. it's a bug or a flaw of 
> design.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

incubator-airflow git commit: [AIRFLOW-2086][AIRFLOW-2393] Customize default dagrun number in tree view

2018-05-09 Thread arthur 
Repository: incubator-airflow
Updated Branches:
  refs/heads/master 2728138f1 -> 2a55ffe0c


[AIRFLOW-2086][AIRFLOW-2393] Customize default dagrun number in tree view

Closes #3279 from feng-tao/reduce-tree-view

This introduces a new configuration variable to set the default
number of dag runs displayed in the tree view. For large DAGs, this
could cause timeouts in the webserver.


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/2a55ffe0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/2a55ffe0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/2a55ffe0

Branch: refs/heads/master
Commit: 2a55ffe0cd2c04fde2eecdbeff2fa727067069e1
Parents: 2728138
Author: Tao feng 
Authored: Wed May 9 08:45:06 2018 -0700
Committer: Arthur Wiedmer 
Committed: Wed May 9 08:45:17 2018 -0700

--
 UPDATING.md  |  3 +++
 airflow/config_templates/default_airflow.cfg |  3 +++
 airflow/www/templates/airflow/dag.html   |  2 +-
 airflow/www/templates/airflow/dags.html  |  2 +-
 airflow/www/templates/airflow/list_dags.html |  2 +-
 airflow/www/views.py | 19 ---
 airflow/www_rbac/templates/airflow/dag.html  |  2 +-
 airflow/www_rbac/templates/airflow/dags.html |  2 +-
 airflow/www_rbac/views.py| 14 +-
 9 files changed, 32 insertions(+), 17 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/2a55ffe0/UPDATING.md
--
diff --git a/UPDATING.md b/UPDATING.md
index defd95b..c9e1395 100644
--- a/UPDATING.md
+++ b/UPDATING.md
@@ -5,6 +5,9 @@ assists users migrating to a new version.
 
 ## Airflow Master
 
+### Add a configuration variable(default_dag_run_display_number) to control 
numbers of dag run for display
+Add a configuration variable(default_dag_run_display_number) under webserver 
section to control num of dag run to show in UI.
+
 ### Default executor for SubDagOperator is changed to SequentialExecutor
 
 ### New Webserver UI with Role-Based Access Control

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/2a55ffe0/airflow/config_templates/default_airflow.cfg
--
diff --git a/airflow/config_templates/default_airflow.cfg 
b/airflow/config_templates/default_airflow.cfg
index b91961e..33b99ff 100644
--- a/airflow/config_templates/default_airflow.cfg
+++ b/airflow/config_templates/default_airflow.cfg
@@ -280,6 +280,9 @@ rbac = False
 # Define the color of navigation bar
 navbar_color = #007A87
 
+# Default dagrun to show in UI
+default_dag_run_display_number = 25
+
 
 [email]
 email_backend = airflow.utils.email.send_email_smtp

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/2a55ffe0/airflow/www/templates/airflow/dag.html
--
diff --git a/airflow/www/templates/airflow/dag.html 
b/airflow/www/templates/airflow/dag.html
index ed84f27..18242d3 100644
--- a/airflow/www/templates/airflow/dag.html
+++ b/airflow/www/templates/airflow/dag.html
@@ -57,7 +57,7 @@
   
   
 Graph View
-  
+  
   
 Tree View
   

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/2a55ffe0/airflow/www/templates/airflow/dags.html
--
diff --git a/airflow/www/templates/airflow/dags.html 
b/airflow/www/templates/airflow/dags.html
index d22bfb3..2397890 100644
--- a/airflow/www/templates/airflow/dags.html
+++ b/airflow/www/templates/airflow/dags.html
@@ -145,7 +145,7 @@
 
 
 
-
+
 
 
 

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/2a55ffe0/airflow/www/templates/airflow/list_dags.html
--
diff --git a/airflow/www/templates/airflow/list_dags.html 
b/airflow/www/templates/airflow/list_dags.html
index e8533d7..c7f2497 100644
--- a/airflow/www/templates/airflow/list_dags.html
+++ b/airflow/www/templates/airflow/list_dags.html
@@ -147,7 +147,7 @@
 
 
 
-  
+  
   
   
   

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/2a55ffe0/airflow/www/views.py
--
diff --git a/airflow/www/views.py b/airflow/www/views.py
index 6e2f1fc..c36d55f 100644
--- a/airflow/www/views.py
+++ b/airflow/www/views.py
@@ -7,9 +7,9 @@
 # to you under the Apache

[jira] [Created] (AIRFLOW-2443) Fix doc links to pythonhosted

2018-05-09 Thread Anand (JIRA) 
Anand created AIRFLOW-2443:
--

 Summary: Fix doc links to pythonhosted
 Key: AIRFLOW-2443
 URL: https://issues.apache.org/jira/browse/AIRFLOW-2443
 Project: Apache Airflow
  Issue Type: Bug
  Components: ui
Affects Versions: 1.9.0
Reporter: Anand
 Fix For: Airflow 1.9.0


Airflow 1.9 UI still  redirects to incorrect documentation link on 
pythonhosted.org, though seems to have been fixed in master in this bug - 
https://issues.apache.org/jira/browse/AIRFLOW-2115 . 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-2250) execution_date logic seems broken in 1.9

2018-05-09 Thread sulphide (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

sulphide updated AIRFLOW-2250:
--
Description: 
after upgrading to 1.9 new dags get execution_date's that are not aligned with 
our hardcoded start_date. 



 among other things, this breaks our sensors...

[2018-03-23 20:06:21,781] \{base_task_runner.py:98} INFO - Subtask: [2018-03-23 
20:06:21,780] \{sensors.py:243} INFO - Poking for 
upstream_2017-12-06_2000.upstream_task on 2018-03-22T19:51:52.050448 
 
in this case our ExternalTaskSensor has a start_date with a start_time of 2100 
and is pointing at the upstream dag and task with an execution_delta of 1 hour. 

THE PROBLEM: the dagrun gets an execution_date of 2018-03-022T19:51:52.050448 
but should be getting an execution_date of 2018-03-22T21:00:00

it seems that if we manually set the execution date on the first dag run then 
every subsequent dag run has the right execution date (so far), so this is a 
problem with generating an execution_date for a dag run with no history within 
the schedule interval.

  was:
after upgrading to 1.9 some of our dags get execution_date's that are not 
aligned with our hardcoded start_date. this breaks our sensors, e.g.

 

[2018-03-23 20:06:21,781] \{base_task_runner.py:98} INFO - Subtask: [2018-03-23 
20:06:21,780] \{sensors.py:243} INFO - Poking for 
upstream_2017-12-06_2000.upstream_task on 2018-03-22T19:51:52.050448 

 
in this case our ExternalTaskSensor has a start_date with a start_time of 2100 
and is pointing at the upstream dag and task with an execution_delta of 1 hour. 

the problem: the dagrun gets an execution_date of 2018-03-022T19:51:52.050448 
but should be getting an execution_date of 2018-03-22T21:00:00

it seems that if we manually set the execution date on the first dag run then 
every subsequent dag run has the right execution date (so far).


> execution_date logic seems broken in 1.9
> 
>
> Key: AIRFLOW-2250
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2250
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: sulphide
>Priority: Major
> Attachments: Screen Shot 2018-03-23 at 4.47.03 PM.png
>
>
> after upgrading to 1.9 new dags get execution_date's that are not aligned 
> with our hardcoded start_date. 
>  among other things, this breaks our sensors...
> [2018-03-23 20:06:21,781] \{base_task_runner.py:98} INFO - Subtask: 
> [2018-03-23 20:06:21,780] \{sensors.py:243} INFO - Poking for 
> upstream_2017-12-06_2000.upstream_task on 2018-03-22T19:51:52.050448 
>  
> in this case our ExternalTaskSensor has a start_date with a start_time of 
> 2100 and is pointing at the upstream dag and task with an execution_delta of 
> 1 hour. 
> THE PROBLEM: the dagrun gets an execution_date of 2018-03-022T19:51:52.050448 
> but should be getting an execution_date of 2018-03-22T21:00:00
> it seems that if we manually set the execution date on the first dag run then 
> every subsequent dag run has the right execution date (so far), so this is a 
> problem with generating an execution_date for a dag run with no history 
> within the schedule interval.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-2250) execution_date logic seems broken in 1.9

2018-05-09 Thread sulphide (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

sulphide updated AIRFLOW-2250:
--
Description: 
after upgrading to 1.9 some of our dags get execution_date's that are not 
aligned with our hardcoded start_date. this breaks our sensors, e.g.

 

[2018-03-23 20:06:21,781] \{base_task_runner.py:98} INFO - Subtask: [2018-03-23 
20:06:21,780] \{sensors.py:243} INFO - Poking for 
upstream_2017-12-06_2000.upstream_task on 2018-03-22T19:51:52.050448 

 
in this case our ExternalTaskSensor has a start_date with a start_time of 2100 
and is pointing at the upstream dag and task with an execution_delta of 1 hour. 

the problem: the dagrun gets an execution_date of 2018-03-022T19:51:52.050448 
but should be getting an execution_date of 2018-03-22T21:00:00

it seems that if we manually set the execution date on the first dag run then 
every subsequent dag run has the right execution date (so far).

  was:
after upgrading to 1.9 some of our dags get execution_date's that are not 
aligned with our hardcoded start_date. this breaks our sensors, e.g.

 

[2018-03-23 20:06:21,781] \{base_task_runner.py:98} INFO - Subtask: [2018-03-23 
20:06:21,780] \{sensors.py:243} INFO - Poking for 
upstream_2017-12-06_2000.upstream_task on 2018-03-22T19:51:52.050448 

 
in this case our ExternalTaskSensor has a start_date with a start_time of 2100 
and is pointing at the upstream dag and task with an execution_delta of 1 hour. 

the problem: the task_instance gets an execution_date of 
2018-03-022T19:51:52.050448 but should be getting an execution_date of 
2018-03-22T21:00:00

it seems that if we manually set the execution date on the first dag run then 
every subsequent dag run has the right execution date (so far).


> execution_date logic seems broken in 1.9
> 
>
> Key: AIRFLOW-2250
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2250
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: sulphide
>Priority: Major
> Attachments: Screen Shot 2018-03-23 at 4.47.03 PM.png
>
>
> after upgrading to 1.9 some of our dags get execution_date's that are not 
> aligned with our hardcoded start_date. this breaks our sensors, e.g.
>  
> [2018-03-23 20:06:21,781] \{base_task_runner.py:98} INFO - Subtask: 
> [2018-03-23 20:06:21,780] \{sensors.py:243} INFO - Poking for 
> upstream_2017-12-06_2000.upstream_task on 2018-03-22T19:51:52.050448 
>  
> in this case our ExternalTaskSensor has a start_date with a start_time of 
> 2100 and is pointing at the upstream dag and task with an execution_delta of 
> 1 hour. 
> the problem: the dagrun gets an execution_date of 2018-03-022T19:51:52.050448 
> but should be getting an execution_date of 2018-03-22T21:00:00
> it seems that if we manually set the execution date on the first dag run then 
> every subsequent dag run has the right execution date (so far).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-2250) execution_date logic seems broken in 1.9

2018-05-09 Thread sulphide (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-2250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468913#comment-16468913
 ] 

sulphide commented on AIRFLOW-2250:
---

we've noticed that even if we 'fix' a dag with the above workaround if we have 
a dag with catchup=False and more than a day has passed since the dag was 
turned on then the dag will create the broken execution dates when we turn it 
on. so this bug is related to the dag's ability to create its first execution 
date.

> execution_date logic seems broken in 1.9
> 
>
> Key: AIRFLOW-2250
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2250
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: sulphide
>Priority: Major
> Attachments: Screen Shot 2018-03-23 at 4.47.03 PM.png
>
>
> after upgrading to 1.9 some of our dags get execution_date's that are not 
> aligned with our hardcoded start_date. this breaks our sensors, e.g.
>  
> [2018-03-23 20:06:21,781] \{base_task_runner.py:98} INFO - Subtask: 
> [2018-03-23 20:06:21,780] \{sensors.py:243} INFO - Poking for 
> upstream_2017-12-06_2000.upstream_task on 2018-03-22T19:51:52.050448 
>  
> in this case our ExternalTaskSensor has a start_date with a start_time of 
> 2100 and is pointing at the upstream dag and task with an execution_delta of 
> 1 hour. 
> the problem: the task_instance gets an execution_date of 
> 2018-03-022T19:51:52.050448 but should be getting an execution_date of 
> 2018-03-22T21:00:00
> it seems that if we manually set the execution date on the first dag run then 
> every subsequent dag run has the right execution date (so far).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)