[jira] [Created] (AIRFLOW-1660) Change webpage width to full-width

[Eric Luo (JIRA)]

Eric Luo created AIRFLOW-1660:
-

 Summary: Change webpage width to full-width
 Key: AIRFLOW-1660
 URL: https://issues.apache.org/jira/browse/AIRFLOW-1660
 Project: Apache Airflow
  Issue Type: Improvement
Reporter: Eric Luo


Change webpage width to full-width



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

incubator-airflow git commit: [AIRFLOW-1659] Fix invalid obj attribute bug in file_task_handler.py

[criccomini]

Repository: incubator-airflow
Updated Branches:
  refs/heads/v1-9-test b5d1657e3 -> bee823c6c


[AIRFLOW-1659] Fix invalid obj attribute bug in file_task_handler.py

Closes #2645 from jgao54/fix-file_task_handler-bug


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/bee823c6
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/bee823c6
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/bee823c6

Branch: refs/heads/v1-9-test
Commit: bee823c6c6edc48d862324b5cee472b58e344c5f
Parents: b5d1657
Author: Joy Gao 
Authored: Thu Sep 28 15:53:19 2017 -0700
Committer: Chris Riccomini 
Committed: Thu Sep 28 15:54:25 2017 -0700

--
 airflow/utils/log/file_task_handler.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/bee823c6/airflow/utils/log/file_task_handler.py
--
diff --git a/airflow/utils/log/file_task_handler.py 
b/airflow/utils/log/file_task_handler.py
index b31c968..6038fbf 100644
--- a/airflow/utils/log/file_task_handler.py
+++ b/airflow/utils/log/file_task_handler.py
@@ -116,7 +116,7 @@ class FileTaskHandler(logging.Handler):
 except (AirflowConfigException, ValueError):
 pass
 
-response = requests.get(url, timeout=self.timeout)
+response = requests.get(url, timeout=timeout)
 
 # Check if the resource was properly fetched
 response.raise_for_status()

[jira] [Commented] (AIRFLOW-1659) Fix invalid attribute bug in FileTaskHandler

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16185055#comment-16185055
 ] 

ASF subversion and git services commented on AIRFLOW-1659:
--

Commit a1b2c0feb01f0891331500e94bb3d08fee1947f8 in incubator-airflow's branch 
refs/heads/master from [~joy.gao54]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=a1b2c0f ]

[AIRFLOW-1659] Fix invalid obj attribute bug in file_task_handler.py

Closes #2645 from jgao54/fix-file_task_handler-bug


> Fix invalid attribute bug in FileTaskHandler
> 
>
> Key: AIRFLOW-1659
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1659
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: logging
>Reporter: Joy Gao
>Assignee: Joy Gao
> Fix For: 1.9.0
>
>
> The following line of code is failing in FileTaskHandler
> {code}
> response = requests.get(url, timeout=self.timeout)
> {code}
> self.timeout is not a valid attribute, should use local variable `timeout`



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (AIRFLOW-1659) Fix invalid attribute bug in FileTaskHandler

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16185059#comment-16185059
 ] 

ASF subversion and git services commented on AIRFLOW-1659:
--

Commit bee823c6c6edc48d862324b5cee472b58e344c5f in incubator-airflow's branch 
refs/heads/v1-9-test from [~joy.gao54]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=bee823c ]

[AIRFLOW-1659] Fix invalid obj attribute bug in file_task_handler.py

Closes #2645 from jgao54/fix-file_task_handler-bug


> Fix invalid attribute bug in FileTaskHandler
> 
>
> Key: AIRFLOW-1659
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1659
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: logging
>Reporter: Joy Gao
>Assignee: Joy Gao
> Fix For: 1.9.0
>
>
> The following line of code is failing in FileTaskHandler
> {code}
> response = requests.get(url, timeout=self.timeout)
> {code}
> self.timeout is not a valid attribute, should use local variable `timeout`



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

incubator-airflow git commit: [AIRFLOW-1659] Fix invalid obj attribute bug in file_task_handler.py

[criccomini]

Repository: incubator-airflow
Updated Branches:
  refs/heads/master b3e985a31 -> a1b2c0feb


[AIRFLOW-1659] Fix invalid obj attribute bug in file_task_handler.py

Closes #2645 from jgao54/fix-file_task_handler-bug


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/a1b2c0fe
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/a1b2c0fe
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/a1b2c0fe

Branch: refs/heads/master
Commit: a1b2c0feb01f0891331500e94bb3d08fee1947f8
Parents: b3e985a
Author: Joy Gao 
Authored: Thu Sep 28 15:53:19 2017 -0700
Committer: Chris Riccomini 
Committed: Thu Sep 28 15:53:19 2017 -0700

--
 airflow/utils/log/file_task_handler.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/a1b2c0fe/airflow/utils/log/file_task_handler.py
--
diff --git a/airflow/utils/log/file_task_handler.py 
b/airflow/utils/log/file_task_handler.py
index b31c968..6038fbf 100644
--- a/airflow/utils/log/file_task_handler.py
+++ b/airflow/utils/log/file_task_handler.py
@@ -116,7 +116,7 @@ class FileTaskHandler(logging.Handler):
 except (AirflowConfigException, ValueError):
 pass
 
-response = requests.get(url, timeout=self.timeout)
+response = requests.get(url, timeout=timeout)
 
 # Check if the resource was properly fetched
 response.raise_for_status()

[jira] [Created] (AIRFLOW-1659) Fix invalid attribute bug in FileTaskHandler

[Joy Gao (JIRA)]

Joy Gao created AIRFLOW-1659:


 Summary: Fix invalid attribute bug in FileTaskHandler
 Key: AIRFLOW-1659
 URL: https://issues.apache.org/jira/browse/AIRFLOW-1659
 Project: Apache Airflow
  Issue Type: Bug
  Components: logging
Reporter: Joy Gao
Assignee: Joy Gao
 Fix For: 1.9.0


The following line of code is failing in FileTaskHandler

{code}
response = requests.get(url, timeout=self.timeout)
{code}

self.timeout is not a valid attribute, should use local variable `timeout`



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (AIRFLOW-1196) Make trigger_dag_id a templated field of TriggerDagRunOperator

[Arthur Vigil (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arthur Vigil updated AIRFLOW-1196:
--
Affects Version/s: (was: Airflow 1.8)
   Airflow 2.0

> Make trigger_dag_id a templated field of TriggerDagRunOperator
> --
>
> Key: AIRFLOW-1196
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1196
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: operators
>Reporter: Arthur Vigil
>Assignee: Arthur Vigil
>Priority: Trivial
>  Labels: easyfix, improvement
>
> TriggerDagRunOperator currently has no templated fields. Adding 
> `trigger_dag_id` as a templated field should be a trivial change that 
> improves its flexibility and usefulness.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (AIRFLOW-1196) Make trigger_dag_id a templated field of TriggerDagRunOperator

[Arthur Vigil (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arthur Vigil updated AIRFLOW-1196:
--
Affects Version/s: (was: Airflow 2.0)

> Make trigger_dag_id a templated field of TriggerDagRunOperator
> --
>
> Key: AIRFLOW-1196
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1196
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: operators
>Reporter: Arthur Vigil
>Assignee: Arthur Vigil
>Priority: Trivial
>  Labels: easyfix, improvement
>
> TriggerDagRunOperator currently has no templated fields. Adding 
> `trigger_dag_id` as a templated field should be a trivial change that 
> improves its flexibility and usefulness.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (AIRFLOW-1658) Kill (possibly) still running Druid indexing job after max timeout is exceeded

[Daniel van der Ende (JIRA)]

Daniel van der Ende created AIRFLOW-1658:


 Summary: Kill (possibly) still running Druid indexing job after 
max timeout is exceeded
 Key: AIRFLOW-1658
 URL: https://issues.apache.org/jira/browse/AIRFLOW-1658
 Project: Apache Airflow
  Issue Type: Improvement
  Components: hooks
Reporter: Daniel van der Ende
Assignee: Daniel van der Ende
Priority: Minor


Right now, the Druid hook contains a parameter max_ingestion_time. If the total 
execution time of the Druid indexing job exceeds this timeout, an 
AirflowException is thrown. However, this does not necessarily mean that the 
Druid task failed (a busy Hadoop cluster could also be to blame for slow 
performance for example). If the Airflow task is then retried, you end up with 
multiple Druid tasks performing the same work. 
To easily prevent this, we can call the shutdown endpoint on the task id that 
is still running.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (AIRFLOW-1657) Handle failure of Qubole Operator for s3distcp hadoop command

[Rupesh Bansal (JIRA)]

Rupesh Bansal created AIRFLOW-1657:
--

 Summary: Handle failure of Qubole Operator for s3distcp hadoop 
command
 Key: AIRFLOW-1657
 URL: https://issues.apache.org/jira/browse/AIRFLOW-1657
 Project: Apache Airflow
  Issue Type: Bug
Reporter: Rupesh Bansal
Assignee: Rupesh Bansal






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (AIRFLOW-988) SLA Miss Callbacks Are Repeated if Email is Not being Used

[Chris Riccomini (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Riccomini updated AIRFLOW-988:

Fix Version/s: 1.9.0

> SLA Miss Callbacks Are Repeated if Email is Not being Used
> --
>
> Key: AIRFLOW-988
> URL: https://issues.apache.org/jira/browse/AIRFLOW-988
> Project: Apache Airflow
>  Issue Type: Bug
>Affects Versions: Airflow 1.8
>Reporter: Zachary Lawson
>Assignee: Charlie Jones
> Fix For: 1.9.0
>
>
> There is an issue in the current v1-8-stable branch. Looking at the jobs.py 
> module, if the system does not have email set up but does have a 
> sla_miss_callback defined in the DAG, that sla_miss_callback is repeated for 
> that job infinitely as long as the airflow scheduler is running. The 
> offending code seems to be in the query to the airflow meta database which 
> filters to sla_miss records that have *either* email_sent or 
> notification_sent as false ([see lines 
> 606-613|https://github.com/apache/incubator-airflow/blob/v1-8-stable/airflow/jobs.py#L606-L613]),
>  but then executes the sla_miss_callback function regardless if 
> notification_sent was true ([see lines 
> 644-648|https://github.com/apache/incubator-airflow/blob/v1-8-stable/airflow/jobs.py#L644-L648]).
>  A conditional statement should be put prior to executing the 
> sla_miss_callback to check whether a notification has been sent to prevent 
> this.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (AIRFLOW-1635) Allow creating Google Cloud Platform connection without requiring a JSON file

[Chris Riccomini (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Riccomini resolved AIRFLOW-1635.
--
Resolution: Fixed

> Allow creating Google Cloud Platform connection without requiring a JSON file
> -
>
> Key: AIRFLOW-1635
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1635
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: gcp
>Affects Versions: 1.8.2
>Reporter: Barry Hart
>Assignee: Barry Hart
> Fix For: 1.10.0
>
> Attachments: AIRFLOW-1635.png
>
>
> Most connection types can be created purely from the Airflow UI. Google Cloud 
> Platform connections do not support this, because they require a JSON file to 
> be present on disk. This is awkward for users who only have UI access (i.e. 
> no direct access to the server's file system) because it requires 
> coordination with a system administrator to add or update a connection.
> I propose that Airflow offer two ways to set up a Google cloud connection:
> * The current method of placing a file on disk and entering its path.
> * New method where the Airflow user/administrator pastes the JSON *contents* 
> into the Airflow UI. This will be a new field in the UI.
> If both a path and JSON data are provided, the path will take precedence. 
> This is somewhat arbitrary; typically only one field would contain a value.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

incubator-airflow git commit: [AIRFLOW-1635] Allow creating GCP connection without requiring a JSON file

[criccomini]

Repository: incubator-airflow
Updated Branches:
  refs/heads/master a87ced4c1 -> b3e985a31


[AIRFLOW-1635] Allow creating GCP connection without requiring a JSON file

Closes #2640 from barrywhart/airflow-1635-gcp-
json-data-master


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/b3e985a3
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/b3e985a3
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/b3e985a3

Branch: refs/heads/master
Commit: b3e985a3146272ecfd3ceaaa0d8567e4e9e117d4
Parents: a87ced4
Author: Barry Hart 
Authored: Thu Sep 28 09:19:43 2017 -0700
Committer: Chris Riccomini 
Committed: Thu Sep 28 09:19:43 2017 -0700

--
 airflow/contrib/hooks/gcp_api_base_hook.py | 27 +++--
 airflow/www/views.py   |  2 ++
 2 files changed, 27 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/b3e985a3/airflow/contrib/hooks/gcp_api_base_hook.py
--
diff --git a/airflow/contrib/hooks/gcp_api_base_hook.py 
b/airflow/contrib/hooks/gcp_api_base_hook.py
index 28721d3..e6ca240 100644
--- a/airflow/contrib/hooks/gcp_api_base_hook.py
+++ b/airflow/contrib/hooks/gcp_api_base_hook.py
@@ -12,6 +12,8 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 #
+import json
+
 import httplib2
 from oauth2client.client import GoogleCredentials
 from oauth2client.service_account import ServiceAccountCredentials
@@ -59,20 +61,23 @@ class GoogleCloudBaseHook(BaseHook, LoggingMixin):
 Returns the Credentials object for Google API
 """
 key_path = self._get_field('key_path', False)
+keyfile_dict = self._get_field('keyfile_dict', False)
 scope = self._get_field('scope', False)
 
 kwargs = {}
 if self.delegate_to:
 kwargs['sub'] = self.delegate_to
 
-if not key_path:
+if not key_path and not keyfile_dict:
 self.log.info('Getting connection using `gcloud auth` user, since 
no key file '
  'is defined for hook.')
 credentials = GoogleCredentials.get_application_default()
-else:
+elif key_path:
 if not scope:
 raise AirflowException('Scope should be defined when using a 
key file.')
 scopes = [s.strip() for s in scope.split(',')]
+
+# Get credentials from a JSON file.
 if key_path.endswith('.json'):
 self.log.info('Getting connection using a JSON key file.')
 credentials = ServiceAccountCredentials\
@@ -82,6 +87,24 @@ class GoogleCloudBaseHook(BaseHook, LoggingMixin):
'use a JSON key file.')
 else:
 raise AirflowException('Unrecognised extension for key file.')
+else:
+if not scope:
+raise AirflowException('Scope should be defined when using key 
JSON.')
+scopes = [s.strip() for s in scope.split(',')]
+
+# Get credentials from JSON data provided in the UI.
+try:
+keyfile_dict = json.loads(keyfile_dict)
+
+# Depending on how the JSON was formatted, it may contain
+# escaped newlines. Convert those to actual newlines.
+keyfile_dict['private_key'] = 
keyfile_dict['private_key'].replace(
+'\\n', '\n')
+
+credentials = ServiceAccountCredentials\
+.from_json_keyfile_dict(keyfile_dict, scopes)
+except json.decoder.JSONDecodeError:
+raise AirflowException('Invalid key JSON.')
 return credentials
 
 def _get_access_token(self):

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/b3e985a3/airflow/www/views.py
--
diff --git a/airflow/www/views.py b/airflow/www/views.py
index a0a0999..ad27238 100644
--- a/airflow/www/views.py
+++ b/airflow/www/views.py
@@ -2582,6 +2582,7 @@ class ConnectionModelView(wwwutils.SuperUserMixin, 
AirflowModelView):
 'extra__jdbc__drv_clsname',
 'extra__google_cloud_platform__project',
 'extra__google_cloud_platform__key_path',
+'extra__google_cloud_platform__keyfile_dict',
 'extra__google_cloud_platform__scope',
 )
 verbose_name = "Connection"
@@ -2603,6 +2604,7 @@ class ConnectionModelView(wwwutils.SuperUserMixin, 
AirflowModelView):
 'extra__jdbc__drv_clsname': StringField('Driver Class'),
 'extra__google_cloud_platform__project': 

[jira] [Commented] (AIRFLOW-1635) Allow creating Google Cloud Platform connection without requiring a JSON file

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16184387#comment-16184387
 ] 

ASF subversion and git services commented on AIRFLOW-1635:
--

Commit b3e985a3146272ecfd3ceaaa0d8567e4e9e117d4 in incubator-airflow's branch 
refs/heads/master from [~barrywhart]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=b3e985a ]

[AIRFLOW-1635] Allow creating GCP connection without requiring a JSON file

Closes #2640 from barrywhart/airflow-1635-gcp-
json-data-master


> Allow creating Google Cloud Platform connection without requiring a JSON file
> -
>
> Key: AIRFLOW-1635
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1635
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: gcp
>Affects Versions: 1.8.2
>Reporter: Barry Hart
>Assignee: Barry Hart
> Fix For: 1.10.0
>
> Attachments: AIRFLOW-1635.png
>
>
> Most connection types can be created purely from the Airflow UI. Google Cloud 
> Platform connections do not support this, because they require a JSON file to 
> be present on disk. This is awkward for users who only have UI access (i.e. 
> no direct access to the server's file system) because it requires 
> coordination with a system administrator to add or update a connection.
> I propose that Airflow offer two ways to set up a Google cloud connection:
> * The current method of placing a file on disk and entering its path.
> * New method where the Airflow user/administrator pastes the JSON *contents* 
> into the Airflow UI. This will be a new field in the UI.
> If both a path and JSON data are provided, the path will take precedence. 
> This is somewhat arbitrary; typically only one field would contain a value.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (AIRFLOW-988) SLA Miss Callbacks Are Repeated if Email is Not being Used

[Charlie Jones (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Charlie Jones reassigned AIRFLOW-988:
-

Assignee: Charlie Jones

> SLA Miss Callbacks Are Repeated if Email is Not being Used
> --
>
> Key: AIRFLOW-988
> URL: https://issues.apache.org/jira/browse/AIRFLOW-988
> Project: Apache Airflow
>  Issue Type: Bug
>Affects Versions: Airflow 1.8
>Reporter: Zachary Lawson
>Assignee: Charlie Jones
>
> There is an issue in the current v1-8-stable branch. Looking at the jobs.py 
> module, if the system does not have email set up but does have a 
> sla_miss_callback defined in the DAG, that sla_miss_callback is repeated for 
> that job infinitely as long as the airflow scheduler is running. The 
> offending code seems to be in the query to the airflow meta database which 
> filters to sla_miss records that have *either* email_sent or 
> notification_sent as false ([see lines 
> 606-613|https://github.com/apache/incubator-airflow/blob/v1-8-stable/airflow/jobs.py#L606-L613]),
>  but then executes the sla_miss_callback function regardless if 
> notification_sent was true ([see lines 
> 644-648|https://github.com/apache/incubator-airflow/blob/v1-8-stable/airflow/jobs.py#L644-L648]).
>  A conditional statement should be put prior to executing the 
> sla_miss_callback to check whether a notification has been sent to prevent 
> this.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (AIRFLOW-1656) UI - Can't set base date/number of runs in DAG tree view when DAG has no schedule

[JIRA]

Michał Ciepłucha created AIRFLOW-1656:
-

 Summary: UI - Can't set base date/number of runs in  DAG tree view 
when DAG has no schedule
 Key: AIRFLOW-1656
 URL: https://issues.apache.org/jira/browse/AIRFLOW-1656
 Project: Apache Airflow
  Issue Type: Bug
Reporter: Michał Ciepłucha


For DAGs without schedule we see all historical runs on DAG Tree view. We can't 
set base date nor limit number of runs to be displayed. That makes this view 
unusable.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

incubator-airflow git commit: [AIRFLOW-1650] Fix custom celery config loading

[bolke]

Repository: incubator-airflow
Updated Branches:
  refs/heads/v1-9-test b1e5c6ede -> b5d1657e3


[AIRFLOW-1650] Fix custom celery config loading

Celery config loading was broken as it was just passing
a string. This fixes it by loading it as a module with an
attribute. Inspired by Django's module loading.

(cherry picked from commit 3aa05cb227a12dd8ec3f375b8444adad67a2718d)
Signed-off-by: Bolke de Bruin 


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/b5d1657e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/b5d1657e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/b5d1657e

Branch: refs/heads/v1-9-test
Commit: b5d1657e3540114288d0d8c65e4ce9ca00fba360
Parents: b1e5c6e
Author: Bolke de Bruin 
Authored: Wed Sep 27 20:03:56 2017 +0200
Committer: Bolke de Bruin 
Committed: Thu Sep 28 15:48:48 2017 +0200

--
 airflow/executors/celery_executor.py |  6 --
 airflow/utils/module_loading.py  | 35 +++
 tests/utils/test_module_loading.py   | 30 ++
 3 files changed, 69 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/b5d1657e/airflow/executors/celery_executor.py
--
diff --git a/airflow/executors/celery_executor.py 
b/airflow/executors/celery_executor.py
index 7e363db..d3809b3 100644
--- a/airflow/executors/celery_executor.py
+++ b/airflow/executors/celery_executor.py
@@ -23,7 +23,7 @@ from airflow.exceptions import AirflowException
 from airflow.executors.base_executor import BaseExecutor
 from airflow import configuration
 from airflow.utils.log.logging_mixin import LoggingMixin
-
+from airflow.utils.module_loading import import_string
 
 PARALLELISM = configuration.get('core', 'PARALLELISM')
 
@@ -33,7 +33,9 @@ airflow worker
 '''
 
 if configuration.has_option('celery', 'celery_config_options'):
-celery_configuration = configuration.get('celery', 'celery_config_options')
+celery_configuration = import_string(
+configuration.get('celery', 'celery_config_options')
+)
 else:
 celery_configuration = DEFAULT_CELERY_CONFIG
 

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/b5d1657e/airflow/utils/module_loading.py
--
diff --git a/airflow/utils/module_loading.py b/airflow/utils/module_loading.py
new file mode 100644
index 000..f2138ea
--- /dev/null
+++ b/airflow/utils/module_loading.py
@@ -0,0 +1,35 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from importlib import import_module
+
+
+def import_string(dotted_path):
+"""
+Import a dotted module path and return the attribute/class designated by 
the
+last name in the path. Raise ImportError if the import failed.
+"""
+try:
+module_path, class_name = dotted_path.rsplit('.', 1)
+except ValueError:
+raise ImportError("{} doesn't look like a module 
path".format(dotted_path))
+
+module = import_module(module_path)
+
+try:
+return getattr(module, class_name)
+except AttributeError as err:
+raise ImportError('Module "{}" does not define a "{}" 
attribute/class'.format(
+module_path, class_name)
+)

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/b5d1657e/tests/utils/test_module_loading.py
--
diff --git a/tests/utils/test_module_loading.py 
b/tests/utils/test_module_loading.py
new file mode 100644
index 000..2eafb1f
--- /dev/null
+++ b/tests/utils/test_module_loading.py
@@ -0,0 +1,30 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the 

[jira] [Commented] (AIRFLOW-1650) Celery custom config is broken

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16184204#comment-16184204
 ] 

ASF subversion and git services commented on AIRFLOW-1650:
--

Commit b5d1657e3540114288d0d8c65e4ce9ca00fba360 in incubator-airflow's branch 
refs/heads/v1-9-test from [~bolke]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=b5d1657 ]

[AIRFLOW-1650] Fix custom celery config loading

Celery config loading was broken as it was just passing
a string. This fixes it by loading it as a module with an
attribute. Inspired by Django's module loading.

(cherry picked from commit 3aa05cb227a12dd8ec3f375b8444adad67a2718d)
Signed-off-by: Bolke de Bruin 


> Celery custom config is broken
> --
>
> Key: AIRFLOW-1650
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1650
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: celery, configuration
>Reporter: Bolke de Bruin
> Fix For: 1.9.0
>
>
> Celery custom config loading is broken as is just loads a string instead of 
> loading a config



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (AIRFLOW-1650) Celery custom config is broken

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16184199#comment-16184199
 ] 

ASF subversion and git services commented on AIRFLOW-1650:
--

Commit 3aa05cb227a12dd8ec3f375b8444adad67a2718d in incubator-airflow's branch 
refs/heads/master from [~bolke]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=3aa05cb ]

[AIRFLOW-1650] Fix custom celery config loading

Celery config loading was broken as it was just passing
a string. This fixes it by loading it as a module with an
attribute. Inspired by Django's module loading.


> Celery custom config is broken
> --
>
> Key: AIRFLOW-1650
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1650
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: celery, configuration
>Reporter: Bolke de Bruin
> Fix For: 1.9.0
>
>
> Celery custom config loading is broken as is just loads a string instead of 
> loading a config



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[1/2] incubator-airflow git commit: [AIRFLOW-1650] Fix custom celery config loading

[bolke]

Repository: incubator-airflow
Updated Branches:
  refs/heads/master 6a9dc8ad1 -> a87ced4c1


[AIRFLOW-1650] Fix custom celery config loading

Celery config loading was broken as it was just passing
a string. This fixes it by loading it as a module with an
attribute. Inspired by Django's module loading.


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/3aa05cb2
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/3aa05cb2
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/3aa05cb2

Branch: refs/heads/master
Commit: 3aa05cb227a12dd8ec3f375b8444adad67a2718d
Parents: dca9ab3
Author: Bolke de Bruin 
Authored: Wed Sep 27 20:03:56 2017 +0200
Committer: Bolke de Bruin 
Committed: Wed Sep 27 20:03:56 2017 +0200

--
 airflow/executors/celery_executor.py |  6 --
 airflow/utils/module_loading.py  | 35 +++
 tests/utils/test_module_loading.py   | 30 ++
 3 files changed, 69 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/3aa05cb2/airflow/executors/celery_executor.py
--
diff --git a/airflow/executors/celery_executor.py 
b/airflow/executors/celery_executor.py
index 7e363db..d3809b3 100644
--- a/airflow/executors/celery_executor.py
+++ b/airflow/executors/celery_executor.py
@@ -23,7 +23,7 @@ from airflow.exceptions import AirflowException
 from airflow.executors.base_executor import BaseExecutor
 from airflow import configuration
 from airflow.utils.log.logging_mixin import LoggingMixin
-
+from airflow.utils.module_loading import import_string
 
 PARALLELISM = configuration.get('core', 'PARALLELISM')
 
@@ -33,7 +33,9 @@ airflow worker
 '''
 
 if configuration.has_option('celery', 'celery_config_options'):
-celery_configuration = configuration.get('celery', 'celery_config_options')
+celery_configuration = import_string(
+configuration.get('celery', 'celery_config_options')
+)
 else:
 celery_configuration = DEFAULT_CELERY_CONFIG
 

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/3aa05cb2/airflow/utils/module_loading.py
--
diff --git a/airflow/utils/module_loading.py b/airflow/utils/module_loading.py
new file mode 100644
index 000..f2138ea
--- /dev/null
+++ b/airflow/utils/module_loading.py
@@ -0,0 +1,35 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from importlib import import_module
+
+
+def import_string(dotted_path):
+"""
+Import a dotted module path and return the attribute/class designated by 
the
+last name in the path. Raise ImportError if the import failed.
+"""
+try:
+module_path, class_name = dotted_path.rsplit('.', 1)
+except ValueError:
+raise ImportError("{} doesn't look like a module 
path".format(dotted_path))
+
+module = import_module(module_path)
+
+try:
+return getattr(module, class_name)
+except AttributeError as err:
+raise ImportError('Module "{}" does not define a "{}" 
attribute/class'.format(
+module_path, class_name)
+)

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/3aa05cb2/tests/utils/test_module_loading.py
--
diff --git a/tests/utils/test_module_loading.py 
b/tests/utils/test_module_loading.py
new file mode 100644
index 000..2eafb1f
--- /dev/null
+++ b/tests/utils/test_module_loading.py
@@ -0,0 +1,30 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import unittest
+
+from 

[2/2] incubator-airflow git commit: Merge branch 'celery_loadconfig'

[bolke]

Merge branch 'celery_loadconfig'


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/a87ced4c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/a87ced4c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/a87ced4c

Branch: refs/heads/master
Commit: a87ced4c1ca84f3a144f15ca5146110904168929
Parents: 6a9dc8a 3aa05cb
Author: Bolke de Bruin 
Authored: Thu Sep 28 15:46:50 2017 +0200
Committer: Bolke de Bruin 
Committed: Thu Sep 28 15:46:50 2017 +0200

--
 airflow/executors/celery_executor.py |  6 --
 airflow/utils/module_loading.py  | 35 +++
 tests/utils/test_module_loading.py   | 30 ++
 3 files changed, 69 insertions(+), 2 deletions(-)
--

[jira] [Created] (AIRFLOW-1655) UI - DAGs not displayed on first page have no info on recent tasks/runs

[JIRA]

Michał Ciepłucha created AIRFLOW-1655:
-

 Summary: UI - DAGs not displayed on first page have no info on 
recent tasks/runs 
 Key: AIRFLOW-1655
 URL: https://issues.apache.org/jira/browse/AIRFLOW-1655
 Project: Apache Airflow
  Issue Type: Bug
Reporter: Michał Ciepłucha


When number of my DAGs has grown over 25 (first page default) I don't see 
additional information for DAGs from 2nd page (and all next). The missing info 
is: 
Recent Tasks
Last Run
DAG Runs
On/Off switch is not rendered as well (just a checkbox)





--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (AIRFLOW-1654) Show tooltips for link icons in DAGs view

[Shintaro Murakami (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shintaro Murakami updated AIRFLOW-1654:
---
Attachment: screenshot-1.png

> Show tooltips for link icons in DAGs view
> -
>
> Key: AIRFLOW-1654
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1654
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Shintaro Murakami
>Priority: Minor
> Attachments: screenshot-1.png
>
>
> It is hard to me to remember what icons mean.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (AIRFLOW-1654) Show tooltips for link icons in DAGs view

[Shintaro Murakami (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shintaro Murakami updated AIRFLOW-1654:
---
Attachment: (was: スクリーンショット 2017-09-28 17.48.18.png)

> Show tooltips for link icons in DAGs view
> -
>
> Key: AIRFLOW-1654
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1654
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Shintaro Murakami
>Priority: Minor
>
> It is hard to me to remember what icons mean.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (AIRFLOW-1654) Show tooltips for link icons in DAGs view

[Shintaro Murakami (JIRA)]

Shintaro Murakami created AIRFLOW-1654:
--

 Summary: Show tooltips for link icons in DAGs view
 Key: AIRFLOW-1654
 URL: https://issues.apache.org/jira/browse/AIRFLOW-1654
 Project: Apache Airflow
  Issue Type: Improvement
Reporter: Shintaro Murakami
Priority: Minor
 Attachments: スクリーンショット 2017-09-28 17.48.18.png

It is hard to me to remember what icons mean.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (AIRFLOW-1595) SqliteHook is broken

[Shintaro Murakami (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183854#comment-16183854
 ] 

Shintaro Murakami commented on AIRFLOW-1595:


I created PR(https://github.com/apache/incubator-airflow/pull/2598)

> SqliteHook is broken
> 
>
> Key: AIRFLOW-1595
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1595
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: hooks
>Affects Versions: 1.8.1
>Reporter: Shintaro Murakami
>
> SqliteHook is built using the host attribute of connection, but correctly we 
> should use the schema attribute. The path to the DB parsed from the URI is 
> set as schema.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)