incubator-airflow git commit: [AIRFLOW-500] Use id for github allowed teams

2016-10-08 Thread bolke 
Repository: incubator-airflow
Updated Branches:
  refs/heads/master bae8bc739 -> a66cf75e2


[AIRFLOW-500] Use id for github allowed teams

The team string is not unique across an organization
and therefore we should use the long id instead.

Closes #1788 from mylons/master


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/a66cf75e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/a66cf75e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/a66cf75e

Branch: refs/heads/master
Commit: a66cf75e239bf7ef9c6ffc49f20bf6810a7e76b6
Parents: bae8bc7
Author: Mike Lyons 
Authored: Sat Oct 8 23:27:12 2016 +0200
Committer: Bolke de Bruin 
Committed: Sat Oct 8 23:27:27 2016 +0200

--
 .../auth/backends/github_enterprise_auth.py | 24 ++--
 docs/security.rst   |  2 +-
 2 files changed, 18 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/a66cf75e/airflow/contrib/auth/backends/github_enterprise_auth.py
--
diff --git a/airflow/contrib/auth/backends/github_enterprise_auth.py 
b/airflow/contrib/auth/backends/github_enterprise_auth.py
index f9d6426..91126c7 100644
--- a/airflow/contrib/auth/backends/github_enterprise_auth.py
+++ b/airflow/contrib/auth/backends/github_enterprise_auth.py
@@ -138,13 +138,21 @@ class GHEAuthBackend(object):
 
 def ghe_team_check(self, username, ghe_token):
 try:
-teams = [team.strip()
- for team in
- get_config_param('allowed_teams').split(',')]
+# the response from ghe returns the id of the team as an integer
+try:
+allowed_teams = [int(team.strip())
+ for team in
+ get_config_param('allowed_teams').split(',')]
+except ValueError:
+# this is to deprecate using the string name for a team
+raise ValueError('it appears that you are using the string 
name for a team, '
+ 'please use the id number instead')
+
 except AirflowConfigException:
 # No allowed teams defined, let anyone in GHE in.
 return True
 
+# https://developer.github.com/v3/orgs/teams/#list-user-teams
 resp = self.ghe_oauth.get(self.ghe_api_route('/user/teams'),
   token=(ghe_token, ''))
 
@@ -154,14 +162,16 @@ class GHEAuthBackend(object):
 resp.status if resp else 'None'))
 
 for team in resp.data:
-# team json object has a slug cased team name field aptly named
-# 'slug'
-if team['slug'] in teams:
+# mylons: previously this line used to be if team['slug'] in teams
+# however, teams are part of organizations. organizations are 
unique,
+# but teams are not therefore 'slug' for a team is not necessarily 
unique.
+# use id instead
+if team['id'] in allowed_teams:
 return True
 
 _log.debug('Denying access for user "%s", not a member of "%s"',
username,
-   str(teams))
+   str(allowed_teams))
 
 return False
 

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/a66cf75e/docs/security.rst
--
diff --git a/docs/security.rst b/docs/security.rst
index 872102f..29f228d 100644
--- a/docs/security.rst
+++ b/docs/security.rst
@@ -240,7 +240,7 @@ your GHE installation will be able to login to Airflow.
 client_id = oauth_key_from_github_enterprise
 client_secret = oauth_secret_from_github_enterprise
 oauth_callback_route = /example/ghe_oauth/callback
-allowed_teams = example_team_1, example_team_2
+allowed_teams = 1, 345, 23
 
 Setting up GHE Authentication
 ^

[jira] [Updated] (AIRFLOW-558) Add Support for dag.backfill=(True|False) Option

2016-10-08 Thread Ben Tallman (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ben Tallman updated AIRFLOW-558:

Priority: Major  (was: Critical)

> Add Support for dag.backfill=(True|False) Option
> 
>
> Key: AIRFLOW-558
> URL: https://issues.apache.org/jira/browse/AIRFLOW-558
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: core
>Reporter: Ben Tallman
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AIRFLOW-558) Add Support for dag.backfill=(True|False) Option

2016-10-08 Thread Ben Tallman (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ben Tallman updated AIRFLOW-558:

Issue Type: Improvement  (was: Bug)

> Add Support for dag.backfill=(True|False) Option
> 
>
> Key: AIRFLOW-558
> URL: https://issues.apache.org/jira/browse/AIRFLOW-558
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: core
>Reporter: Ben Tallman
>Priority: Critical
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-540) Allow BigQuery operators to specify default dataset

2016-10-08 Thread Sam McVeety (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15558449#comment-15558449
 ] 

Sam McVeety commented on AIRFLOW-540:
-

Likely superseded by https://issues.apache.org/jira/browse/AIRFLOW-559.

> Allow BigQuery operators to specify default dataset
> ---
>
> Key: AIRFLOW-540
> URL: https://issues.apache.org/jira/browse/AIRFLOW-540
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: contrib
>Reporter: Sam McVeety
>Priority: Minor
>
> BigQuery allows the specification of a defaultDataset 
> (https://cloud.google.com/bigquery/docs/reference/v2/jobs/query), which can 
> simplify and modularize the SQL being used.
> Perhaps a more general solution to this (and 
> https://issues.apache.org/jira/browse/AIRFLOW-539) would be to support 
> arbitrary kwargs to the various BigQueryHook methods.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AIRFLOW-559) Add support for BigQuery kwarg parameters

2016-10-08 Thread Sam McVeety (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sam McVeety updated AIRFLOW-559:

   Assignee: (was: Ilya Rakoshes)
Description: 
Many of the operators in 
https://github.com/apache/incubator-airflow/tree/master/airflow/contrib/operators
 add parameters over time, and plumbing these through multiple layers of calls 
isn't always a high priority.

The operators (and hooks) should support an end-to-end kwargs parameter that 
allows for new fields (e.g. useLegacySql, defaultDataset) to be added by users 
without needing to change the underlying code.   

  was:Many of the operators in 
https://github.com/apache/incubator-airflow/tree/master/airflow/contrib/operators
 are implicitly using the "useLegacySql" option to BigQuery.  Providing the 
option to negate this will allow users to migrate to Standard SQL 
(https://cloud.google.com/bigquery/sql-reference/enabling-standard-sql).


> Add support for BigQuery kwarg parameters
> -
>
> Key: AIRFLOW-559
> URL: https://issues.apache.org/jira/browse/AIRFLOW-559
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: contrib
>Reporter: Sam McVeety
>Priority: Minor
> Fix For: Airflow 1.8
>
>
> Many of the operators in 
> https://github.com/apache/incubator-airflow/tree/master/airflow/contrib/operators
>  add parameters over time, and plumbing these through multiple layers of 
> calls isn't always a high priority.
> The operators (and hooks) should support an end-to-end kwargs parameter that 
> allows for new fields (e.g. useLegacySql, defaultDataset) to be added by 
> users without needing to change the underlying code.   



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (AIRFLOW-559) Add support for BigQuery kwarg parameters

2016-10-08 Thread Sam McVeety (JIRA) 
Sam McVeety created AIRFLOW-559:
---

 Summary: Add support for BigQuery kwarg parameters
 Key: AIRFLOW-559
 URL: https://issues.apache.org/jira/browse/AIRFLOW-559
 Project: Apache Airflow
  Issue Type: Improvement
  Components: contrib
Reporter: Sam McVeety
Assignee: Ilya Rakoshes
Priority: Minor
 Fix For: Airflow 1.8


Many of the operators in 
https://github.com/apache/incubator-airflow/tree/master/airflow/contrib/operators
 are implicitly using the "useLegacySql" option to BigQuery.  Providing the 
option to negate this will allow users to migrate to Standard SQL 
(https://cloud.google.com/bigquery/sql-reference/enabling-standard-sql).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-139) Executing VACUUM with PostgresOperator

2016-10-08 Thread Siddharth Anand (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15558422#comment-15558422
 ] 

Siddharth Anand commented on AIRFLOW-139:
-

FYI, I haven't had a chance to follow this very closely, but we won't be 
merging fixes that don't work for a range of postgres versions [9.3, 9.6] 
(inclusive). We might in fact support versions as far back as 9.2. Also, we 
don't use Redshift and I hear that has different versioning, so we may just 
need to break that out into a different operator unless someone has already 
done that. 

-s

> Executing VACUUM with PostgresOperator
> --
>
> Key: AIRFLOW-139
> URL: https://issues.apache.org/jira/browse/AIRFLOW-139
> Project: Apache Airflow
>  Issue Type: Bug
>Affects Versions: Airflow 1.7.0
>Reporter: Rafael
>
> Dear Airflow Maintainers,
> h1. Environment
> * Airflow version: *v1.7.0*
> * Airflow components: *PostgresOperator*
> * Python Version: *Python 3.5.1*
> * Operating System: *15.4.0 Darwin*
> h1. Description of Issue
> I am trying to execute a `VACUUM` command as part of DAG with the 
> `PostgresOperator`, which fails with the following error:
> {quote}
> [2016-05-14 16:14:01,849] {__init__.py:36} INFO - Using executor 
> SequentialExecutor
> Traceback (most recent call last):
>   File "/usr/local/bin/airflow", line 15, in 
> args.func(args)
>   File 
> "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/airflow/bin/cli.py",
>  line 203, in run
> pool=args.pool,
>   File 
> "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/airflow/models.py",
>  line 1067, in run
> result = task_copy.execute(context=context)
>   File 
> "/usr/local/lib/python3.5/site-packages/airflow/operators/postgres_operator.py",
>  line 39, in execute
> self.hook.run(self.sql, self.autocommit, parameters=self.parameters)
>   File 
> "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/airflow/hooks/dbapi_hook.py",
>  line 109, in run
> cur.execute(s)
> psycopg2.InternalError: VACUUM cannot run inside a transaction block
> {quote}
> I could create a small python script that performs the operation, as 
> explained in [this stackoverflow 
> entry](http://stackoverflow.com/questions/1017463/postgresql-how-to-run-vacuum-from-code-outside-transaction-block).
>  However, I would like to know first if the `VACUUM` command should be 
> supported by the `PostgresOperator`.
> h1. Reproducing the Issue
> The operator can be declared as follows:
> {quote}
> conn = ('postgres_default')
> t4 = PostgresOperator(
> task_id='vacuum',
> postgres_conn_id=conn,
> sql=("VACUUM public.table"),
> dag=dag
> )
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (AIRFLOW-28) Add @latest, @now, @start_date, @end_date to the `airflow test` CLI

2016-10-08 Thread Siddharth Anand (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-28?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siddharth Anand reopened AIRFLOW-28:

  Assignee: (was: Siddharth Anand)

> Add @latest, @now, @start_date, @end_date to the `airflow test` CLI
> ---
>
> Key: AIRFLOW-28
> URL: https://issues.apache.org/jira/browse/AIRFLOW-28
> Project: Apache Airflow
>  Issue Type: Wish
>Reporter: Bence Nagy
>Priority: Minor
>
> It's often quite a drag to have to calculate and type a valid datestring for 
> {{airflow test}} when I just want to see whether my code runs. It'd be really 
> nice if I could just run {{airflow test dag task @latest}} and have airflow 
> think about the execution date instead of me.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-139) Executing VACUUM with PostgresOperator

2016-10-08 Thread Bence Nagy (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15557702#comment-15557702
 ] 

Bence Nagy commented on AIRFLOW-139:


{quote}
In all honesty, I'm not sure what was the purpose of the original fix and I 
think those lines should be removed.
{quote}

Hey, author of that commit here. I don't entirely remember, but I'm absolutely 
sure that connections just did not work at all for us before we changed this 
(failing with an error message letting us know that autocommit is not supported 
anymore). I think we might have been using PostgreSQL 9.4 back then, and 
probably a now year-old version of libpq/psycopg2. So please test with those 
versions as well.

> Executing VACUUM with PostgresOperator
> --
>
> Key: AIRFLOW-139
> URL: https://issues.apache.org/jira/browse/AIRFLOW-139
> Project: Apache Airflow
>  Issue Type: Bug
>Affects Versions: Airflow 1.7.0
>Reporter: Rafael
>
> Dear Airflow Maintainers,
> h1. Environment
> * Airflow version: *v1.7.0*
> * Airflow components: *PostgresOperator*
> * Python Version: *Python 3.5.1*
> * Operating System: *15.4.0 Darwin*
> h1. Description of Issue
> I am trying to execute a `VACUUM` command as part of DAG with the 
> `PostgresOperator`, which fails with the following error:
> {quote}
> [2016-05-14 16:14:01,849] {__init__.py:36} INFO - Using executor 
> SequentialExecutor
> Traceback (most recent call last):
>   File "/usr/local/bin/airflow", line 15, in 
> args.func(args)
>   File 
> "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/airflow/bin/cli.py",
>  line 203, in run
> pool=args.pool,
>   File 
> "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/airflow/models.py",
>  line 1067, in run
> result = task_copy.execute(context=context)
>   File 
> "/usr/local/lib/python3.5/site-packages/airflow/operators/postgres_operator.py",
>  line 39, in execute
> self.hook.run(self.sql, self.autocommit, parameters=self.parameters)
>   File 
> "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/airflow/hooks/dbapi_hook.py",
>  line 109, in run
> cur.execute(s)
> psycopg2.InternalError: VACUUM cannot run inside a transaction block
> {quote}
> I could create a small python script that performs the operation, as 
> explained in [this stackoverflow 
> entry](http://stackoverflow.com/questions/1017463/postgresql-how-to-run-vacuum-from-code-outside-transaction-block).
>  However, I would like to know first if the `VACUUM` command should be 
> supported by the `PostgresOperator`.
> h1. Reproducing the Issue
> The operator can be declared as follows:
> {quote}
> conn = ('postgres_default')
> t4 = PostgresOperator(
> task_id='vacuum',
> postgres_conn_id=conn,
> sql=("VACUUM public.table"),
> dag=dag
> )
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)