[jira] [Commented] (AIRFLOW-2882) Add import and export for pool cli using JSON

2018-09-05 Thread ASF GitHub Bot (JIRA) 


[ 
https://issues.apache.org/jira/browse/AIRFLOW-2882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16604943#comment-16604943
 ] 

ASF GitHub Bot commented on AIRFLOW-2882:
-

r39132 closed pull request #3730: [AIRFLOW-2882] Add import and export for pool 
cli using JSON
URL: https://github.com/apache/incubator-airflow/pull/3730
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/airflow/bin/cli.py b/airflow/bin/cli.py
index e22427cf40..4ff1ae3679 100644
--- a/airflow/bin/cli.py
+++ b/airflow/bin/cli.py
@@ -267,6 +267,7 @@ def _tabulate(pools):
  tablefmt="fancy_grid")
 
 try:
+imp = getattr(args, 'import')
 if args.get is not None:
 pools = [api_client.get_pool(name=args.get)]
 elif args.set:
@@ -275,6 +276,14 @@ def _tabulate(pools):
 description=args.set[2])]
 elif args.delete:
 pools = [api_client.delete_pool(name=args.delete)]
+elif imp:
+if os.path.exists(imp):
+pools = pool_import_helper(imp)
+else:
+print("Missing pools file.")
+pools = api_client.get_pools()
+elif args.export:
+pools = pool_export_helper(args.export)
 else:
 pools = api_client.get_pools()
 except (AirflowException, IOError) as err:
@@ -283,6 +292,43 @@ def _tabulate(pools):
 log.info(_tabulate(pools=pools))
 
 
+def pool_import_helper(filepath):
+with open(filepath, 'r') as poolfile:
+pl = poolfile.read()
+try:
+d = json.loads(pl)
+except Exception as e:
+print("Please check the validity of the json file: " + str(e))
+else:
+try:
+pools = []
+n = 0
+for k, v in d.items():
+if isinstance(v, dict) and len(v) == 2:
+pools.append(api_client.create_pool(name=k,
+slots=v["slots"],
+
description=v["description"]))
+n += 1
+else:
+pass
+except Exception:
+pass
+finally:
+print("{} of {} pool(s) successfully updated.".format(n, len(d)))
+return pools
+
+
+def pool_export_helper(filepath):
+pool_dict = {}
+pools = api_client.get_pools()
+for pool in pools:
+pool_dict[pool[0]] = {"slots": pool[1], "description": pool[2]}
+with open(filepath, 'w') as poolfile:
+poolfile.write(json.dumps(pool_dict, sort_keys=True, indent=4))
+print("{} pools successfully exported to {}".format(len(pool_dict), 
filepath))
+return pools
+
+
 @cli_utils.action_logging
 def variables(args):
 if args.get:
@@ -1551,6 +1597,14 @@ class CLIFactory(object):
 ("-x", "--delete"),
 metavar="NAME",
 help="Delete a pool"),
+'pool_import': Arg(
+("-i", "--import"),
+metavar="FILEPATH",
+help="Import pool from JSON file"),
+'pool_export': Arg(
+("-e", "--export"),
+metavar="FILEPATH",
+help="Export pool to JSON file"),
 # variables
 'set': Arg(
 ("-s", "--set"),
@@ -1870,7 +1924,7 @@ class CLIFactory(object):
 }, {
 'func': pool,
 'help': "CRUD operations on pools",
-"args": ('pool_set', 'pool_get', 'pool_delete'),
+"args": ('pool_set', 'pool_get', 'pool_delete', 'pool_import', 
'pool_export'),
 }, {
 'func': variables,
 'help': "CRUD operations on variables",
diff --git a/tests/core.py b/tests/core.py
index f8b8691912..b59ae7048d 100644
--- a/tests/core.py
+++ b/tests/core.py
@@ -1512,6 +1512,42 @@ def test_pool_no_args(self):
 except Exception as e:
 self.fail("The 'pool' command raised unexpectedly: %s" % e)
 
+def test_pool_import_export(self):
+# Create two pools first
+pool_config_input = {
+"foo": {
+"description": "foo_test",
+"slots": 1
+},
+"baz": {
+"description": "baz_test",
+"slots": 2
+}
+}
+with open('pools_import.json', mode='w') as f:
+json.dump(pool_config_input, f)
+
+# Import json
+try:
+cli.pool(self.parser.parse_args(['pool', '-i', 
'pools_import.json']))
+except Exception as e:
+self.fail("The 'pool -i pools_import.json' failed: %s" % e)
+
+#

[jira] [Commented] (AIRFLOW-2882) Add import and export for pool cli using JSON

2018-08-09 Thread ASF GitHub Bot (JIRA) 


[ 
https://issues.apache.org/jira/browse/AIRFLOW-2882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575700#comment-16575700
 ] 

ASF GitHub Bot commented on AIRFLOW-2882:
-

r39132 closed pull request #3730: [AIRFLOW-2882] Add import and export for pool 
cli using JSON
URL: https://github.com/apache/incubator-airflow/pull/3730
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/airflow/bin/cli.py b/airflow/bin/cli.py
index 45b7903d3e..656fc60d47 100644
--- a/airflow/bin/cli.py
+++ b/airflow/bin/cli.py
@@ -267,6 +267,7 @@ def _tabulate(pools):
  tablefmt="fancy_grid")
 
 try:
+imp = getattr(args, 'import')
 if args.get is not None:
 pools = [api_client.get_pool(name=args.get)]
 elif args.set:
@@ -275,6 +276,14 @@ def _tabulate(pools):
 description=args.set[2])]
 elif args.delete:
 pools = [api_client.delete_pool(name=args.delete)]
+elif imp:
+if os.path.exists(imp):
+pools = pool_import_helper(imp)
+else:
+print("Missing pools file.")
+pools = api_client.get_pools()
+elif args.export:
+pools = pool_export_helper(args.export)
 else:
 pools = api_client.get_pools()
 except (AirflowException, IOError) as err:
@@ -283,6 +292,43 @@ def _tabulate(pools):
 log.info(_tabulate(pools=pools))
 
 
+def pool_import_helper(filepath):
+with open(filepath, 'r') as poolfile:
+pl = poolfile.read()
+try:
+d = json.loads(pl)
+except Exception:
+print("Invalid pool file.")
+else:
+try:
+pools = []
+n = 0
+for k, v in d.items():
+if isinstance(v, dict) and len(v) == 2:
+pools.append(api_client.create_pool(name=k,
+slots=v["slots"],
+
description=v["description"]))
+n += 1
+else:
+pass
+except Exception:
+pass
+finally:
+print("{} of {} pool(s) successfully updated.".format(n, len(d)))
+return pools
+
+
+def pool_export_helper(filepath):
+pool_dict = {}
+pools = api_client.get_pools()
+for pool in pools:
+pool_dict[pool[0]] = {"slots": pool[1], "description": pool[2]}
+with open(filepath, 'w') as poolfile:
+poolfile.write(json.dumps(pool_dict, sort_keys=True, indent=4))
+print("{} pools successfully exported to {}".format(len(pool_dict), 
filepath))
+return pools
+
+
 @cli_utils.action_logging
 def variables(args):
 if args.get:
@@ -1546,6 +1592,14 @@ class CLIFactory(object):
 ("-x", "--delete"),
 metavar="NAME",
 help="Delete a pool"),
+'pool_import': Arg(
+("-i", "--import"),
+metavar="FILEPATH",
+help="Import pool from JSON file"),
+'pool_export': Arg(
+("-e", "--export"),
+metavar="FILEPATH",
+help="Export pool to JSON file"),
 # variables
 'set': Arg(
 ("-s", "--set"),
@@ -1865,7 +1919,7 @@ class CLIFactory(object):
 }, {
 'func': pool,
 'help': "CRUD operations on pools",
-"args": ('pool_set', 'pool_get', 'pool_delete'),
+"args": ('pool_set', 'pool_get', 'pool_delete', 'pool_import', 
'pool_export'),
 }, {
 'func': variables,
 'help': "CRUD operations on variables",


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add import and export for pool cli using JSON
> -
>
> Key: AIRFLOW-2882
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2882
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: cli
>Reporter: Chengzhi Zhao
>Assignee: Chengzhi Zhao
>Priority: Minor
> Fix For: 1.10
>
>
> Currently, the pool only support set and get by one elements,
> `airflow pool [-h] [-s NAME SLOT_COUNT POOL_DESCRIPTION] [-g NAME] [-x NAME]`
> I'd like to add import & export as Variables so cli use JSON file to perform 
>

[jira] [Commented] (AIRFLOW-2882) Add import and export for pool cli using JSON

2018-08-09 Thread ASF GitHub Bot (JIRA) 


[ 
https://issues.apache.org/jira/browse/AIRFLOW-2882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575701#comment-16575701
 ] 

ASF GitHub Bot commented on AIRFLOW-2882:
-

ChengzhiZhao opened a new pull request #3730: [AIRFLOW-2882] Add import and 
export for pool cli using JSON
URL: https://github.com/apache/incubator-airflow/pull/3730
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [✓] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-2882
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
   
   ### Description
   
   - [✓] Here are some details about my PR, including screenshots of any UI 
changes:
   Currently, the pool only support set and get by one elements,
   `airflow pool [-h] [-s NAME SLOT_COUNT POOL_DESCRIPTION] [-g NAME] [-x NAME]`
   With this changes, user can perform import & export similar as Variables so 
cli use JSON file to perform import & export operations. This will make 
deployment easier by setting up the pools via CICD pipeline as well. 
   For example, the import json file`{"s3_pool": {"slots":5, 
"description":"This is my test s3_pool"}}`
   --for import `airflow pool -i pools.json`
   --for export `airflow pool -e pools_e.json`
   
   ### Tests
   
   - [✓] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   Using `{
   "s3_pool": {"slots":5, "description":"This is my test s3_pool"},
   "s3_pool2": {"slots":5, "description":"This is my test s3_pool"}
   }`
   
   - Tested `airflow pool -i pools.json` and use `airflow pool --get s3_pool` 
╒═╤═╤═╕
   │ Pool│   Slots │ Description │
   ╞═╪═╪═╡
   │ s3_pool │   5 │ This is my test s3_pool │
   ╘═╧═╧═╛
   
   - Tested `airflow pool -e pools_e.json` and got 
   
   ```
   {
   "s3_pool": {
   "description": "This is my test s3_pool",
   "slots": 5
   },
   "s3_pool2": {
   "description": "This is my test s3_pool",
   "slots": 5
   }
   }
   ```
   
   ### Commits
   
   - [✓ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [✓] In case of new functionality, my PR adds documentation that describes 
how to use it.
 Added named arguments:
   `airflow pool -help`
   ```
 -i FILEPATH, --import FILEPATH
   Import pool from JSON file
 -e FILEPATH, --export FILEPATH
   Export pool to JSON file
   ```
   
   ### Code Quality
   
   - [✓] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add import and export for pool cli using JSON
> -
>
> Key: AIRFLOW-2882
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2882
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: cli
>Reporter: Chengzhi Zhao
>Assignee: Chengzhi Zhao
>Priority: Minor
> Fix For: 1.10
>
>
> Currently, the pool only support set and get by one elements,
> `airflow pool [-h] [-s NAME SLOT_COUNT POOL_DESCRIPTION] [-g NAME] [-x NAME]`
> I'd like to add import & export as Variables so cli use JSON file to perform 
> import & export operations. This will make deployment easier by setting up 
> the pools via CICD pipeline as well. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-2882) Add import and export for pool cli using JSON

2018-08-09 Thread ASF GitHub Bot (JIRA) 


[ 
https://issues.apache.org/jira/browse/AIRFLOW-2882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575682#comment-16575682
 ] 

ASF GitHub Bot commented on AIRFLOW-2882:
-

ChengzhiZhao opened a new pull request #3730: [AIRFLOW-2882] Add import and 
export for pool cli using JSON
URL: https://github.com/apache/incubator-airflow/pull/3730
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [✓] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-2882
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
   
   ### Description
   
   - [✓] Here are some details about my PR, including screenshots of any UI 
changes:
   Currently, the pool only support set and get by one elements,
   `airflow pool [-h] [-s NAME SLOT_COUNT POOL_DESCRIPTION] [-g NAME] [-x NAME]`
   With this changes, user can perform import & export similar as Variables so 
cli use JSON file to perform import & export operations. This will make 
deployment easier by setting up the pools via CICD pipeline as well. 
   For example, the import json file`{"s3_pool": {"slots":5, 
"description":"This is my test s3_pool"}}`
   --for import `airflow pool -i pools.json`
   --for export `airflow pool -e pools_e.json`
   
   ### Tests
   
   - [✓] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   Using `{
   "s3_pool": {"slots":5, "description":"This is my test s3_pool"},
   "s3_pool2": {"slots":5, "description":"This is my test s3_pool"}
   }`
   
   - Tested `airflow pool -i pools.json` and use `airflow pool --get s3_pool` 
╒═╤═╤═╕
   │ Pool│   Slots │ Description │
   ╞═╪═╪═╡
   │ s3_pool │   5 │ This is my test s3_pool │
   ╘═╧═╧═╛
   
   - Tested `airflow pool -e pools_e.json` and got 
   
   ```
   {
   "s3_pool": {
   "description": "This is my test s3_pool",
   "slots": 5
   },
   "s3_pool2": {
   "description": "This is my test s3_pool",
   "slots": 5
   }
   }
   ```
   
   ### Commits
   
   - [✓ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [✓] In case of new functionality, my PR adds documentation that describes 
how to use it.
 Added named arguments:
   `airflow pool -help`
   ```
 -i FILEPATH, --import FILEPATH
   Import pool from JSON file
 -e FILEPATH, --export FILEPATH
   Export pool to JSON file
   ```
   
   ### Code Quality
   
   - [✓] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add import and export for pool cli using JSON
> -
>
> Key: AIRFLOW-2882
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2882
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: cli
>Reporter: Chengzhi Zhao
>Assignee: Chengzhi Zhao
>Priority: Minor
> Fix For: 1.10
>
>
> Currently, the pool only support set and get by one elements,
> `airflow pool [-h] [-s NAME SLOT_COUNT POOL_DESCRIPTION] [-g NAME] [-x NAME]`
> I'd like to add import & export as Variables so cli use JSON file to perform 
> import & export operations. This will make deployment easier by setting up 
> the pools via CICD pipeline as well. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-2882) Add import and export for pool cli using JSON

2018-08-09 Thread Siddharth Anand (JIRA) 


[ 
https://issues.apache.org/jira/browse/AIRFLOW-2882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575400#comment-16575400
 ] 

Siddharth Anand commented on AIRFLOW-2882:
--

[~ChengzhiZhao] +1

> Add import and export for pool cli using JSON
> -
>
> Key: AIRFLOW-2882
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2882
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: cli
>Reporter: Chengzhi Zhao
>Assignee: Chengzhi Zhao
>Priority: Minor
> Fix For: 1.10
>
>
> Currently, the pool only support set and get by one elements,
> `airflow pool [-h] [-s NAME SLOT_COUNT POOL_DESCRIPTION] [-g NAME] [-x NAME]`
> I'd like to add import & export as Variables and Connections so cli use JSON 
> file to perform import & export operations. This will make deployment easier 
> by setting up the pools via CICD pipeline as well. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-2882) Add import and export for pool cli using JSON

2018-08-09 Thread Chengzhi Zhao (JIRA) 


[ 
https://issues.apache.org/jira/browse/AIRFLOW-2882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575372#comment-16575372
 ] 

Chengzhi Zhao commented on AIRFLOW-2882:


Please let me know what you think about adding it to cli and I will work on it. 
Thanks! 

> Add import and export for pool cli using JSON
> -
>
> Key: AIRFLOW-2882
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2882
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: cli
>Reporter: Chengzhi Zhao
>Assignee: Chengzhi Zhao
>Priority: Minor
> Fix For: 1.10
>
>
> Currently, the pool only support set and get by one elements,
> `airflow pool [-h] [-s NAME SLOT_COUNT POOL_DESCRIPTION] [-g NAME] [-x NAME]`
> I'd like to add import & export as Variables and Connections so cli use JSON 
> file to perform import & export operations. This will make deployment easier 
> by setting up the pools via CICD pipeline as well. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)