[incubator-dlab] 01/01: Merge pull request #90 from apache/DLAB-597-fix
This is an automated email from the ASF dual-hosted git repository. ioleksandr pushed a commit to branch develop in repository https://gitbox.apache.org/repos/asf/incubator-dlab.git commit 8dae9085dfabed5674e8fb1d6f7f2d981a70de7e Merge: aff5303 404eb3c Author: i.oleksandr AuthorDate: Tue Apr 9 17:54:25 2019 +0300 Merge pull request #90 from apache/DLAB-597-fix [DLAB-597]: Fixed issue with getting list of Service Accounts infrastructure-provisioning/src/general/lib/gcp/meta_lib.py | 11 +++ 1 file changed, 7 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@dlab.apache.org For additional commands, e-mail: commits-h...@dlab.apache.org
[incubator-dlab] branch DLAB-597-fix deleted (was 404eb3c)
This is an automated email from the ASF dual-hosted git repository. ioleksandr pushed a change to branch DLAB-597-fix in repository https://gitbox.apache.org/repos/asf/incubator-dlab.git. was 404eb3c [DLAB-597]: Fixed issue with getting list of Service Accounts The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@dlab.apache.org For additional commands, e-mail: commits-h...@dlab.apache.org
[incubator-dlab] branch develop updated (aff5303 -> 8dae908)
This is an automated email from the ASF dual-hosted git repository. ioleksandr pushed a change to branch develop in repository https://gitbox.apache.org/repos/asf/incubator-dlab.git. from aff5303 Merge pull request #89 from apache/DLAB-596 add 404eb3c [DLAB-597]: Fixed issue with getting list of Service Accounts new 8dae908 Merge pull request #90 from apache/DLAB-597-fix The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: infrastructure-provisioning/src/general/lib/gcp/meta_lib.py | 11 +++ 1 file changed, 7 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@dlab.apache.org For additional commands, e-mail: commits-h...@dlab.apache.org
[incubator-dlab] branch DLAB-597 deleted (was 385e2be)
This is an automated email from the ASF dual-hosted git repository. omartushevskyi pushed a change to branch DLAB-597 in repository https://gitbox.apache.org/repos/asf/incubator-dlab.git. was 385e2be [DLAB-597]: debugging This change permanently discards the following revisions: discard 385e2be [DLAB-597]: debugging discard ff34b5e [DLAB-597]: Fixed issue with getting list of Service Accounts discard 02b8ad6 [DLAB-597]: Fixed issue with getting list of Service Accounts discard e4af162 [DLAB-597]: Fixed issue with getting list of Service Accounts discard 481969c debugging - To unsubscribe, e-mail: commits-unsubscr...@dlab.apache.org For additional commands, e-mail: commits-h...@dlab.apache.org
[incubator-dlab] 01/01: [DLAB-597]: Fixed issue with getting list of Service Accounts
This is an automated email from the ASF dual-hosted git repository.
omartushevskyi pushed a commit to branch DLAB-597-fix
in repository https://gitbox.apache.org/repos/asf/incubator-dlab.git
commit 404eb3c27a6524cefcd47256536b4c28a6078d29
Author: Oleh Martushevskyi
AuthorDate: Tue Apr 9 17:26:26 2019 +0300
[DLAB-597]: Fixed issue with getting list of Service Accounts
---
infrastructure-provisioning/src/general/lib/gcp/meta_lib.py | 11 +++
1 file changed, 7 insertions(+), 4 deletions(-)
diff --git a/infrastructure-provisioning/src/general/lib/gcp/meta_lib.py
b/infrastructure-provisioning/src/general/lib/gcp/meta_lib.py
index b122245..fa8b5a2 100644
--- a/infrastructure-provisioning/src/general/lib/gcp/meta_lib.py
+++ b/infrastructure-provisioning/src/general/lib/gcp/meta_lib.py
@@ -332,10 +332,13 @@ class GCPMeta:
while next_page:
result2 =
self.service_iam.projects().serviceAccounts().list(name='projects/{}'.format(self.project),
pageToken=page_token).execute()
-for account in result2['accounts']:
-service_account_names.append(account['displayName'])
-if 'nextPageToken' in result2:
-page_token = result2['nextPageToken']
+if result2:
+for account in result2['accounts']:
+service_account_names.append(account['displayName'])
+if 'nextPageToken' in result2:
+page_token = result2['nextPageToken']
+else:
+next_page = False
else:
next_page = False
return service_account_names
-
To unsubscribe, e-mail: commits-unsubscr...@dlab.apache.org
For additional commands, e-mail: commits-h...@dlab.apache.org
[incubator-dlab] branch DLAB-597-fix created (now 404eb3c)
This is an automated email from the ASF dual-hosted git repository. omartushevskyi pushed a change to branch DLAB-597-fix in repository https://gitbox.apache.org/repos/asf/incubator-dlab.git. at 404eb3c [DLAB-597]: Fixed issue with getting list of Service Accounts This branch includes the following new commits: new 404eb3c [DLAB-597]: Fixed issue with getting list of Service Accounts The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@dlab.apache.org For additional commands, e-mail: commits-h...@dlab.apache.org
[incubator-dlab] branch DLAB-597 updated: [DLAB-597]: debugging
This is an automated email from the ASF dual-hosted git repository.
omartushevskyi pushed a commit to branch DLAB-597
in repository https://gitbox.apache.org/repos/asf/incubator-dlab.git
The following commit(s) were added to refs/heads/DLAB-597 by this push:
new 385e2be [DLAB-597]: debugging
385e2be is described below
commit 385e2bec7c87876a59260347ecbe3ba2fa405d32
Author: Oleh Martushevskyi
AuthorDate: Tue Apr 9 17:22:43 2019 +0300
[DLAB-597]: debugging
---
infrastructure-provisioning/src/general/lib/gcp/meta_lib.py | 1 +
1 file changed, 1 insertion(+)
diff --git a/infrastructure-provisioning/src/general/lib/gcp/meta_lib.py
b/infrastructure-provisioning/src/general/lib/gcp/meta_lib.py
index fa8b5a2..e1ecc0f 100644
--- a/infrastructure-provisioning/src/general/lib/gcp/meta_lib.py
+++ b/infrastructure-provisioning/src/general/lib/gcp/meta_lib.py
@@ -332,6 +332,7 @@ class GCPMeta:
while next_page:
result2 =
self.service_iam.projects().serviceAccounts().list(name='projects/{}'.format(self.project),
pageToken=page_token).execute()
+print(type(result2))
if result2:
for account in result2['accounts']:
service_account_names.append(account['displayName'])
-
To unsubscribe, e-mail: commits-unsubscr...@dlab.apache.org
For additional commands, e-mail: commits-h...@dlab.apache.org
[incubator-dlab] branch DLAB-597 updated: [DLAB-597]: Fixed issue with getting list of Service Accounts
This is an automated email from the ASF dual-hosted git repository. omartushevskyi pushed a commit to branch DLAB-597 in repository https://gitbox.apache.org/repos/asf/incubator-dlab.git The following commit(s) were added to refs/heads/DLAB-597 by this push: new ff34b5e [DLAB-597]: Fixed issue with getting list of Service Accounts ff34b5e is described below commit ff34b5eb705a2998342554a11e2c38aaf227ac52 Author: Oleh Martushevskyi AuthorDate: Tue Apr 9 17:20:12 2019 +0300 [DLAB-597]: Fixed issue with getting list of Service Accounts --- infrastructure-provisioning/src/general/lib/gcp/meta_lib.py | 6 -- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/infrastructure-provisioning/src/general/lib/gcp/meta_lib.py b/infrastructure-provisioning/src/general/lib/gcp/meta_lib.py index 7796af8..fa8b5a2 100644 --- a/infrastructure-provisioning/src/general/lib/gcp/meta_lib.py +++ b/infrastructure-provisioning/src/general/lib/gcp/meta_lib.py @@ -335,8 +335,10 @@ class GCPMeta: if result2: for account in result2['accounts']: service_account_names.append(account['displayName']) -if 'nextPageToken' in result2: -page_token = result2['nextPageToken'] +if 'nextPageToken' in result2: +page_token = result2['nextPageToken'] +else: +next_page = False else: next_page = False return service_account_names - To unsubscribe, e-mail: commits-unsubscr...@dlab.apache.org For additional commands, e-mail: commits-h...@dlab.apache.org
[incubator-dlab] branch DLAB-597 updated: [DLAB-597]: Fixed issue with getting list of Service Accounts
This is an automated email from the ASF dual-hosted git repository.
omartushevskyi pushed a commit to branch DLAB-597
in repository https://gitbox.apache.org/repos/asf/incubator-dlab.git
The following commit(s) were added to refs/heads/DLAB-597 by this push:
new 02b8ad6 [DLAB-597]: Fixed issue with getting list of Service Accounts
02b8ad6 is described below
commit 02b8ad606edb0f97c777ee3d7532e2a052d849a4
Author: Oleh Martushevskyi
AuthorDate: Tue Apr 9 17:11:12 2019 +0300
[DLAB-597]: Fixed issue with getting list of Service Accounts
---
infrastructure-provisioning/src/general/lib/gcp/meta_lib.py | 8
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/infrastructure-provisioning/src/general/lib/gcp/meta_lib.py
b/infrastructure-provisioning/src/general/lib/gcp/meta_lib.py
index 6138a1d..7796af8 100644
--- a/infrastructure-provisioning/src/general/lib/gcp/meta_lib.py
+++ b/infrastructure-provisioning/src/general/lib/gcp/meta_lib.py
@@ -335,10 +335,10 @@ class GCPMeta:
if result2:
for account in result2['accounts']:
service_account_names.append(account['displayName'])
-if 'nextPageToken' in result2:
-page_token = result2['nextPageToken']
-else:
-next_page = False
+if 'nextPageToken' in result2:
+page_token = result2['nextPageToken']
+else:
+next_page = False
return service_account_names
except Exception as err:
logging.info("Error with getting list service accounts: " +
str(err) + "\n Traceback: " + traceback.print_exc(file=sys.stdout))
-
To unsubscribe, e-mail: commits-unsubscr...@dlab.apache.org
For additional commands, e-mail: commits-h...@dlab.apache.org
[incubator-dlab] branch DLAB-597 updated: [DLAB-597]: Fixed issue with getting list of Service Accounts
This is an automated email from the ASF dual-hosted git repository.
omartushevskyi pushed a commit to branch DLAB-597
in repository https://gitbox.apache.org/repos/asf/incubator-dlab.git
The following commit(s) were added to refs/heads/DLAB-597 by this push:
new e4af162 [DLAB-597]: Fixed issue with getting list of Service Accounts
e4af162 is described below
commit e4af16257959d4bd5af3836d7a4ac4b066537bb8
Author: Oleh Martushevskyi
AuthorDate: Tue Apr 9 16:44:15 2019 +0300
[DLAB-597]: Fixed issue with getting list of Service Accounts
---
.../src/general/lib/gcp/meta_lib.py | 15 +++
1 file changed, 7 insertions(+), 8 deletions(-)
diff --git a/infrastructure-provisioning/src/general/lib/gcp/meta_lib.py
b/infrastructure-provisioning/src/general/lib/gcp/meta_lib.py
index c2558fd..6138a1d 100644
--- a/infrastructure-provisioning/src/general/lib/gcp/meta_lib.py
+++ b/infrastructure-provisioning/src/general/lib/gcp/meta_lib.py
@@ -332,14 +332,13 @@ class GCPMeta:
while next_page:
result2 =
self.service_iam.projects().serviceAccounts().list(name='projects/{}'.format(self.project),
pageToken=page_token).execute()
-print("")
-print(result2)
-for account in result2['accounts']:
-service_account_names.append(account['displayName'])
-if 'nextPageToken' in result2:
-page_token = result2['nextPageToken']
-else:
-next_page = False
+if result2:
+for account in result2['accounts']:
+service_account_names.append(account['displayName'])
+if 'nextPageToken' in result2:
+page_token = result2['nextPageToken']
+else:
+next_page = False
return service_account_names
except Exception as err:
logging.info("Error with getting list service accounts: " +
str(err) + "\n Traceback: " + traceback.print_exc(file=sys.stdout))
-
To unsubscribe, e-mail: commits-unsubscr...@dlab.apache.org
For additional commands, e-mail: commits-h...@dlab.apache.org
[incubator-dlab] branch DLAB-597 created (now 481969c)
This is an automated email from the ASF dual-hosted git repository. omartushevskyi pushed a change to branch DLAB-597 in repository https://gitbox.apache.org/repos/asf/incubator-dlab.git. at 481969c debugging This branch includes the following new commits: new 481969c debugging The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@dlab.apache.org For additional commands, e-mail: commits-h...@dlab.apache.org
[incubator-dlab] 01/01: debugging
This is an automated email from the ASF dual-hosted git repository.
omartushevskyi pushed a commit to branch DLAB-597
in repository https://gitbox.apache.org/repos/asf/incubator-dlab.git
commit 481969c77b2d5784cad982097581953dea6c608b
Author: Oleh Martushevskyi
AuthorDate: Tue Apr 9 16:32:54 2019 +0300
debugging
---
infrastructure-provisioning/src/general/lib/gcp/meta_lib.py | 2 ++
1 file changed, 2 insertions(+)
diff --git a/infrastructure-provisioning/src/general/lib/gcp/meta_lib.py
b/infrastructure-provisioning/src/general/lib/gcp/meta_lib.py
index b122245..c2558fd 100644
--- a/infrastructure-provisioning/src/general/lib/gcp/meta_lib.py
+++ b/infrastructure-provisioning/src/general/lib/gcp/meta_lib.py
@@ -332,6 +332,8 @@ class GCPMeta:
while next_page:
result2 =
self.service_iam.projects().serviceAccounts().list(name='projects/{}'.format(self.project),
pageToken=page_token).execute()
+print("")
+print(result2)
for account in result2['accounts']:
service_account_names.append(account['displayName'])
if 'nextPageToken' in result2:
-
To unsubscribe, e-mail: commits-unsubscr...@dlab.apache.org
For additional commands, e-mail: commits-h...@dlab.apache.org
[incubator-dlab] 01/01: Merge pull request #89 from apache/DLAB-596
This is an automated email from the ASF dual-hosted git repository. ioleksandr pushed a commit to branch develop in repository https://gitbox.apache.org/repos/asf/incubator-dlab.git commit aff5303ed70dd069cd34049365c3f5d9f8e3eece Merge: f74984a ae2926a Author: i.oleksandr AuthorDate: Tue Apr 9 16:17:34 2019 +0300 Merge pull request #89 from apache/DLAB-596 [DLAB-596]: Fixed issue with header structure after Dataengine creati… .../src/general/scripts/os/common_configure_spark.py | 20 +++- 1 file changed, 11 insertions(+), 9 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@dlab.apache.org For additional commands, e-mail: commits-h...@dlab.apache.org
[incubator-dlab] branch DLAB-596 deleted (was ae2926a)
This is an automated email from the ASF dual-hosted git repository. ioleksandr pushed a change to branch DLAB-596 in repository https://gitbox.apache.org/repos/asf/incubator-dlab.git. was ae2926a [DLAB-596]: Fixed issue with header structure after Dataengine creation with custom Spark configuration The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@dlab.apache.org For additional commands, e-mail: commits-h...@dlab.apache.org
[incubator-dlab] branch develop updated (f74984a -> aff5303)
This is an automated email from the ASF dual-hosted git repository. ioleksandr pushed a change to branch develop in repository https://gitbox.apache.org/repos/asf/incubator-dlab.git. from f74984a Merge pull request #88 from apache/DLAB-594 add ae2926a [DLAB-596]: Fixed issue with header structure after Dataengine creation with custom Spark configuration new aff5303 Merge pull request #89 from apache/DLAB-596 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../src/general/scripts/os/common_configure_spark.py | 20 +++- 1 file changed, 11 insertions(+), 9 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@dlab.apache.org For additional commands, e-mail: commits-h...@dlab.apache.org
[incubator-dlab] branch DLAB-596 created (now ae2926a)
This is an automated email from the ASF dual-hosted git repository. omartushevskyi pushed a change to branch DLAB-596 in repository https://gitbox.apache.org/repos/asf/incubator-dlab.git. at ae2926a [DLAB-596]: Fixed issue with header structure after Dataengine creation with custom Spark configuration This branch includes the following new commits: new ae2926a [DLAB-596]: Fixed issue with header structure after Dataengine creation with custom Spark configuration The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@dlab.apache.org For additional commands, e-mail: commits-h...@dlab.apache.org
[incubator-dlab] 01/01: [DLAB-596]: Fixed issue with header structure after Dataengine creation with custom Spark configuration
This is an automated email from the ASF dual-hosted git repository.
omartushevskyi pushed a commit to branch DLAB-596
in repository https://gitbox.apache.org/repos/asf/incubator-dlab.git
commit ae2926a18dc97f64c5f04b1acb2abc41ee0d9001
Author: Oleh Martushevskyi
AuthorDate: Tue Apr 9 16:11:05 2019 +0300
[DLAB-596]: Fixed issue with header structure after Dataengine creation
with custom Spark configuration
---
.../src/general/scripts/os/common_configure_spark.py | 20 +++-
1 file changed, 11 insertions(+), 9 deletions(-)
diff --git
a/infrastructure-provisioning/src/general/scripts/os/common_configure_spark.py
b/infrastructure-provisioning/src/general/scripts/os/common_configure_spark.py
index 0d104db..fa15496 100644
---
a/infrastructure-provisioning/src/general/scripts/os/common_configure_spark.py
+++
b/infrastructure-provisioning/src/general/scripts/os/common_configure_spark.py
@@ -53,21 +53,23 @@ def update_spark_defaults_conf(spark_conf):
def add_custom_spark_properties(cluster_name):
try:
+dlab_header = sudo('cat /tmp/{0}/notebook_spark-defaults_local.conf |
grep "^#"'.format(cluster_name))
spark_configurations =
ast.literal_eval(os.environ['spark_configurations'])
new_spark_defaults = list()
spark_defaults = sudo('cat
/opt/{0}/spark/conf/spark-defaults.conf'.format(cluster_name))
current_spark_properties = spark_defaults.split('\n')
for param in current_spark_properties:
-for config in spark_configurations:
-if config['Classification'] == 'spark-defaults':
-for property in config['Properties']:
-if property == param.split(' ')[0]:
-param = property + ' ' +
config['Properties'][property]
-else:
-new_spark_defaults.append(property + ' ' +
config['Properties'][property])
-new_spark_defaults.append(param)
+if param.split(' ')[0] != '#':
+for config in spark_configurations:
+if config['Classification'] == 'spark-defaults':
+for property in config['Properties']:
+if property == param.split(' ')[0]:
+param = property + ' ' +
config['Properties'][property]
+else:
+new_spark_defaults.append(property + ' ' +
config['Properties'][property])
+new_spark_defaults.append(param)
new_spark_defaults = set(new_spark_defaults)
-sudo('echo "" >
/opt/{0}/spark/conf/spark-defaults.conf'.format(cluster_name))
+sudo("echo '{0}' >
/opt/{1}/spark/conf/spark-defaults.conf".format(dlab_header, cluster_name))
for prop in new_spark_defaults:
prop = prop.rstrip()
sudo('echo "{0}" >>
/opt/{1}/spark/conf/spark-defaults.conf'.format(prop, cluster_name))
-
To unsubscribe, e-mail: commits-unsubscr...@dlab.apache.org
For additional commands, e-mail: commits-h...@dlab.apache.org
