[incubator-dlab] branch DLAB-523 deleted (was d27782d)
This is an automated email from the ASF dual-hosted git repository. ioleksandr pushed a change to branch DLAB-523 in repository https://gitbox.apache.org/repos/asf/incubator-dlab.git. was d27782d [GCP]: Fix Data Engine creation fails on DeepLearning -remove old and unsupported library Caffe -change versionc of cuda for Data Engine on deeplearning The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@dlab.apache.org For additional commands, e-mail: commits-h...@dlab.apache.org
[incubator-dlab] branch develop updated (2d42d33 -> 7c6f1c9)
This is an automated email from the ASF dual-hosted git repository. omartushevskyi pushed a change to branch develop in repository https://gitbox.apache.org/repos/asf/incubator-dlab.git. from 2d42d33 Merge pull request #83 from Dyoma33/DLAB-77-dev add d27782d [GCP]: Fix Data Engine creation fails on DeepLearning -remove old and unsupported library Caffe -change versionc of cuda for Data Engine on deeplearning new 7c6f1c9 Merge pull request #84 from apache/DLAB-523 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../src/dataengine/scripts/configure_dataengine.py | 8 +++- .../scripts/configure_deep_learning_node.py| 2 - .../src/general/lib/os/debian/notebook_lib.py | 39 -- .../src/general/lib/os/redhat/notebook_lib.py | 46 -- 4 files changed, 6 insertions(+), 89 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@dlab.apache.org For additional commands, e-mail: commits-h...@dlab.apache.org
[incubator-dlab] 01/01: Merge pull request #84 from apache/DLAB-523
This is an automated email from the ASF dual-hosted git repository. omartushevskyi pushed a commit to branch develop in repository https://gitbox.apache.org/repos/asf/incubator-dlab.git commit 7c6f1c9d0916967ede8b841f2dc1d8a7cedfda2f Merge: 2d42d33 d27782d Author: Oleh Martushevskyi AuthorDate: Fri Apr 5 18:32:12 2019 +0300 Merge pull request #84 from apache/DLAB-523 [GCP]: Fix Data Engine creation fails on DeepLearning .../src/dataengine/scripts/configure_dataengine.py | 8 +++- .../scripts/configure_deep_learning_node.py| 2 - .../src/general/lib/os/debian/notebook_lib.py | 39 -- .../src/general/lib/os/redhat/notebook_lib.py | 46 -- 4 files changed, 6 insertions(+), 89 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@dlab.apache.org For additional commands, e-mail: commits-h...@dlab.apache.org
[incubator-dlab] branch DLAB-523 created (now d27782d)
This is an automated email from the ASF dual-hosted git repository. ioleksandr pushed a change to branch DLAB-523 in repository https://gitbox.apache.org/repos/asf/incubator-dlab.git. at d27782d [GCP]: Fix Data Engine creation fails on DeepLearning -remove old and unsupported library Caffe -change versionc of cuda for Data Engine on deeplearning This branch includes the following new commits: new d27782d [GCP]: Fix Data Engine creation fails on DeepLearning -remove old and unsupported library Caffe -change versionc of cuda for Data Engine on deeplearning The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@dlab.apache.org For additional commands, e-mail: commits-h...@dlab.apache.org
[incubator-dlab] 01/01: [GCP]: Fix Data Engine creation fails on DeepLearning -remove old and unsupported library Caffe -change versionc of cuda for Data Engine on deeplearning
This is an automated email from the ASF dual-hosted git repository. ioleksandr pushed a commit to branch DLAB-523 in repository https://gitbox.apache.org/repos/asf/incubator-dlab.git commit d27782dd503223810edd458edc565760e544d50c Author: Oleksandr_Isniuk AuthorDate: Fri Apr 5 18:23:34 2019 +0300 [GCP]: Fix Data Engine creation fails on DeepLearning -remove old and unsupported library Caffe -change versionc of cuda for Data Engine on deeplearning --- .../src/dataengine/scripts/configure_dataengine.py | 8 +++- .../scripts/configure_deep_learning_node.py| 2 - .../src/general/lib/os/debian/notebook_lib.py | 39 -- .../src/general/lib/os/redhat/notebook_lib.py | 46 -- 4 files changed, 6 insertions(+), 89 deletions(-) diff --git a/infrastructure-provisioning/src/dataengine/scripts/configure_dataengine.py b/infrastructure-provisioning/src/dataengine/scripts/configure_dataengine.py index 9a4fb29..e5aacb8 100644 --- a/infrastructure-provisioning/src/dataengine/scripts/configure_dataengine.py +++ b/infrastructure-provisioning/src/dataengine/scripts/configure_dataengine.py @@ -73,6 +73,12 @@ files_dir = '/root/files/' local_spark_path = '/opt/spark/' jars_dir = '/opt/jars/' r_libs = ['R6', 'pbdZMQ', 'RCurl', 'devtools', 'reshape2', 'caTools', 'rJava', 'ggplot2'] +if os.environ['application'] == 'deeplearning': +tensorflow_version = '1.4.0' +cuda_version = '8.0' +cuda_file_name = 'cuda_8.0.44_linux-run' +cudnn_version = '6.0' +cudnn_file_name = 'cudnn-8.0-linux-x64-v6.0.tgz' def start_spark(os_user, master_ip, node): @@ -165,8 +171,6 @@ if __name__ == "__main__": # INSTALL DEEP LEARNING FRAMEWORKS if os.environ['application'] == 'deeplearning': -print("Installing Caffe") -install_caffe(args.os_user, args.region, caffe_version) print("Installing Caffe2") install_caffe2(args.os_user, caffe2_version, cmake_version) print("Installing Torch") diff --git a/infrastructure-provisioning/src/deeplearning/scripts/configure_deep_learning_node.py b/infrastructure-provisioning/src/deeplearning/scripts/configure_deep_learning_node.py index ec188ed..3903bbb 100644 --- a/infrastructure-provisioning/src/deeplearning/scripts/configure_deep_learning_node.py +++ b/infrastructure-provisioning/src/deeplearning/scripts/configure_deep_learning_node.py @@ -121,8 +121,6 @@ if __name__ == "__main__": install_theano(args.os_user, theano_version) print("Installing Keras") install_keras(args.os_user, keras_version) -print("Installing Caffe") -install_caffe(args.os_user, args.region, caffe_version) print("Installing Caffe2") install_caffe2(args.os_user, caffe2_version, cmake_version) print("Installing Torch") diff --git a/infrastructure-provisioning/src/general/lib/os/debian/notebook_lib.py b/infrastructure-provisioning/src/general/lib/os/debian/notebook_lib.py index ecb5f02..91da82d 100644 --- a/infrastructure-provisioning/src/general/lib/os/debian/notebook_lib.py +++ b/infrastructure-provisioning/src/general/lib/os/debian/notebook_lib.py @@ -405,45 +405,6 @@ def get_available_os_pkgs(): sys.exit(1) -def install_caffe(os_user, region, caffe_version): -if not exists('/home/{}/.ensure_dir/caffe_ensured'.format(os_user)): -env.shell = "/bin/bash -l -c -i" -sudo('apt-get install -y python-dev') -sudo('apt-get install -y python3-dev') -sudo('apt-get install -y libprotobuf-dev libleveldb-dev libsnappy-dev libopencv-dev libhdf5-serial-dev protobuf-compiler') -sudo('apt-get install -y --no-install-recommends libboost-all-dev') -sudo('apt-get install -y libatlas-base-dev libopenblas-dev') -sudo('apt-get install -y libgflags-dev libgoogle-glog-dev liblmdb-dev') -with cd('/usr/lib/x86_64-linux-gnu/'): -sudo('ln -s libhdf5_serial_hl.so.10.0.2 libhdf5_hl.so') -sudo('ln -s libhdf5_serial.so.10.1.0 libhdf5.so') -sudo('git clone https://github.com/BVLC/caffe.git') -with cd('/home/{}/caffe/'.format(os_user)): -sudo('git checkout {}'.format(caffe_version)) -sudo('pip2 install -r python/requirements.txt --no-cache-dir') -sudo('pip3 install -r python/requirements.txt --no-cache-dir') -sudo('echo "CUDA_DIR := /usr/local/cuda" > Makefile.config') -cuda_arch = sudo("/opt/cuda-8.0/extras/demo_suite/deviceQuery | grep 'CUDA Capability' | tr -d ' ' | cut -f2 -d ':'") -sudo('echo "CUDA_ARCH := -gencode arch=compute_{0},code=sm_{0}" >> Makefile.config'.format(cuda_arch.replace('.', ''))) -sudo('echo "PYTHON_INCLUDE := /usr/include/python2.7 /usr/local/lib/python2.7/dist-packages/numpy/core/include" >> Makefile.config') -sudo('echo "BLAS := open" >> Makefile.config') -sudo('echo "BLAS_INCLUDE := /usr/include/openblas" >> Makefile.config')
[incubator-dlab] 01/01: [DLAB-456]: fixed issue with header structure
This is an automated email from the ASF dual-hosted git repository. omartushevskyi pushed a commit to branch DLAB-456 in repository https://gitbox.apache.org/repos/asf/incubator-dlab.git commit 0858bba082403fab7644968aafb4d63bbcc61e77 Author: Oleh Martushevskyi AuthorDate: Fri Apr 5 16:26:59 2019 +0300 [DLAB-456]: fixed issue with header structure --- .../src/general/lib/aws/actions_lib.py | 41 -- .../src/general/lib/azure/actions_lib.py | 41 -- .../src/general/lib/gcp/actions_lib.py | 41 -- 3 files changed, 69 insertions(+), 54 deletions(-) diff --git a/infrastructure-provisioning/src/general/lib/aws/actions_lib.py b/infrastructure-provisioning/src/general/lib/aws/actions_lib.py index f6f37d0..642d495 100644 --- a/infrastructure-provisioning/src/general/lib/aws/actions_lib.py +++ b/infrastructure-provisioning/src/general/lib/aws/actions_lib.py @@ -1588,21 +1588,23 @@ def configure_local_spark(jars_dir, templates_dir, memory_type='driver'): sudo('echo "spark.{0}.memory {1}m" >> /opt/spark/conf/spark-defaults.conf'.format(memory_type, spark_memory)) if 'spark_configurations' in os.environ: +dlab_header = sudo('cat /tmp/notebook_spark-defaults_local.conf | grep "^#"') spark_configurations = ast.literal_eval(os.environ['spark_configurations']) new_spark_defaults = list() spark_defaults = sudo('cat /opt/spark/conf/spark-defaults.conf') current_spark_properties = spark_defaults.split('\n') for param in current_spark_properties: -for config in spark_configurations: -if config['Classification'] == 'spark-defaults': -for property in config['Properties']: -if property == param.split(' ')[0]: -param = property + ' ' + config['Properties'][property] -else: -new_spark_defaults.append(property + ' ' + config['Properties'][property]) -new_spark_defaults.append(param) +if param.split(' ')[0] != '#': +for config in spark_configurations: +if config['Classification'] == 'spark-defaults': +for property in config['Properties']: +if property == param.split(' ')[0]: +param = property + ' ' + config['Properties'][property] +else: +new_spark_defaults.append(property + ' ' + config['Properties'][property]) +new_spark_defaults.append(param) new_spark_defaults = set(new_spark_defaults) -sudo('echo "" > /opt/spark/conf/spark-defaults.conf') +sudo("echo '{}' > /opt/spark/conf/spark-defaults.conf".format(dlab_header)) for prop in new_spark_defaults: prop = prop.rstrip() sudo('echo "{}" >> /opt/spark/conf/spark-defaults.conf'.format(prop)) @@ -1755,21 +1757,24 @@ def configure_dataengine_spark(cluster_name, jars_dir, cluster_dir, datalake_ena local('cp -f /tmp/{0}/notebook_spark-defaults_local.conf {1}spark/conf/spark-defaults.conf'.format(cluster_name, cluster_dir)) if spark_configs: +dlab_header = local('cat /tmp/{0}/notebook_spark-defaults_local.conf | grep "^#"'.format(cluster_name), +capture=True) spark_configurations = ast.literal_eval(spark_configs) new_spark_defaults = list() spark_defaults = local('cat {0}spark/conf/spark-defaults.conf'.format(cluster_dir), capture=True) current_spark_properties = spark_defaults.split('\n') for param in current_spark_properties: -for config in spark_configurations: -if config['Classification'] == 'spark-defaults': -for property in config['Properties']: -if property == param.split(' ')[0]: -param = property + ' ' + config['Properties'][property] -else: -new_spark_defaults.append(property + ' ' + config['Properties'][property]) -new_spark_defaults.append(param) +if param.split(' ')[0] != '#': +for config in spark_configurations: +if config['Classification'] == 'spark-defaults': +for property in config['Properties']: +if property == param.split(' ')[0]: +param = property + ' ' +
[incubator-dlab] branch bugfix-DLAB-570 created (now e4d5dd4)
This is an automated email from the ASF dual-hosted git repository. bhliva pushed a change to branch bugfix-DLAB-570 in repository https://gitbox.apache.org/repos/asf/incubator-dlab.git. at e4d5dd4 DLAB-570 fixed bug connected with wrong key status in case of fail This branch includes the following new commits: new e4d5dd4 DLAB-570 fixed bug connected with wrong key status in case of fail The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@dlab.apache.org For additional commands, e-mail: commits-h...@dlab.apache.org
[incubator-dlab] branch develop updated (31518ab -> d2deca8)
This is an automated email from the ASF dual-hosted git repository. bhliva pushed a change to branch develop in repository https://gitbox.apache.org/repos/asf/incubator-dlab.git. from 31518ab Merge pull request #81 from ioleksandr/DLAB-551 add 06f1da6 DLAB-582 added API for getting DLab version new d2deca8 Merge pull request #82 from apache/feature-DLAB-582 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: ...CodeResponse.java => InfrastructureMetaInfoDTO.java} | 12 +++- services/self-service/pom.xml | 6 ++ .../resources/InfrastructureInfoResource.java | 9 + .../backendapi/service/InfrastructureInfoService.java | 3 +++ .../service/impl/InfrastructureInfoServiceBase.java | 11 +++ .../resources/InfrastructureInfoResourceTest.java | 17 + 6 files changed, 53 insertions(+), 5 deletions(-) copy services/dlab-model/src/main/java/com/epam/dlab/dto/{gcp/auth/GcpOauth2AuthorizationCodeResponse.java => InfrastructureMetaInfoDTO.java} (81%) - To unsubscribe, e-mail: commits-unsubscr...@dlab.apache.org For additional commands, e-mail: commits-h...@dlab.apache.org
[incubator-dlab] 01/01: Merge pull request #82 from apache/feature-DLAB-582
This is an automated email from the ASF dual-hosted git repository. bhliva pushed a commit to branch develop in repository https://gitbox.apache.org/repos/asf/incubator-dlab.git commit d2deca88dc9e70171bfc0e65c9d73120c480334a Merge: 31518ab 06f1da6 Author: bhliva AuthorDate: Fri Apr 5 12:30:03 2019 +0300 Merge pull request #82 from apache/feature-DLAB-582 DLAB-582 added API for getting DLab version .../com/epam/dlab/dto/InfrastructureMetaInfoDTO.java} | 16 +--- services/self-service/pom.xml | 6 ++ .../resources/InfrastructureInfoResource.java | 9 + .../backendapi/service/InfrastructureInfoService.java | 3 +++ .../service/impl/InfrastructureInfoServiceBase.java | 11 +++ .../resources/InfrastructureInfoResourceTest.java | 17 + 6 files changed, 55 insertions(+), 7 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@dlab.apache.org For additional commands, e-mail: commits-h...@dlab.apache.org
[incubator-dlab] branch feature-DLAB-582 created (now 06f1da6)
This is an automated email from the ASF dual-hosted git repository. bhliva pushed a change to branch feature-DLAB-582 in repository https://gitbox.apache.org/repos/asf/incubator-dlab.git. at 06f1da6 DLAB-582 added API for getting DLab version This branch includes the following new commits: new 06f1da6 DLAB-582 added API for getting DLab version The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@dlab.apache.org For additional commands, e-mail: commits-h...@dlab.apache.org
[incubator-dlab] 01/01: DLAB-582 added API for getting DLab version
This is an automated email from the ASF dual-hosted git repository. bhliva pushed a commit to branch feature-DLAB-582 in repository https://gitbox.apache.org/repos/asf/incubator-dlab.git commit 06f1da6f14fecc7a1572d2cb9a79d46b57ecc5a5 Author: bhliva AuthorDate: Fri Apr 5 12:28:19 2019 +0300 DLAB-582 added API for getting DLab version --- .../com/epam/dlab/dto/InfrastructureMetaInfoDTO.java} | 16 +--- services/self-service/pom.xml | 6 ++ .../resources/InfrastructureInfoResource.java | 9 + .../backendapi/service/InfrastructureInfoService.java | 3 +++ .../service/impl/InfrastructureInfoServiceBase.java | 11 +++ .../resources/InfrastructureInfoResourceTest.java | 17 + 6 files changed, 55 insertions(+), 7 deletions(-) diff --git a/services/self-service/src/main/java/com/epam/dlab/backendapi/service/InfrastructureInfoService.java b/services/dlab-model/src/main/java/com/epam/dlab/dto/InfrastructureMetaInfoDTO.java similarity index 69% copy from services/self-service/src/main/java/com/epam/dlab/backendapi/service/InfrastructureInfoService.java copy to services/dlab-model/src/main/java/com/epam/dlab/dto/InfrastructureMetaInfoDTO.java index a3e08d2..c873fab 100644 --- a/services/self-service/src/main/java/com/epam/dlab/backendapi/service/InfrastructureInfoService.java +++ b/services/dlab-model/src/main/java/com/epam/dlab/dto/InfrastructureMetaInfoDTO.java @@ -17,13 +17,15 @@ * under the License. */ -package com.epam.dlab.backendapi.service; +package com.epam.dlab.dto; -import com.epam.dlab.backendapi.resources.dto.HealthStatusPageDTO; -import com.epam.dlab.backendapi.resources.dto.InfrastructureInfo; +import lombok.Builder; +import lombok.Data; -public interface InfrastructureInfoService { - InfrastructureInfo getUserResources(String user); - - HealthStatusPageDTO getHeathStatus(String user, boolean fullReport, boolean isAdmin); +@Data +@Builder +public class InfrastructureMetaInfoDTO { + private final String branch; + private final String version; + private final String commit; } diff --git a/services/self-service/pom.xml b/services/self-service/pom.xml index 9ff451b..a71c9c4 100644 --- a/services/self-service/pom.xml +++ b/services/self-service/pom.xml @@ -134,6 +134,12 @@ oshi-core ${com.github.oshi.core.version} + +com.jcabi +jcabi-manifests +1.1 + + com.smoketurner diff --git a/services/self-service/src/main/java/com/epam/dlab/backendapi/resources/InfrastructureInfoResource.java b/services/self-service/src/main/java/com/epam/dlab/backendapi/resources/InfrastructureInfoResource.java index dfedb8e..6b44c33 100644 --- a/services/self-service/src/main/java/com/epam/dlab/backendapi/resources/InfrastructureInfoResource.java +++ b/services/self-service/src/main/java/com/epam/dlab/backendapi/resources/InfrastructureInfoResource.java @@ -88,4 +88,13 @@ public class InfrastructureInfoResource { return infrastructureInfoService.getUserResources(userInfo.getName()); } + + @GET + @Path("/meta") + @ApiOperation("Return metainfo regarding application version etc") + public Response getVersion(@ApiParam(hidden = true) @Auth UserInfo userInfo) { + return Response.ok(infrastructureInfoService.getInfrastructureMetaInfo()) + .build(); + + } } diff --git a/services/self-service/src/main/java/com/epam/dlab/backendapi/service/InfrastructureInfoService.java b/services/self-service/src/main/java/com/epam/dlab/backendapi/service/InfrastructureInfoService.java index a3e08d2..b54edb9 100644 --- a/services/self-service/src/main/java/com/epam/dlab/backendapi/service/InfrastructureInfoService.java +++ b/services/self-service/src/main/java/com/epam/dlab/backendapi/service/InfrastructureInfoService.java @@ -21,9 +21,12 @@ package com.epam.dlab.backendapi.service; import com.epam.dlab.backendapi.resources.dto.HealthStatusPageDTO; import com.epam.dlab.backendapi.resources.dto.InfrastructureInfo; +import com.epam.dlab.dto.InfrastructureMetaInfoDTO; public interface InfrastructureInfoService { InfrastructureInfo getUserResources(String user); HealthStatusPageDTO getHeathStatus(String user, boolean fullReport, boolean isAdmin); + + InfrastructureMetaInfoDTO getInfrastructureMetaInfo(); } diff --git a/services/self-service/src/main/java/com/epam/dlab/backendapi/service/impl/InfrastructureInfoServiceBase.java b/services/self-service/src/main/java/com/epam/dlab/backendapi/service/impl/InfrastructureInfoServiceBase.java index 0f0584c..6981531 100644 --- a/services/self-service/src/main/java/com/epam/dlab/backendapi/service/impl/InfrastructureInfoServiceBase.java +++