[incubator-dlab] branch DLAB-523 deleted (was d27782d)
This is an automated email from the ASF dual-hosted git repository. ioleksandr pushed a change to branch DLAB-523 in repository https://gitbox.apache.org/repos/asf/incubator-dlab.git. was d27782d [GCP]: Fix Data Engine creation fails on DeepLearning -remove old and unsupported library Caffe -change versionc of cuda for Data Engine on deeplearning The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@dlab.apache.org For additional commands, e-mail: commits-h...@dlab.apache.org
[incubator-dlab] branch develop updated (2d42d33 -> 7c6f1c9)
This is an automated email from the ASF dual-hosted git repository. omartushevskyi pushed a change to branch develop in repository https://gitbox.apache.org/repos/asf/incubator-dlab.git. from 2d42d33 Merge pull request #83 from Dyoma33/DLAB-77-dev add d27782d [GCP]: Fix Data Engine creation fails on DeepLearning -remove old and unsupported library Caffe -change versionc of cuda for Data Engine on deeplearning new 7c6f1c9 Merge pull request #84 from apache/DLAB-523 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../src/dataengine/scripts/configure_dataengine.py | 8 +++- .../scripts/configure_deep_learning_node.py| 2 - .../src/general/lib/os/debian/notebook_lib.py | 39 -- .../src/general/lib/os/redhat/notebook_lib.py | 46 -- 4 files changed, 6 insertions(+), 89 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@dlab.apache.org For additional commands, e-mail: commits-h...@dlab.apache.org
[incubator-dlab] 01/01: Merge pull request #84 from apache/DLAB-523
This is an automated email from the ASF dual-hosted git repository. omartushevskyi pushed a commit to branch develop in repository https://gitbox.apache.org/repos/asf/incubator-dlab.git commit 7c6f1c9d0916967ede8b841f2dc1d8a7cedfda2f Merge: 2d42d33 d27782d Author: Oleh Martushevskyi AuthorDate: Fri Apr 5 18:32:12 2019 +0300 Merge pull request #84 from apache/DLAB-523 [GCP]: Fix Data Engine creation fails on DeepLearning .../src/dataengine/scripts/configure_dataengine.py | 8 +++- .../scripts/configure_deep_learning_node.py| 2 - .../src/general/lib/os/debian/notebook_lib.py | 39 -- .../src/general/lib/os/redhat/notebook_lib.py | 46 -- 4 files changed, 6 insertions(+), 89 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@dlab.apache.org For additional commands, e-mail: commits-h...@dlab.apache.org
[incubator-dlab] branch DLAB-523 created (now d27782d)
This is an automated email from the ASF dual-hosted git repository. ioleksandr pushed a change to branch DLAB-523 in repository https://gitbox.apache.org/repos/asf/incubator-dlab.git. at d27782d [GCP]: Fix Data Engine creation fails on DeepLearning -remove old and unsupported library Caffe -change versionc of cuda for Data Engine on deeplearning This branch includes the following new commits: new d27782d [GCP]: Fix Data Engine creation fails on DeepLearning -remove old and unsupported library Caffe -change versionc of cuda for Data Engine on deeplearning The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@dlab.apache.org For additional commands, e-mail: commits-h...@dlab.apache.org
[incubator-dlab] 01/01: [GCP]: Fix Data Engine creation fails on DeepLearning -remove old and unsupported library Caffe -change versionc of cuda for Data Engine on deeplearning
This is an automated email from the ASF dual-hosted git repository.
ioleksandr pushed a commit to branch DLAB-523
in repository https://gitbox.apache.org/repos/asf/incubator-dlab.git
commit d27782dd503223810edd458edc565760e544d50c
Author: Oleksandr_Isniuk
AuthorDate: Fri Apr 5 18:23:34 2019 +0300
[GCP]: Fix Data Engine creation fails on DeepLearning
-remove old and unsupported library Caffe
-change versionc of cuda for Data Engine on deeplearning
---
.../src/dataengine/scripts/configure_dataengine.py | 8 +++-
.../scripts/configure_deep_learning_node.py| 2 -
.../src/general/lib/os/debian/notebook_lib.py | 39 --
.../src/general/lib/os/redhat/notebook_lib.py | 46 --
4 files changed, 6 insertions(+), 89 deletions(-)
diff --git
a/infrastructure-provisioning/src/dataengine/scripts/configure_dataengine.py
b/infrastructure-provisioning/src/dataengine/scripts/configure_dataengine.py
index 9a4fb29..e5aacb8 100644
--- a/infrastructure-provisioning/src/dataengine/scripts/configure_dataengine.py
+++ b/infrastructure-provisioning/src/dataengine/scripts/configure_dataengine.py
@@ -73,6 +73,12 @@ files_dir = '/root/files/'
local_spark_path = '/opt/spark/'
jars_dir = '/opt/jars/'
r_libs = ['R6', 'pbdZMQ', 'RCurl', 'devtools', 'reshape2', 'caTools', 'rJava',
'ggplot2']
+if os.environ['application'] == 'deeplearning':
+tensorflow_version = '1.4.0'
+cuda_version = '8.0'
+cuda_file_name = 'cuda_8.0.44_linux-run'
+cudnn_version = '6.0'
+cudnn_file_name = 'cudnn-8.0-linux-x64-v6.0.tgz'
def start_spark(os_user, master_ip, node):
@@ -165,8 +171,6 @@ if __name__ == "__main__":
# INSTALL DEEP LEARNING FRAMEWORKS
if os.environ['application'] == 'deeplearning':
-print("Installing Caffe")
-install_caffe(args.os_user, args.region, caffe_version)
print("Installing Caffe2")
install_caffe2(args.os_user, caffe2_version, cmake_version)
print("Installing Torch")
diff --git
a/infrastructure-provisioning/src/deeplearning/scripts/configure_deep_learning_node.py
b/infrastructure-provisioning/src/deeplearning/scripts/configure_deep_learning_node.py
index ec188ed..3903bbb 100644
---
a/infrastructure-provisioning/src/deeplearning/scripts/configure_deep_learning_node.py
+++
b/infrastructure-provisioning/src/deeplearning/scripts/configure_deep_learning_node.py
@@ -121,8 +121,6 @@ if __name__ == "__main__":
install_theano(args.os_user, theano_version)
print("Installing Keras")
install_keras(args.os_user, keras_version)
-print("Installing Caffe")
-install_caffe(args.os_user, args.region, caffe_version)
print("Installing Caffe2")
install_caffe2(args.os_user, caffe2_version, cmake_version)
print("Installing Torch")
diff --git
a/infrastructure-provisioning/src/general/lib/os/debian/notebook_lib.py
b/infrastructure-provisioning/src/general/lib/os/debian/notebook_lib.py
index ecb5f02..91da82d 100644
--- a/infrastructure-provisioning/src/general/lib/os/debian/notebook_lib.py
+++ b/infrastructure-provisioning/src/general/lib/os/debian/notebook_lib.py
@@ -405,45 +405,6 @@ def get_available_os_pkgs():
sys.exit(1)
-def install_caffe(os_user, region, caffe_version):
-if not exists('/home/{}/.ensure_dir/caffe_ensured'.format(os_user)):
-env.shell = "/bin/bash -l -c -i"
-sudo('apt-get install -y python-dev')
-sudo('apt-get install -y python3-dev')
-sudo('apt-get install -y libprotobuf-dev libleveldb-dev libsnappy-dev
libopencv-dev libhdf5-serial-dev protobuf-compiler')
-sudo('apt-get install -y --no-install-recommends libboost-all-dev')
-sudo('apt-get install -y libatlas-base-dev libopenblas-dev')
-sudo('apt-get install -y libgflags-dev libgoogle-glog-dev liblmdb-dev')
-with cd('/usr/lib/x86_64-linux-gnu/'):
-sudo('ln -s libhdf5_serial_hl.so.10.0.2 libhdf5_hl.so')
-sudo('ln -s libhdf5_serial.so.10.1.0 libhdf5.so')
-sudo('git clone https://github.com/BVLC/caffe.git')
-with cd('/home/{}/caffe/'.format(os_user)):
-sudo('git checkout {}'.format(caffe_version))
-sudo('pip2 install -r python/requirements.txt --no-cache-dir')
-sudo('pip3 install -r python/requirements.txt --no-cache-dir')
-sudo('echo "CUDA_DIR := /usr/local/cuda" > Makefile.config')
-cuda_arch = sudo("/opt/cuda-8.0/extras/demo_suite/deviceQuery |
grep 'CUDA Capability' | tr -d ' ' | cut -f2 -d ':'")
-sudo('echo "CUDA_ARCH := -gencode arch=compute_{0},code=sm_{0}" >>
Makefile.config'.format(cuda_arch.replace('.', '')))
-sudo('echo "PYTHON_INCLUDE := /usr/include/python2.7
/usr/local/lib/python2.7/dist-packages/numpy/core/include" >> Makefile.config')
-sudo('echo "BLAS := open" >> Makefile.config')
-sudo('echo "BLAS_INCLUDE := /usr/include/openblas" >>
Makefile.config')
[incubator-dlab] 01/01: [DLAB-456]: fixed issue with header structure
This is an automated email from the ASF dual-hosted git repository.
omartushevskyi pushed a commit to branch DLAB-456
in repository https://gitbox.apache.org/repos/asf/incubator-dlab.git
commit 0858bba082403fab7644968aafb4d63bbcc61e77
Author: Oleh Martushevskyi
AuthorDate: Fri Apr 5 16:26:59 2019 +0300
[DLAB-456]: fixed issue with header structure
---
.../src/general/lib/aws/actions_lib.py | 41 --
.../src/general/lib/azure/actions_lib.py | 41 --
.../src/general/lib/gcp/actions_lib.py | 41 --
3 files changed, 69 insertions(+), 54 deletions(-)
diff --git a/infrastructure-provisioning/src/general/lib/aws/actions_lib.py
b/infrastructure-provisioning/src/general/lib/aws/actions_lib.py
index f6f37d0..642d495 100644
--- a/infrastructure-provisioning/src/general/lib/aws/actions_lib.py
+++ b/infrastructure-provisioning/src/general/lib/aws/actions_lib.py
@@ -1588,21 +1588,23 @@ def configure_local_spark(jars_dir, templates_dir,
memory_type='driver'):
sudo('echo "spark.{0}.memory {1}m" >>
/opt/spark/conf/spark-defaults.conf'.format(memory_type,
spark_memory))
if 'spark_configurations' in os.environ:
+dlab_header = sudo('cat /tmp/notebook_spark-defaults_local.conf |
grep "^#"')
spark_configurations =
ast.literal_eval(os.environ['spark_configurations'])
new_spark_defaults = list()
spark_defaults = sudo('cat /opt/spark/conf/spark-defaults.conf')
current_spark_properties = spark_defaults.split('\n')
for param in current_spark_properties:
-for config in spark_configurations:
-if config['Classification'] == 'spark-defaults':
-for property in config['Properties']:
-if property == param.split(' ')[0]:
-param = property + ' ' +
config['Properties'][property]
-else:
-new_spark_defaults.append(property + ' ' +
config['Properties'][property])
-new_spark_defaults.append(param)
+if param.split(' ')[0] != '#':
+for config in spark_configurations:
+if config['Classification'] == 'spark-defaults':
+for property in config['Properties']:
+if property == param.split(' ')[0]:
+param = property + ' ' +
config['Properties'][property]
+else:
+new_spark_defaults.append(property + ' ' +
config['Properties'][property])
+new_spark_defaults.append(param)
new_spark_defaults = set(new_spark_defaults)
-sudo('echo "" > /opt/spark/conf/spark-defaults.conf')
+sudo("echo '{}' >
/opt/spark/conf/spark-defaults.conf".format(dlab_header))
for prop in new_spark_defaults:
prop = prop.rstrip()
sudo('echo "{}" >>
/opt/spark/conf/spark-defaults.conf'.format(prop))
@@ -1755,21 +1757,24 @@ def configure_dataengine_spark(cluster_name, jars_dir,
cluster_dir, datalake_ena
local('cp -f /tmp/{0}/notebook_spark-defaults_local.conf
{1}spark/conf/spark-defaults.conf'.format(cluster_name,
cluster_dir))
if spark_configs:
+dlab_header = local('cat /tmp/{0}/notebook_spark-defaults_local.conf |
grep "^#"'.format(cluster_name),
+capture=True)
spark_configurations = ast.literal_eval(spark_configs)
new_spark_defaults = list()
spark_defaults = local('cat
{0}spark/conf/spark-defaults.conf'.format(cluster_dir), capture=True)
current_spark_properties = spark_defaults.split('\n')
for param in current_spark_properties:
-for config in spark_configurations:
-if config['Classification'] == 'spark-defaults':
-for property in config['Properties']:
-if property == param.split(' ')[0]:
-param = property + ' ' +
config['Properties'][property]
-else:
-new_spark_defaults.append(property + ' ' +
config['Properties'][property])
-new_spark_defaults.append(param)
+if param.split(' ')[0] != '#':
+for config in spark_configurations:
+if config['Classification'] == 'spark-defaults':
+for property in config['Properties']:
+if property == param.split(' ')[0]:
+param = property + ' ' +
[incubator-dlab] branch bugfix-DLAB-570 created (now e4d5dd4)
This is an automated email from the ASF dual-hosted git repository. bhliva pushed a change to branch bugfix-DLAB-570 in repository https://gitbox.apache.org/repos/asf/incubator-dlab.git. at e4d5dd4 DLAB-570 fixed bug connected with wrong key status in case of fail This branch includes the following new commits: new e4d5dd4 DLAB-570 fixed bug connected with wrong key status in case of fail The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@dlab.apache.org For additional commands, e-mail: commits-h...@dlab.apache.org
[incubator-dlab] branch develop updated (31518ab -> d2deca8)
This is an automated email from the ASF dual-hosted git repository.
bhliva pushed a change to branch develop
in repository https://gitbox.apache.org/repos/asf/incubator-dlab.git.
from 31518ab Merge pull request #81 from ioleksandr/DLAB-551
add 06f1da6 DLAB-582 added API for getting DLab version
new d2deca8 Merge pull request #82 from apache/feature-DLAB-582
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
...CodeResponse.java => InfrastructureMetaInfoDTO.java} | 12 +++-
services/self-service/pom.xml | 6 ++
.../resources/InfrastructureInfoResource.java | 9 +
.../backendapi/service/InfrastructureInfoService.java | 3 +++
.../service/impl/InfrastructureInfoServiceBase.java | 11 +++
.../resources/InfrastructureInfoResourceTest.java | 17 +
6 files changed, 53 insertions(+), 5 deletions(-)
copy
services/dlab-model/src/main/java/com/epam/dlab/dto/{gcp/auth/GcpOauth2AuthorizationCodeResponse.java
=> InfrastructureMetaInfoDTO.java} (81%)
-
To unsubscribe, e-mail: commits-unsubscr...@dlab.apache.org
For additional commands, e-mail: commits-h...@dlab.apache.org
[incubator-dlab] 01/01: Merge pull request #82 from apache/feature-DLAB-582
This is an automated email from the ASF dual-hosted git repository. bhliva pushed a commit to branch develop in repository https://gitbox.apache.org/repos/asf/incubator-dlab.git commit d2deca88dc9e70171bfc0e65c9d73120c480334a Merge: 31518ab 06f1da6 Author: bhliva AuthorDate: Fri Apr 5 12:30:03 2019 +0300 Merge pull request #82 from apache/feature-DLAB-582 DLAB-582 added API for getting DLab version .../com/epam/dlab/dto/InfrastructureMetaInfoDTO.java} | 16 +--- services/self-service/pom.xml | 6 ++ .../resources/InfrastructureInfoResource.java | 9 + .../backendapi/service/InfrastructureInfoService.java | 3 +++ .../service/impl/InfrastructureInfoServiceBase.java | 11 +++ .../resources/InfrastructureInfoResourceTest.java | 17 + 6 files changed, 55 insertions(+), 7 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@dlab.apache.org For additional commands, e-mail: commits-h...@dlab.apache.org
[incubator-dlab] branch feature-DLAB-582 created (now 06f1da6)
This is an automated email from the ASF dual-hosted git repository. bhliva pushed a change to branch feature-DLAB-582 in repository https://gitbox.apache.org/repos/asf/incubator-dlab.git. at 06f1da6 DLAB-582 added API for getting DLab version This branch includes the following new commits: new 06f1da6 DLAB-582 added API for getting DLab version The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@dlab.apache.org For additional commands, e-mail: commits-h...@dlab.apache.org
[incubator-dlab] 01/01: DLAB-582 added API for getting DLab version
This is an automated email from the ASF dual-hosted git repository.
bhliva pushed a commit to branch feature-DLAB-582
in repository https://gitbox.apache.org/repos/asf/incubator-dlab.git
commit 06f1da6f14fecc7a1572d2cb9a79d46b57ecc5a5
Author: bhliva
AuthorDate: Fri Apr 5 12:28:19 2019 +0300
DLAB-582 added API for getting DLab version
---
.../com/epam/dlab/dto/InfrastructureMetaInfoDTO.java} | 16 +---
services/self-service/pom.xml | 6 ++
.../resources/InfrastructureInfoResource.java | 9 +
.../backendapi/service/InfrastructureInfoService.java | 3 +++
.../service/impl/InfrastructureInfoServiceBase.java | 11 +++
.../resources/InfrastructureInfoResourceTest.java | 17 +
6 files changed, 55 insertions(+), 7 deletions(-)
diff --git
a/services/self-service/src/main/java/com/epam/dlab/backendapi/service/InfrastructureInfoService.java
b/services/dlab-model/src/main/java/com/epam/dlab/dto/InfrastructureMetaInfoDTO.java
similarity index 69%
copy from
services/self-service/src/main/java/com/epam/dlab/backendapi/service/InfrastructureInfoService.java
copy to
services/dlab-model/src/main/java/com/epam/dlab/dto/InfrastructureMetaInfoDTO.java
index a3e08d2..c873fab 100644
---
a/services/self-service/src/main/java/com/epam/dlab/backendapi/service/InfrastructureInfoService.java
+++
b/services/dlab-model/src/main/java/com/epam/dlab/dto/InfrastructureMetaInfoDTO.java
@@ -17,13 +17,15 @@
* under the License.
*/
-package com.epam.dlab.backendapi.service;
+package com.epam.dlab.dto;
-import com.epam.dlab.backendapi.resources.dto.HealthStatusPageDTO;
-import com.epam.dlab.backendapi.resources.dto.InfrastructureInfo;
+import lombok.Builder;
+import lombok.Data;
-public interface InfrastructureInfoService {
- InfrastructureInfo getUserResources(String user);
-
- HealthStatusPageDTO getHeathStatus(String user, boolean fullReport,
boolean isAdmin);
+@Data
+@Builder
+public class InfrastructureMetaInfoDTO {
+ private final String branch;
+ private final String version;
+ private final String commit;
}
diff --git a/services/self-service/pom.xml b/services/self-service/pom.xml
index 9ff451b..a71c9c4 100644
--- a/services/self-service/pom.xml
+++ b/services/self-service/pom.xml
@@ -134,6 +134,12 @@
oshi-core
${com.github.oshi.core.version}
+
+com.jcabi
+jcabi-manifests
+1.1
+
+
com.smoketurner
diff --git
a/services/self-service/src/main/java/com/epam/dlab/backendapi/resources/InfrastructureInfoResource.java
b/services/self-service/src/main/java/com/epam/dlab/backendapi/resources/InfrastructureInfoResource.java
index dfedb8e..6b44c33 100644
---
a/services/self-service/src/main/java/com/epam/dlab/backendapi/resources/InfrastructureInfoResource.java
+++
b/services/self-service/src/main/java/com/epam/dlab/backendapi/resources/InfrastructureInfoResource.java
@@ -88,4 +88,13 @@ public class InfrastructureInfoResource {
return
infrastructureInfoService.getUserResources(userInfo.getName());
}
+
+ @GET
+ @Path("/meta")
+ @ApiOperation("Return metainfo regarding application version etc")
+ public Response getVersion(@ApiParam(hidden = true) @Auth UserInfo
userInfo) {
+ return
Response.ok(infrastructureInfoService.getInfrastructureMetaInfo())
+ .build();
+
+ }
}
diff --git
a/services/self-service/src/main/java/com/epam/dlab/backendapi/service/InfrastructureInfoService.java
b/services/self-service/src/main/java/com/epam/dlab/backendapi/service/InfrastructureInfoService.java
index a3e08d2..b54edb9 100644
---
a/services/self-service/src/main/java/com/epam/dlab/backendapi/service/InfrastructureInfoService.java
+++
b/services/self-service/src/main/java/com/epam/dlab/backendapi/service/InfrastructureInfoService.java
@@ -21,9 +21,12 @@ package com.epam.dlab.backendapi.service;
import com.epam.dlab.backendapi.resources.dto.HealthStatusPageDTO;
import com.epam.dlab.backendapi.resources.dto.InfrastructureInfo;
+import com.epam.dlab.dto.InfrastructureMetaInfoDTO;
public interface InfrastructureInfoService {
InfrastructureInfo getUserResources(String user);
HealthStatusPageDTO getHeathStatus(String user, boolean fullReport,
boolean isAdmin);
+
+ InfrastructureMetaInfoDTO getInfrastructureMetaInfo();
}
diff --git
a/services/self-service/src/main/java/com/epam/dlab/backendapi/service/impl/InfrastructureInfoServiceBase.java
b/services/self-service/src/main/java/com/epam/dlab/backendapi/service/impl/InfrastructureInfoServiceBase.java
index 0f0584c..6981531 100644
---
a/services/self-service/src/main/java/com/epam/dlab/backendapi/service/impl/InfrastructureInfoServiceBase.java
+++
