[MediaWiki-commits] [Gerrit] operations/puppet[production]: Revert "Revert "Deploy MjoLniR with new deploy repository""

2017-12-06 Thread Gehel (Code Review) 
Gehel has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/394002 )

Change subject: Revert "Revert "Deploy MjoLniR with new deploy repository""
..


Revert "Revert "Deploy MjoLniR with new deploy repository""

The problems with mjolnir vs MjoLniR have been resolved, it
required adjustments to scap.cfg in the repo.

This is a standard revert with one addition, we now create an
empty directory /etc/mjolnir owned by deploy-service. This gives
the deploy repo a sane place to install a configuration script
to that can be auto-magicaly found by mjolnir.

This reverts commit 6a7753a14ac3cb66593eabbad30e8ac72e184751.

Change-Id: I599341bd16ecba0a2b8d8132fde6fe3d1443d754
---
M hieradata/role/common/deployment_server.yaml
A modules/mjolnir/manifests/init.pp
M modules/profile/manifests/mjolnir/kafka_daemon.pp
M modules/profile/templates/mjolnir/kafka-daemon.service.erb
M modules/role/manifests/elasticsearch/analytics.pp
5 files changed, 28 insertions(+), 14 deletions(-)

Approvals:
  EBernhardson: Looks good to me, but someone else must approve
  jenkins-bot: Verified
  Gehel: Looks good to me, approved



diff --git a/hieradata/role/common/deployment_server.yaml 
b/hieradata/role/common/deployment_server.yaml
index 4f899a6..5c5e154 100644
--- a/hieradata/role/common/deployment_server.yaml
+++ b/hieradata/role/common/deployment_server.yaml
@@ -173,8 +173,8 @@
   # Netbox software
   netbox/deploy:
 repository: operations/software/netbox-deploy
-  relforge/mjolnir:
-repository: search/MjoLniR
+  search/mjolnir/deploy:
+repository: search/MjoLniR/deploy
   statsv/statsv:
 repository: analytics/statsv
   "docker-pkg/deploy":
diff --git a/modules/mjolnir/manifests/init.pp 
b/modules/mjolnir/manifests/init.pp
new file mode 100644
index 000..07e559f
--- /dev/null
+++ b/modules/mjolnir/manifests/init.pp
@@ -0,0 +1,21 @@
+# = Class: mjolnir
+#
+# This class installs the MjoLniR (Machine Learned Ranking) data
+# processing package.
+#
+class mjolnir {
+require_package('virtualenv', 'zip')
+
+file { '/etc/mjolnir':
+ensure => 'directory',
+owner  => 'deploy-service',
+group  => 'deploy-service',
+mode   => '0755',
+}
+
+scap::target { 'search/mjolnir/deploy':
+deploy_user => 'deploy-service',
+}
+}
+
+
diff --git a/modules/profile/manifests/mjolnir/kafka_daemon.pp 
b/modules/profile/manifests/mjolnir/kafka_daemon.pp
index c1dcaf9..ba8cc1c 100644
--- a/modules/profile/manifests/mjolnir/kafka_daemon.pp
+++ b/modules/profile/manifests/mjolnir/kafka_daemon.pp
@@ -9,18 +9,11 @@
 # it is named just 'eqiad'.
 $kafka_config = kafka_config('eqiad'),
 ) {
-scap::target { 'relforge/mjolnir':
-  deploy_user => 'deploy-service',
-}
-
-# This is a limited subset of what the full mjolnir package requires 
because
-# the daemon is a small part of the overall application. The daemon only 
needs
-# to read/write kafka topics and send requests to localhost.
-require_package('python-kafka', 'python-requests')
+class { 'mjolnir': }
 
 systemd::service { 'mjolnir-kafka-daemon':
 content => template('profile/mjolnir/kafka-daemon.service.erb'),
-require => Scap::Target['relforge/mjolnir'],
+require => Class['mjolnir'],
 }
 
 }
diff --git a/modules/profile/templates/mjolnir/kafka-daemon.service.erb 
b/modules/profile/templates/mjolnir/kafka-daemon.service.erb
index b6947ba..7e40bd8 100644
--- a/modules/profile/templates/mjolnir/kafka-daemon.service.erb
+++ b/modules/profile/templates/mjolnir/kafka-daemon.service.erb
@@ -5,9 +5,7 @@
 [Service]
 User=nobody
 Group=nogroup
-WorkingDirectory=/srv/deployment/relforge/mjolnir
-Environment=PYTHONPATH=/srv/deployment/relforge/mjolnir
-ExecStart=/usr/bin/python2 
/srv/deployment/relforge/mjolnir/mjolnir/cli/kafka_daemon.py --brokers <%= 
@kafka_config['brokers']['string'] %>
+ExecStart=/srv/deployment/search/mjolnir/venv/bin/mjolnir-utilities.py 
kafka_daemon --brokers <%= @kafka_config['brokers']['string'] %>
 StandardInput=null
 StandardOutput=journal
 StandardError=journal
diff --git a/modules/role/manifests/elasticsearch/analytics.pp 
b/modules/role/manifests/elasticsearch/analytics.pp
index a46391e..66d7789 100644
--- a/modules/role/manifests/elasticsearch/analytics.pp
+++ b/modules/role/manifests/elasticsearch/analytics.pp
@@ -1,5 +1,7 @@
 # Supports CirrusSearch usage on the analytics cluster
 class role::elasticsearch::analytics {
+class { 'mjolnir': }
+
 # wikimedia/discovery/analytics will be deployed to this node
 scap::target { 'wikimedia/discovery/analytics':
 deploy_user => 'deploy-service',

-- 
To view, visit https://gerrit.wikimedia.org/r/394002
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I599341bd16ecba0a2b8d8132fde6fe3d1443d754
Gerrit-PatchSet: 5
Gerrit-Project:

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Revert "Revert "Deploy MjoLniR with new deploy repository""

2017-11-28 Thread EBernhardson (Code Review) 
EBernhardson has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/394002 )

Change subject: Revert "Revert "Deploy MjoLniR with new deploy repository""
..

Revert "Revert "Deploy MjoLniR with new deploy repository""

The problems with mjolnir vs MjoLniR have been resolved, it
required adjustments to scap.cfg in the repo.

This is a standard revert with one addition, we now create an
empty directory /etc/mjolnir owned by deploy-service. This gives
the deploy repo a sane place to install a configuration script
to that can be auto-magicaly found by mjolnir.

This reverts commit 6a7753a14ac3cb66593eabbad30e8ac72e184751.

Change-Id: I599341bd16ecba0a2b8d8132fde6fe3d1443d754
---
M hieradata/role/common/deployment_server.yaml
A modules/mjolnir/manifests/init.pp
M modules/profile/manifests/mjolnir/kafka_daemon.pp
M modules/profile/templates/mjolnir/kafka-daemon.service.erb
M modules/role/manifests/elasticsearch/analytics.pp
5 files changed, 28 insertions(+), 14 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/02/394002/1

diff --git a/hieradata/role/common/deployment_server.yaml 
b/hieradata/role/common/deployment_server.yaml
index 7c55e05..7444810 100644
--- a/hieradata/role/common/deployment_server.yaml
+++ b/hieradata/role/common/deployment_server.yaml
@@ -173,8 +173,8 @@
   # Netbox software
   netbox/deploy:
 repository: operations/software/netbox-deploy
-  relforge/mjolnir:
-repository: search/MjoLniR
+  search/mjolnir/deploy:
+repository: search/MjoLniR/deploy
   statsv/statsv:
 repository: analytics/statsv
   "docker-pkg/deploy":
diff --git a/modules/mjolnir/manifests/init.pp 
b/modules/mjolnir/manifests/init.pp
new file mode 100644
index 000..e4bb89f
--- /dev/null
+++ b/modules/mjolnir/manifests/init.pp
@@ -0,0 +1,21 @@
+# = Class: mjolnir
+#
+# This class installs the MjoLniR (Machine Learned Ranking) data
+# processing package.
+#
+class mjolnir {
+require_package('virtualenv', 'zip')
+
+file { '/etc/mjolnir':
+ensure => 'directory',
+user   => 'deploy-service',
+group  => 'deploy-service',
+mode   => 0755
+}
+
+scap::target { 'search/mjolnir/deploy':
+deploy_user => 'deploy-service',
+}
+}
+
+
diff --git a/modules/profile/manifests/mjolnir/kafka_daemon.pp 
b/modules/profile/manifests/mjolnir/kafka_daemon.pp
index c1dcaf9..479187c 100644
--- a/modules/profile/manifests/mjolnir/kafka_daemon.pp
+++ b/modules/profile/manifests/mjolnir/kafka_daemon.pp
@@ -9,18 +9,11 @@
 # it is named just 'eqiad'.
 $kafka_config = kafka_config('eqiad'),
 ) {
-scap::target { 'relforge/mjolnir':
-  deploy_user => 'deploy-service',
-}
-
-# This is a limited subset of what the full mjolnir package requires 
because
-# the daemon is a small part of the overall application. The daemon only 
needs
-# to read/write kafka topics and send requests to localhost.
-require_package('python-kafka', 'python-requests')
+class { 'mjolnir': }
 
 systemd::service { 'mjolnir-kafka-daemon':
 content => template('profile/mjolnir/kafka-daemon.service.erb'),
-require => Scap::Target['relforge/mjolnir'],
+require => Scap::Target['search/mjolnir/deploy'],
 }
 
 }
diff --git a/modules/profile/templates/mjolnir/kafka-daemon.service.erb 
b/modules/profile/templates/mjolnir/kafka-daemon.service.erb
index b6947ba..7e40bd8 100644
--- a/modules/profile/templates/mjolnir/kafka-daemon.service.erb
+++ b/modules/profile/templates/mjolnir/kafka-daemon.service.erb
@@ -5,9 +5,7 @@
 [Service]
 User=nobody
 Group=nogroup
-WorkingDirectory=/srv/deployment/relforge/mjolnir
-Environment=PYTHONPATH=/srv/deployment/relforge/mjolnir
-ExecStart=/usr/bin/python2 
/srv/deployment/relforge/mjolnir/mjolnir/cli/kafka_daemon.py --brokers <%= 
@kafka_config['brokers']['string'] %>
+ExecStart=/srv/deployment/search/mjolnir/venv/bin/mjolnir-utilities.py 
kafka_daemon --brokers <%= @kafka_config['brokers']['string'] %>
 StandardInput=null
 StandardOutput=journal
 StandardError=journal
diff --git a/modules/role/manifests/elasticsearch/analytics.pp 
b/modules/role/manifests/elasticsearch/analytics.pp
index a46391e..66d7789 100644
--- a/modules/role/manifests/elasticsearch/analytics.pp
+++ b/modules/role/manifests/elasticsearch/analytics.pp
@@ -1,5 +1,7 @@
 # Supports CirrusSearch usage on the analytics cluster
 class role::elasticsearch::analytics {
+class { 'mjolnir': }
+
 # wikimedia/discovery/analytics will be deployed to this node
 scap::target { 'wikimedia/discovery/analytics':
 deploy_user => 'deploy-service',

-- 
To view, visit https://gerrit.wikimedia.org/r/394002
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I599341bd16ecba0a2b8d8132fde6fe3d1443d754
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet