[MediaWiki-commits] [Gerrit] operations/puppet[production]: Add the cp-jobqueue profile

2017-09-08 Thread Alexandros Kosiaris (Code Review) 
Alexandros Kosiaris has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/376707 )

Change subject: Add the cp-jobqueue profile
..


Add the cp-jobqueue profile

Bug: T175281
Change-Id: I78a09d0d99ae20a38d397da8f8a1faf9dd015fe3
---
M hieradata/role/common/deployment_server.yaml
M hieradata/role/common/scb.yaml
A modules/profile/manifests/cpjobqueue.pp
3 files changed, 58 insertions(+), 0 deletions(-)

Approvals:
  Alexandros Kosiaris: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/hieradata/role/common/deployment_server.yaml 
b/hieradata/role/common/deployment_server.yaml
index ba38195..f82f4d5 100644
--- a/hieradata/role/common/deployment_server.yaml
+++ b/hieradata/role/common/deployment_server.yaml
@@ -91,6 +91,7 @@
 #lvs_service: changeprop
   citoid/deploy: {}
 #lvs_service: citoid
+  cp-jobqueue: {}
   cxserver/deploy: {}
 #lvs_service: cxserver
   dumps/dumps:
diff --git a/hieradata/role/common/scb.yaml b/hieradata/role/common/scb.yaml
index 6f12fb2..f2e4eec 100644
--- a/hieradata/role/common/scb.yaml
+++ b/hieradata/role/common/scb.yaml
@@ -91,3 +91,7 @@
 
 # Recommendation API
 profile::recommendation_api::wdqs_uri: "http://wdqs.discovery.wmnet;
+
+# ChangePropagation for JobQueue
+profile::cpjobqueue::redis_path: "/var/run/nutcracker/redis_%{::site}.sock"
+profile::cpjobqueue::jobrunner_host: https://jobrunner.discovery.wmnet
diff --git a/modules/profile/manifests/cpjobqueue.pp 
b/modules/profile/manifests/cpjobqueue.pp
new file mode 100644
index 000..8ca37b9
--- /dev/null
+++ b/modules/profile/manifests/cpjobqueue.pp
@@ -0,0 +1,53 @@
+# == Class: profile::cpjobqueue
+#
+# profile::cpjobqueue is the profile class for ChangeProp for JobQueue, a
+# special instance of ChangePropagation that is used to pick up job definitions
+# from the main Kafka cluster (EventBus) and sends them for execution on the
+# jobrunners.
+#
+# === Parameters
+#
+# [*redis_path*]
+#   The path to Redis' UNIX socket. Note that the password is taken directly
+#   from the private repository with secrets.
+#
+# [*jobrunner_host*]
+#   The address of the LVS end point for jobrunners.
+#
+# [*kafka_msg_max_bytes*]
+#   The maximum allowed message size in Kafka. This value *must* match the
+#   brokers' configuration of the same name.
+#
+class profile::cpjobqueue(
+$redis_path  = hiera('profile::cpjobqueue::redis_path'),
+$jobrunner_host  = hiera('profile::cpjobqueue::jobrunner_host'),
+$kafka_msg_max_bytes = hiera('kafka_message_max_bytes'),
+) {
+
+include ::passwords::redis
+require ::changeprop::packages
+
+$kafka_config = kafka_config('main')
+
+service::node { 'cp-jobqueue':
+repo  => 'cp-jobqueue',
+port  => 7200,
+healthcheck_url   => '',
+has_spec  => true,
+deployment=> 'scap3',
+deployment_config => true,
+deployment_vars   => {
+broker_list => $kafka_config['brokers']['string'],
+site=> $::site,
+redis_path  => $redis_path,
+redis_pass  => $::passwords::redis::main_password,
+jobrunner_uri   => "${jobrunner_host}/rpc/RunSingleJob.php",
+kafka_max_bytes => $kafka_msg_max_bytes,
+},
+auto_refresh  => false,
+init_restart  => false,
+environment   => {
+'UV_THREADPOOL_SIZE' => 128,
+},
+}
+}

-- 
To view, visit https://gerrit.wikimedia.org/r/376707
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I78a09d0d99ae20a38d397da8f8a1faf9dd015fe3
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Mobrovac 
Gerrit-Reviewer: Alexandros Kosiaris 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: Mobrovac 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Add the cp-jobqueue profile

2017-09-08 Thread Mobrovac (Code Review) 
Mobrovac has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/376707 )

Change subject: Add the cp-jobqueue profile
..

Add the cp-jobqueue profile

Bug: T175281
Change-Id: I78a09d0d99ae20a38d397da8f8a1faf9dd015fe3
---
M hieradata/role/common/deployment_server.yaml
M hieradata/role/common/scb.yaml
A modules/profile/manifests/cpjobqueue.pp
3 files changed, 62 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/07/376707/1

diff --git a/hieradata/role/common/deployment_server.yaml 
b/hieradata/role/common/deployment_server.yaml
index ba38195..f82f4d5 100644
--- a/hieradata/role/common/deployment_server.yaml
+++ b/hieradata/role/common/deployment_server.yaml
@@ -91,6 +91,7 @@
 #lvs_service: changeprop
   citoid/deploy: {}
 #lvs_service: citoid
+  cp-jobqueue: {}
   cxserver/deploy: {}
 #lvs_service: cxserver
   dumps/dumps:
diff --git a/hieradata/role/common/scb.yaml b/hieradata/role/common/scb.yaml
index 6f12fb2..f2e4eec 100644
--- a/hieradata/role/common/scb.yaml
+++ b/hieradata/role/common/scb.yaml
@@ -91,3 +91,7 @@
 
 # Recommendation API
 profile::recommendation_api::wdqs_uri: "http://wdqs.discovery.wmnet;
+
+# ChangePropagation for JobQueue
+profile::cpjobqueue::redis_path: "/var/run/nutcracker/redis_%{::site}.sock"
+profile::cpjobqueue::jobrunner_host: https://jobrunner.discovery.wmnet
diff --git a/modules/profile/manifests/cpjobqueue.pp 
b/modules/profile/manifests/cpjobqueue.pp
new file mode 100644
index 000..0592319
--- /dev/null
+++ b/modules/profile/manifests/cpjobqueue.pp
@@ -0,0 +1,57 @@
+# == Class: profile::cpjobqueue
+# 
+# profile::cpjobqueue is the profile class for ChangeProp for JobQueue, a
+# special instance of ChangePropagation that is used to pick up job definitions
+# from the main Kafka cluster (EventBus) and sends them for execution on the
+# jobrunners.
+#
+# === Parameters
+#
+# [*redis_path*]
+#   The path to Redis' UNIX socket. Note that the password is taken directly
+#   from the private repository with secrets.
+#
+# [*jobrunner_host*]
+#   The address of the LVS end point for jobrunners.
+#
+# [*kafka_msg_max_bytes*]
+#   The maximum allowed message size in Kafka. This value *must* match the
+#   brokers' configuration of the same name.
+#
+class profile::cpjobqueue(
+$redis_path  = hiera('profile::cpjobqueue::redis_path'),
+$jobrunner_host  = hiera('profile::cpjobqueue::jobrunner_host'),
+$kafka_msg_max_bytes = hiera('kafka_message_max_bytes'),
+) {
+
+include ::passwords::redis
+require ::changeprop::packages
+
+$kafka_config = kafka_config('main')
+
+system::role { 'cp-jobqueue':
+description => 'ChangePropagation instance for JobQueue jobs',
+}
+
+service::node { 'cp-jobqueue':
+repo  => 'cp-jobqueue',
+port  => 7200,
+healthcheck_url   => '',
+has_spec  => true,
+deployment=> 'scap3',
+deployment_config => true,
+deployment_vars   => {
+broker_list => $kafka_config['brokers']['string'],
+site=> $::site,
+redis_path  => $redis_path,
+redis_pass  => $::passwords::redis::main_password,
+jobrunner_uri   => "${jobrunner_host}/rpc/RunSingleJob.php",
+kafka_max_bytes => $kafka_msg_max_bytes,
+},
+auto_refresh  => false,
+init_restart  => false,
+environment   => {
+'UV_THREADPOOL_SIZE' => 128,
+},
+}
+}

-- 
To view, visit https://gerrit.wikimedia.org/r/376707
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I78a09d0d99ae20a38d397da8f8a1faf9dd015fe3
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Mobrovac 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits