[MediaWiki-commits] [Gerrit] operations/puppet[production]: Allow search clusters to reindex from eachother

2017-06-15 Thread Gehel (Code Review) 
Gehel has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/344517 )

Change subject: Allow search clusters to reindex from eachother
..


Allow search clusters to reindex from eachother

Whitelists the following potential reindex source/destinations
with the elasticsearch reindex API:

* codfw -> eqiad
* codfw -> relforge
* eqiad -> codfw
* eqiad -> relforge

Change-Id: I2670f5a5fbaa2940a84e7bb20b5ce66856edc7ce
---
M hieradata/labs/deployment-prep/common.yaml
M hieradata/role/codfw/elasticsearch/cirrus.yaml
M hieradata/role/common/elasticsearch/relforge.yaml
M hieradata/role/eqiad/elasticsearch/cirrus.yaml
M modules/elasticsearch/manifests/init.pp
M modules/elasticsearch/templates/elasticsearch_5.yml.erb
M modules/profile/manifests/elasticsearch.pp
7 files changed, 23 insertions(+), 0 deletions(-)

Approvals:
  jenkins-bot: Verified
  DCausse: Looks good to me, but someone else must approve
  Gehel: Looks good to me, approved



diff --git a/hieradata/labs/deployment-prep/common.yaml 
b/hieradata/labs/deployment-prep/common.yaml
index 0ab1324..c5aaf4b 100644
--- a/hieradata/labs/deployment-prep/common.yaml
+++ b/hieradata/labs/deployment-prep/common.yaml
@@ -239,6 +239,7 @@
 "profile::elasticsearch::cluster_name": beta-search
 "profile::elasticsearch::graylog_hosts":
   - deployment-logstash2.deployment-prep.eqiad.wmflabs
+"profile::elasticsearch::reindex_remote_whitelist": ''
 role::logstash::collector::statsd_host: labmon1001.eqiad.wmnet
 "redis::shards":
   jobqueue: _shards
diff --git a/hieradata/role/codfw/elasticsearch/cirrus.yaml 
b/hieradata/role/codfw/elasticsearch/cirrus.yaml
index 190aa99..e6d73ed 100644
--- a/hieradata/role/codfw/elasticsearch/cirrus.yaml
+++ b/hieradata/role/codfw/elasticsearch/cirrus.yaml
@@ -64,3 +64,6 @@
 
 profile::elasticsearch::certificate_name: 'search.svc.codfw.wmnet'
 profile::base::dns_alt_names: 'search.svc.codfw.wmnet'
+
+# Allow reindex from eqiad to codfw
+profile::elasticsearch::reindex_remote_whitelist: 'search.svc.eqiad.wmnet:9243'
diff --git a/hieradata/role/common/elasticsearch/relforge.yaml 
b/hieradata/role/common/elasticsearch/relforge.yaml
index 890650b..6fd4425 100644
--- a/hieradata/role/common/elasticsearch/relforge.yaml
+++ b/hieradata/role/common/elasticsearch/relforge.yaml
@@ -15,3 +15,6 @@
 profile::elasticsearch::awareness_attributes: ''
 profile::elasticsearch::rack: ''
 profile::elasticsearch::row: ''
+
+# Allow reindex from eqiad or codfw to relforge
+profile::elasticsearch::reindex_remote_whitelist: 
'search.svc.eqiad.wmnet:9243,search.svc.codfw.wmnet:9243'
diff --git a/hieradata/role/eqiad/elasticsearch/cirrus.yaml 
b/hieradata/role/eqiad/elasticsearch/cirrus.yaml
index 695c4ca..03a5e36 100644
--- a/hieradata/role/eqiad/elasticsearch/cirrus.yaml
+++ b/hieradata/role/eqiad/elasticsearch/cirrus.yaml
@@ -65,3 +65,6 @@
 
 profile::elasticsearch::certificate_name: 'search.svc.eqiad.wmnet'
 profile::base::dns_alt_names: 'search.svc.eqiad.wmnet'
+
+# Allow reindex from codfw to eqiad
+profile::elasticsearch::reindex_remote_whitelist: 'search.svc.codfw.wmnet:9243'
diff --git a/modules/elasticsearch/manifests/init.pp 
b/modules/elasticsearch/manifests/init.pp
index 1711e7b..9b5e328 100644
--- a/modules/elasticsearch/manifests/init.pp
+++ b/modules/elasticsearch/manifests/init.pp
@@ -72,6 +72,10 @@
 #want the API exposed outside of localhost, so using just localhost
 #is useful in those cases.
 #Default: true (use all hosts defined in unicast_hosts)
+# - $reindex_remote_whitelist: set to a comma delimited list of allowed remote
+#host and port combinations (e.g. otherhost:9243, another:9243,
+#127.0.10.*:9243, localhost:*). Scheme is ignored by the whitelist - 
only host
+#and port are used. Defaults to undef, which means no remote reindex 
can occur.
 #
 # == Sample usage:
 #
@@ -109,6 +113,7 @@
 $version = 5,
 $search_shard_count_limit = 1000,
 $curator_uses_unicast_hosts = true,
+$reindex_remote_whitelist = undef,
 ) {
 
 # Check arguments
diff --git a/modules/elasticsearch/templates/elasticsearch_5.yml.erb 
b/modules/elasticsearch/templates/elasticsearch_5.yml.erb
index 419cdcb..b81932a 100644
--- a/modules/elasticsearch/templates/elasticsearch_5.yml.erb
+++ b/modules/elasticsearch/templates/elasticsearch_5.yml.erb
@@ -432,3 +432,9 @@
 index.load_fixed_bitset_filters_eagerly: false
 
 <% end -%>
+
+<% if @reindex_remote_whitelist and @reindex_remote_whitelist != '' %>
+##
+# Hosts allowed as data sources for reindexing
+reindex.remote.whitelist: "<%= @reindex_remote_whitelist %>"
+<% end %>
diff --git a/modules/profile/manifests/elasticsearch.pp 
b/modules/profile/manifests/elasticsearch.pp
index 8a7bbb2..7ace2d5 100644
--- a/modules/profile/manifests/elasticsearch.pp
+++ b/modules/profile/manifests/elasticsearch.pp
@@ -26,6 +26,7 @@
 $certificate_name =

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Allow search clusters to reindex from eachother

2017-03-23 Thread EBernhardson (Code Review) 
EBernhardson has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/344517 )

Change subject: Allow search clusters to reindex from eachother
..

Allow search clusters to reindex from eachother

Whitelists the following potential reindex source/destinations
with the elasticsearch reindex API:

* codfw -> eqiad
* codfw -> relforge
* eqiad -> codfw
* eqiad -> relforge

Change-Id: I2670f5a5fbaa2940a84e7bb20b5ce66856edc7ce
---
M hieradata/role/codfw/elasticsearch/cirrus.yaml
M hieradata/role/common/elasticsearch/relforge.yaml
M hieradata/role/eqiad/elasticsearch/cirrus.yaml
M modules/elasticsearch/templates/elasticsearch_5.yml.erb
4 files changed, 15 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/17/344517/1

diff --git a/hieradata/role/codfw/elasticsearch/cirrus.yaml 
b/hieradata/role/codfw/elasticsearch/cirrus.yaml
index cd68fb5..ea3f184 100644
--- a/hieradata/role/codfw/elasticsearch/cirrus.yaml
+++ b/hieradata/role/codfw/elasticsearch/cirrus.yaml
@@ -63,3 +63,6 @@
 
 elasticsearch::https::certificate_name: 'search.svc.codfw.wmnet'
 profile::base::dns_alt_names:   'search.svc.codfw.wmnet'
+
+# Allow reindex from eqiad to codfw
+elasticsearch::reindex_remote_whitelist: 'search.svc.eqiad.wmnet:9200'
diff --git a/hieradata/role/common/elasticsearch/relforge.yaml 
b/hieradata/role/common/elasticsearch/relforge.yaml
index f2138f7..086ec0c 100644
--- a/hieradata/role/common/elasticsearch/relforge.yaml
+++ b/hieradata/role/common/elasticsearch/relforge.yaml
@@ -17,3 +17,6 @@
 # once all elasticsearch clusters are upgraded to version 5, we will move
 # elasticsearch to our main repo and remove this configuration
 apt::use_experimental: true
+
+# Allow reindex from eqiad or codfw to relforge
+elasticsearch::reindex_remote_whitelist: 
'search.svc.eqiad.wmnet:9200,search.svc.codfw.wmnet:9200'
diff --git a/hieradata/role/eqiad/elasticsearch/cirrus.yaml 
b/hieradata/role/eqiad/elasticsearch/cirrus.yaml
index 65765c2..dce74b2 100644
--- a/hieradata/role/eqiad/elasticsearch/cirrus.yaml
+++ b/hieradata/role/eqiad/elasticsearch/cirrus.yaml
@@ -64,3 +64,6 @@
 
 elasticsearch::https::certificate_name: 'search.svc.eqiad.wmnet'
 profile::base::dns_alt_names:   'search.svc.eqiad.wmnet'
+
+# Allow reindex from eqiad to codfw
+elasticsearch::reindex_remote_whitelist: 'search.svc.codfw.wmnet:9200'
diff --git a/modules/elasticsearch/templates/elasticsearch_5.yml.erb 
b/modules/elasticsearch/templates/elasticsearch_5.yml.erb
index 87aea61..e2d0bc9 100644
--- a/modules/elasticsearch/templates/elasticsearch_5.yml.erb
+++ b/modules/elasticsearch/templates/elasticsearch_5.yml.erb
@@ -426,3 +426,9 @@
 index.load_fixed_bitset_filters_eagerly: false
 
 <% end -%>
+
+##
+# Hosts allowed as data sources for reindexing
+<% if reindex_remote_whitelist %>
+reindex.remote.whitelist: "<%= @reindex_remote_whitelist %>"
+<% end %>

-- 
To view, visit https://gerrit.wikimedia.org/r/344517
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I2670f5a5fbaa2940a84e7bb20b5ce66856edc7ce
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: EBernhardson 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits