Elukey has submitted this change and it was merged.
Change subject: Introduce the Imply Pivot UI's module and role
......................................................................
Introduce the Imply Pivot UI's module and role
This nodejs service is a UI to explore the data stored in the
Analytics's Druid cluster.
Remarks:
1) We tried to use service-runner but in the end we were not
able to integrate its code with Pivot's one, so we'll work on it
(maybe with upstream) later on.
2) For clarity and simplicity the only supported init is systemd so
Pivot needs to be deployed to a Debian based host.
3) Scap configuration has been provided. After a chat with Marko
the proposal is to have analytics_deploy to own the files and
the deployment key, but let the pivot user to run the nodejs service.
4) The best thing to do, since the target host also runs other stuff,
is to use a Firejail profile.
Bug: T138262
Change-Id: I0898ddf86bc8074b772cfa26d04a90a76cac52ce
---
A hieradata/role/common/analytics_cluster/druid/pivot.yaml
M hieradata/role/common/deployment/server.yaml
M manifests/site.pp
A modules/pivot/files/pivot.profile.firejail
A modules/pivot/manifests/init.pp
A modules/pivot/templates/initscripts/pivot.systemd.erb
A modules/role/manifests/analytics_cluster/druid/pivot.pp
M modules/statistics/manifests/sites/pivot.pp
8 files changed, 161 insertions(+), 7 deletions(-)
Approvals:
Elukey: Looks good to me, approved
Muehlenhoff: Looks good to me, but someone else must approve
Alexandros Kosiaris: Looks good to me, but someone else must approve
jenkins-bot: Verified
diff --git a/hieradata/role/common/analytics_cluster/druid/pivot.yaml
b/hieradata/role/common/analytics_cluster/druid/pivot.yaml
new file mode 100644
index 0000000..e03ae79
--- /dev/null
+++ b/hieradata/role/common/analytics_cluster/druid/pivot.yaml
@@ -0,0 +1 @@
+pivot::druid_broker: 'druid1001.eqiad.wmnet'
\ No newline at end of file
diff --git a/hieradata/role/common/deployment/server.yaml
b/hieradata/role/common/deployment/server.yaml
index 7a12076..44f7725 100644
--- a/hieradata/role/common/deployment/server.yaml
+++ b/hieradata/role/common/deployment/server.yaml
@@ -88,6 +88,8 @@
kartotherian/deploy:
repository: maps/kartotherian/deploy
# lvs_service: kartotherian
+ analytics/pivot:
+ repository: analytics/pivot/deploy
mathoid/deploy: {}
# lvs_service: mathoid
mobileapps/deploy: {}
diff --git a/manifests/site.pp b/manifests/site.pp
index 25015a7..c691260 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -2651,16 +2651,19 @@
node 'stat1001.eqiad.wmnet' {
# stat1001 is mainly used to host Analytics websites like:
- # - http://stats.wikimedia.org (Wikistats)
- # - http://datasets.wikimedia.org
- # - http://metrics.wikimedia.org
+ # - https://stats.wikimedia.org (Wikistats)
+ # - https://datasets.wikimedia.org
+ # - https://metrics.wikimedia.org
# or https://metrics.wmflabs.org/ (Wikimetrics)
+ # - https://pivot.wikimedia.org (Imply's Pivot UI for Druid data)
#
# For a complete and up to date list please check the
# related role/module.
#
# This node is not intended for data processing.
- role(statistics::web)
+ role(statistics::web,
+ analytics_cluster::druid::pivot)
+
include standard
include base::firewall
}
diff --git a/modules/pivot/files/pivot.profile.firejail
b/modules/pivot/files/pivot.profile.firejail
new file mode 100644
index 0000000..acd4e84
--- /dev/null
+++ b/modules/pivot/files/pivot.profile.firejail
@@ -0,0 +1,26 @@
+# system directories
+blacklist /sbin
+blacklist /usr/sbin
+blacklist /usr/local/sbin
+
+# system management
+blacklist ${PATH}/umount
+blacklist ${PATH}/mount
+blacklist ${PATH}/fusermount
+blacklist ${PATH}/su
+blacklist ${PATH}/sudo
+blacklist ${PATH}/xinput
+blacklist ${PATH}/evtest
+blacklist ${PATH}/xev
+blacklist ${PATH}/strace
+blacklist ${PATH}/nc
+blacklist ${PATH}/ncat
+
+blacklist /etc/shadow
+blacklist /etc/ssh
+blacklist /root
+blacklist /home
+noroot
+caps.drop all
+seccomp
+private-dev
diff --git a/modules/pivot/manifests/init.pp b/modules/pivot/manifests/init.pp
new file mode 100644
index 0000000..1db06d9
--- /dev/null
+++ b/modules/pivot/manifests/init.pp
@@ -0,0 +1,94 @@
+# == Class: pivot
+#
+# This class installs and configures the Imply Pivot nodejs application.
+#
+# Context up to September 2016:
+# There is a current dispute between Imply and Metamarkets about a possible
+# copyright infringement related to Imply's pivot UI.
+# The Analytics team set a while back a goal to provide a Pivot UI
+# to their users with the assumption that all the code
+# used/deployed was open souce and freely available. If this assumption will
+# change in the future, for example after a legal sentence, the Analytics team
+# will take the necessary actions.
+# For any question please reach out to the Analytics team:
+# https://www.mediawiki.org/wiki/Analytics#Contact
+#
+# Bug: T138262
+#
+# === Parameters
+#
+# $port The port used by Pivot to accept HTTP connections.
+# Default: 9090
+# $druid_broker The fully qualified domain name (like
druid1001.eqiad.wmnet)
+# of the Druid Broker that the Pivot UI will contact.
+# Default: undef
+# $deployment_user Scap deployment user.
+# Default: 'analytics_deploy'
+# $scap_repo Scap repository.
+# Default: 'analytics/pivot/deploy'
+# $contact_group Contact group for alerts.
+# Default: 'admins'
+
+class pivot(
+ $port = 9090,
+ $druid_broker = undef,
+ $deployment_user = 'analytics_deploy',
+ $scap_repo = 'analytics/pivot/deploy',
+ $contact_group = 'admins',
+) {
+
+ requires_os('debian >= jessie')
+ require_package('nodejs', 'firejail')
+
+ $scap_deployment_base_dir = '/srv/deployment'
+ $pivot_deployment_dir = "${scap_deployment_base_dir}/${scap_repo}"
+
+ scap::target { 'analytics/pivot/deploy':
+ deploy_user => $deployment_user,
+ service_name => 'pivot',
+ }
+
+ group { 'pivot':
+ ensure => present,
+ system => true,
+ }
+
+ user { 'pivot':
+ gid => 'pivot',
+ shell => '/bin/bash',
+ system => true,
+ require => Group['pivot'],
+ }
+
+ file { '/etc/firejail/pivot.profile':
+ ensure => present,
+ owner => 'root',
+ group => 'root',
+ mode => '0444',
+ source => 'puppet:///modules/pivot/pivot.profile.firejail',
+ }
+
+ systemd::syslog { 'pivot':
+ readable_by => 'all',
+ base_dir => '/var/log',
+ group => 'root',
+ }
+
+ base::service_unit { 'pivot':
+ ensure => present,
+ systemd => true,
+ require => [
+ Scap['analytics/pivot/deploy'],
+ File['/etc/firejail/pivot.profile'],
+ User['pivot'],
+ Systemd::Syslog['pivot'],
+ ],
+ }
+
+ monitoring::service { 'pivot':
+ description => 'pivot',
+ check_command => "check_tcp!${port}",
+ contact_group => $contact_group,
+ require => Base::Service_unit['pivot'],
+ }
+}
diff --git a/modules/pivot/templates/initscripts/pivot.systemd.erb
b/modules/pivot/templates/initscripts/pivot.systemd.erb
new file mode 100644
index 0000000..d5bc7cf
--- /dev/null
+++ b/modules/pivot/templates/initscripts/pivot.systemd.erb
@@ -0,0 +1,19 @@
+# NOTE: This file is managed by Puppet
+# Systemd unit for the Imply Pivot UI
+[Unit]
+Description="pivot service"
+After=network.target
+
+[Service]
+User=pivot
+Group=pivot
+Environment="NODE_PATH=<%= @pivot_deployment_dir %>/node_modules"
+Restart=always
+RestartSec=2s
+# wait 60 seconds for a graceful restart before killing the master
+TimeoutStopSec=60
+WorkingDirectory=<%= @pivot_deployment_dir %>
+ExecStart=/usr/bin/firejail --profile=/etc/firejail/pivot.profile --
/usr/bin/nodejs <%= @pivot_deployment_dir %>/build/bin/pivot -p <%= @port %> -d
<%= @druid_broker %>
+
+[Install]
+WantedBy=multi-user.target
\ No newline at end of file
diff --git a/modules/role/manifests/analytics_cluster/druid/pivot.pp
b/modules/role/manifests/analytics_cluster/druid/pivot.pp
new file mode 100644
index 0000000..89450b7
--- /dev/null
+++ b/modules/role/manifests/analytics_cluster/druid/pivot.pp
@@ -0,0 +1,6 @@
+# == Class role::analytics_cluster::druid::pivot
+# Imply's Pivot nodejs UI to explore Druid data
+#
+class role::analytics_cluster::druid::pivot {
+ include ::pivot
+}
\ No newline at end of file
diff --git a/modules/statistics/manifests/sites/pivot.pp
b/modules/statistics/manifests/sites/pivot.pp
index 987bd3c..cb9db32 100644
--- a/modules/statistics/manifests/sites/pivot.pp
+++ b/modules/statistics/manifests/sites/pivot.pp
@@ -1,8 +1,11 @@
-# == Class statistics::sites::yarn
+# == Class statistics::sites::pivot
# pivot.wikimedia.org
#
-# This site will be a simple reverse proxy to the nodejs service serving
-# the pivot UI, used to limit the access to authenticated clients (via LDAP).
+# This site is composed by two parts:
+# 1) a simple Apache reverse proxy to limit the access to authenticated
+# clients (via LDAP);
+# 2) a nodejs application (Imply's pivot UI) deployed via scap (not part of
+# this class).
#
# Context up to September 2016:
# There is a current dispute between Imply and Metamarkets about a possible
--
To view, visit https://gerrit.wikimedia.org/r/312495
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I0898ddf86bc8074b772cfa26d04a90a76cac52ce
Gerrit-PatchSet: 12
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey <ltosc...@wikimedia.org>
Gerrit-Reviewer: Alexandros Kosiaris <akosia...@wikimedia.org>
Gerrit-Reviewer: Elukey <ltosc...@wikimedia.org>
Gerrit-Reviewer: Muehlenhoff <mmuhlenh...@wikimedia.org>
Gerrit-Reviewer: Ottomata <o...@wikimedia.org>
Gerrit-Reviewer: jenkins-bot <>
_______________________________________________
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
