Re: [tor-bugs] #33733 [Internal Services/Tor Sysadmin Team]: How do home directories work?

[Tor Bug Tracker & Wiki]

#33733: How do home directories work?
-+-
 Reporter:  irl  |  Owner:  irl
 Type:  task | Status:
 |  assigned
 Priority:  Medium   |  Milestone:
Component:  Internal Services/Tor Sysadmin Team  |Version:
 Severity:  Normal   | Resolution:
 Keywords:  metrics-team-roadmap-2020April   |  Actual Points:  0.1
Parent ID:  #33715   | Points:  0.5
 Reviewer:   |Sponsor:
-+-
Changes (by irl):

 * points:   => 0.5
 * actualpoints:   => 0.1


--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #33733 [Internal Services/Tor Sysadmin Team]: How do home directories work?

[Tor Bug Tracker & Wiki]

#33733: How do home directories work?
-+-
 Reporter:  irl  |  Owner:  irl
 Type:  task | Status:
 |  assigned
 Priority:  Medium   |  Milestone:
Component:  Internal Services/Tor Sysadmin Team  |Version:
 Severity:  Normal   | Resolution:
 Keywords:  metrics-team-roadmap-2020April   |  Actual Points:
Parent ID:  #33715   | Points:
 Reviewer:   |Sponsor:
-+-
Changes (by gaba):

 * keywords:   => metrics-team-roadmap-2020April


--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #33733 [Internal Services/Tor Sysadmin Team]: How do home directories work?

[Tor Bug Tracker & Wiki]

#33733: How do home directories work?
-+-
 Reporter:  irl  |  Owner:  irl
 Type:  task | Status:
 |  assigned
 Priority:  Medium   |  Milestone:
Component:  Internal Services/Tor Sysadmin Team  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:  #33715   | Points:
 Reviewer:   |Sponsor:
-+-

Comment (by irl):

 From today's journal (no action needed as yet, but maybe interesting to
 see what I'm doing):

 > 16:59 metrics-common looking in good shape
 >
 > The metrics-common role appears to prepare the system for the oninooo-
 backend role with no modification required to the onionoo-backend role, so
 that's a sucess. If the same can be achieved for the exit-scanner role
 then this ticket can likely be closed. The YAML variables would also make
 a decent description for giving to TSA when we want a new host from them.
 >
 > An example looks like this:
 >
 > https://gitweb.torproject.org/user/irl/metrics-
 cloud.git/tree/ansible/group_vars/onionoo_backend.yml
 >
 > It defines the directories we want created, and the users we want
 created. I've not managed to come up with a scenario where we want a user
 but we don't want to have it linger, so there is an assumption that all
 service users will linger.
 >
 > I was wondering how Onionperf might fit into this. We've worked on the
 assumption so far that Onionperf is something that we run on systems that
 we manage, but at some point we might want to have it run as a service on
 a TSA host. We should try to see if we are able to seperate the parts that
 require root privileges from the parts that don't, and what each of those
 parts looks like.
 >
 > The onionoo-backend role is still running so I'll find out tomorrow if
 it worked or not, but it's at the stage where it's actually importing data
 so I figure it's worked.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #33733 [Internal Services/Tor Sysadmin Team]: How do home directories work?

[Tor Bug Tracker & Wiki]

#33733: How do home directories work?
-+-
 Reporter:  irl  |  Owner:  irl
 Type:  task | Status:
 |  assigned
 Priority:  Medium   |  Milestone:
Component:  Internal Services/Tor Sysadmin Team  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:  #33715   | Points:
 Reviewer:   |Sponsor:
-+-
Changes (by anarcat):

 * owner:  tpa => irl
 * status:  needs_review => assigned


Comment:

 waiting for answers from irl

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #33733 [Internal Services/Tor Sysadmin Team]: How do home directories work?

[Tor Bug Tracker & Wiki]

#33733: How do home directories work?
-+-
 Reporter:  irl  |  Owner:  tpa
 Type:  task | Status:
 |  needs_review
 Priority:  Medium   |  Milestone:
Component:  Internal Services/Tor Sysadmin Team  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:  #33715   | Points:
 Reviewer:   |Sponsor:
-+-

Comment (by anarcat):

 >  How does the above puppet compare to the Onionoo setup in Puppet?

 This is the onionoo_backend class, for example:

 {{{
 # onionoo.torproject.org backend host
 class roles::onionoo_backend {
   package { [
 'libcommons-codec-java',
 'libcommons-compress-java',
 'libcommons-lang3-java',
 'libgoogle-gson-java',
 'liblogback-java',
 'libservlet3.1-java',
 'libslf4j-java',
 'libjetty9-java',
 'default-jdk-headless',
   ]:
 ensure => installed,
   }

   file { '/srv/onionoo.torproject.org':
 ensure => directory,
 mode   => '0755',
 owner  => 'onionoo',
 group  => 'onionoo',
   }
   file { '/srv/onionoo.torproject.org/home':
 ensure => directory,
 mode   => '0755',
 owner  => 'onionoo',
 group  => 'onionoo',
   }
   file {'/home/onionoo':
 ensure => link,
 target => '/srv/onionoo.torproject.org/home',
   }
   file { '/srv/onionoo.torproject.org/home-unpriv':
 ensure => directory,
 mode   => '0755',
 owner  => 'onionoo-unpriv',
 group  => 'onionoo-unpriv',
   }
   file {'/home/onionoo-unpriv':
 ensure => link,
 target => '/srv/onionoo.torproject.org/home-unpriv',
   }
   file { '/srv/onionoo.torproject.org/onionoo':
 ensure => directory,
 mode   => '0755',
 owner  => 'onionoo',
 group  => 'onionoo',
   }
   file { '/etc/sudoers.d/onionoo':
 mode   => '0440',
 source => 'puppet:///modules/roles/onionoo_backend/sudoers-onionoo',
   }

   dsa_systemd::linger { 'onionoo': }
   dsa_systemd::linger { 'onionoo-unpriv': }

   # varnish
   class { 'varnish':
 vcl_config  => '/etc/varnish/onionoo.vcl',
 vcl_content => template('roles/onionoo/varnish-onionoo-
 backend.vcl.erb'),
 memory  => '1g',
   }

   # ipsec tunnels between each frontend and each backend (1:1 tunnels)
   $query = 'nodes[certname] { resources { type = "Class" and title =
 "Roles::Onionoo_frontend" } }'
   $peer_names = sort(puppetdb_query($query).map |$value| {
 $value["certname"] })
   $peer_names.each |$peer_name| {
 $network_tag = [$::fqdn, $peer_name].sort().join('::')
 ipsec::network { "ipsec::${network_tag}":
   peer_networks => $base::public_addresses,
 }
   }

   # allow access to varnish from ipsec peers
   ferm::rule { 'ipsec-service-chain':
 domain  => '(ip ip6)',
 description => 'Move incoming ipsec processed packets to a dedicated
 ipsec chain',
 rule=> 'mod policy dir in pol ipsec goto ipsec',
   }
   ferm::rule::simple { 'ipsec-varnish':
 chain => 'ipsec',
 port  => 6081,
   }
 }
 }}}

 It's similar, but not exactly the same, as you can see.

 > Can we make that use the same "standardization" as the check stuff
 before we deploy the new backends (#32268) there?

 weasel has been taking care of this so far, so I can't speak for him.

 i think that would be a great idea.

 >  I think there may be a case not covered by it where we have the onionoo
 and onionoo-unpriv users, but they both share the same service directory.

 I see, yes, that's something that needs to be considered.

 >  I am confused by the difference between the /home/$user and
 /srv/$service/home directories. For the exit scanner the home directory is
 linked into /srv/$service and not into /srv/$service/home but for Onionoo:
 >
 > {{{
 > ssh onionoo-backend-01.torproject.org ls -l '/home/onionoo*'
 > lrwxrwxrwx 1 root root 32 Sep  6  2019 /home/onionoo ->
 /srv/onionoo.torproject.org/home
 > lrwxrwxrwx 1 root root 39 Sep  6  2019 /home/onionoo-unpriv ->
 /srv/onionoo.torproject.org/home-unpriv
 > }}}

 That could be an oversight on my part, when i setup the check service. Do
 you want this fixed?

 > Is the /home path only there to keep LDAP happy?

 That might be so yes.

 > Perhaps I need to add a key to users for the "real" home directory path
 as we can't guess it from some pattern, and then always link /home/$user
 to that specified path.

 I think you can assume /home/$user is valid, although it can be a symlink
 pointing somewhere else.

 >  Regarding where to look for documentation, I had no idea. I guess
 help.tpo would have been the place to look. For this though I 

Re: [tor-bugs] #33733 [Internal Services/Tor Sysadmin Team]: How do home directories work?

[Tor Bug Tracker & Wiki]

#33733: How do home directories work?
-+-
 Reporter:  irl  |  Owner:  tpa
 Type:  task | Status:
 |  needs_review
 Priority:  Medium   |  Milestone:
Component:  Internal Services/Tor Sysadmin Team  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:  #33715   | Points:
 Reviewer:   |Sponsor:
-+-

Comment (by irl):

 How does the above puppet compare to the Onionoo setup in Puppet?

 Can we make that use the same "standardization" as the check stuff before
 we deploy the new backends (#32268) there?

 I think there may be a case not covered by it where we have the onionoo
 and onionoo-unpriv users, but they both share the same service directory.

 I am confused by the difference between the /home/$user and
 /srv/$service/home directories. For the exit scanner the home directory is
 linked into /srv/$service and not into /srv/$service/home but for Onionoo:

 {{{
 ssh onionoo-backend-01.torproject.org ls -l '/home/onionoo*'
 lrwxrwxrwx 1 root root 32 Sep  6  2019 /home/onionoo ->
 /srv/onionoo.torproject.org/home
 lrwxrwxrwx 1 root root 39 Sep  6  2019 /home/onionoo-unpriv ->
 /srv/onionoo.torproject.org/home-unpriv
 }}}

 Is the /home path only there to keep LDAP happy? Perhaps I need to add a
 key to users for the "real" home directory path as we can't guess it from
 some pattern, and then always link /home/$user to that specified path.

 Regarding where to look for documentation, I had no idea. I guess help.tpo
 would have been the place to look. For this though I don't so much need a
 list of steps that you take to do something, or examples of the Puppet
 usage, but more an understanding of the rationale and intention of doing
 it in this way so that I'm not making incorrect assumptions when I
 recreate it for the dev environments.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #33733 [Internal Services/Tor Sysadmin Team]: How do home directories work?

[Tor Bug Tracker & Wiki]

#33733: How do home directories work?
-+-
 Reporter:  irl  |  Owner:  tpa
 Type:  task | Status:
 |  needs_review
 Priority:  Medium   |  Milestone:
Component:  Internal Services/Tor Sysadmin Team  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:  #33715   | Points:
 Reviewer:   |Sponsor:
-+-
Changes (by anarcat):

 * status:  new => needs_review


Comment:

 Replying to [ticket:33733 irl]:
 > There seems to be little consistency here, which isn't what I expect
 from an orchestrated process, so I'm maybe missing something.

 You're right! It's not fully orchestrated yet. Some of those things were
 created by hands before Puppet, most are managed in LDAP (by hand as well)
 and some are managed by Puppet *and* LDAP. It's a bit all over the place.

 You're dealing with old code here. Sorry for the inconvenience! :)

 > Each service has a directory in /srv/{service}.torproject.org/ and then
 sometimes there is a home directory, which is sometimes linked in some way
 to /home/{service}. When there are multiple users for a service, they can
 share the same /srv directory but then have inconsistent naming of home
 directories.

 What do you mean "multiple users"? Could you give an example here
 specifically?

 > Is there some documentation I can read to make sense of this?

 Unfortunately: not much. I created "operational" docs that deal with "how
 to create a normal or role user", for example, in:

 https://help.torproject.org/tsa/howto/create-a-new-user/

 Then some user-facing documentation and policy is available in:

 https://help.torproject.org/tsa/doc/accounts/

 But I understand you might not have found those or that, if you did, they
 were not satisfactory.

 This is not really standardized in Puppet either, as I have just begun
 refactoring that code. The fact that it's also bound to LDAP complicates
 things as well because Puppet cannot (currently?) *write* to LDAP so
 manual operations need to be done on both systems for things to happen.

 My hope is that we can eventually replace LDAP for some of the role stuff,
 and move that into puppet, to have a more uniform experience. But it's a
 far-ranging project, as it involves touching `ud-ldap`, which is a huge
 challenge.

 The closest I've got to "standardizing" this is when we setup the new
 `check` service. I ended up writing this Puppet profile:

 {{{
 # rewrite of the exit scanner
 #
 # not to be confused with the old roles::check that is now deprecated.
 class profile::check inherits apache2 {
   # disable the default site so the FQDN site alias works
   Apache2::Site['00-default'] {
 ensure => absent,
   }

   include apache2::ssl
   include apache2::proxy_http

   $users = ['tordnsel', 'check']
   $users.each |String $user| {
 file { [
   "/srv/${user}.torproject.org",
   "/srv/${user}.torproject.org/home",
   "/srv/${user}.torproject.org/htdocs",
 ]:
   ensure => 'directory',
   mode   => '0775',
   owner  => $user,
   group  => $user,
 }
 # LDAP expects the home there
 file { "/home/${user}":
   ensure => link,
   target => "/srv/${user}.torproject.org",
 }
 loginctl_user { $user:
   linger => enabled,
 }
 file { "/etc/sudoers.d/${user}":
   mode=> '0440',
   content => @("EOF"),
   # FILE MANAGED BY PUPPET, LOCAL CHANGES WILL BE LOST
   # allow ${user} user sudo access to ${user} group
   %${user}  ALL=(${user})   ALL
   | EOF
 }
   }

   $vhost_name = 'check.torproject.org'
   apache2::site { $vhost_name:
 content => template('profile/check/vhost.erb'),
   }

   ssl::service { 'check.torproject.org': notify  => Exec['service apache2
 reload'], key => true, }

   ensure_packages('tor', {ensure => installed})
   # NOTE: tor service is ran by the service admin (through systemd
   # --user), so we mask it here (in systemd --system)
   dsa_systemd::mask { [
 'tor.service',
 'tor@default.service'
   ]:
 notify => Package['tor'],
   }

   # new exit scanner dependencies, see #33362
   ensure_packages(
   # ...
   )
 }
 }}}

 That `  $users.each |String $user| {` should really be a new resource that
 would standardize the way users are created from Puppet's perspective. You
 can also see the symlink hack to take into account the LDAP home directory
 there.

 > Context: I'm putting together our Ansible roles (#33715) that should
 replicate what TPA will give us when we move things to a TPA host after

Re: [tor-bugs] #33733 [Internal Services/Tor Sysadmin Team]: How do home directories work?

[Tor Bug Tracker & Wiki]

#33733: How do home directories work?
-+-
 Reporter:  irl  |  Owner:  tpa
 Type:  task | Status:  new
 Priority:  Medium   |  Milestone:
Component:  Internal Services/Tor Sysadmin Team  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:  #33715   | Points:
 Reviewer:   |Sponsor:
-+-
Changes (by irl):

 * cc: metrics-team (added)


--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

[tor-bugs] #33733 [Internal Services/Tor Sysadmin Team]: How do home directories work?

[Tor Bug Tracker & Wiki]

#33733: How do home directories work?
-+
 Reporter:  irl  |  Owner:  tpa
 Type:  task | Status:  new
 Priority:  Medium   |  Milestone:
Component:  Internal Services/Tor Sysadmin Team  |Version:
 Severity:  Normal   |   Keywords:
Actual Points:   |  Parent ID:  #33715
   Points:   |   Reviewer:
  Sponsor:   |
-+
 There seems to be little consistency here, which isn't what I expect from
 an orchestrated process, so I'm maybe missing something.

 Each service has a directory in /srv/{service}.torproject.org/ and then
 sometimes there is a home directory, which is sometimes linked in some way
 to /home/{service}. When there are multiple users for a service, they can
 share the same /srv directory but then have inconsistent naming of home
 directories.

 Is there some documentation I can read to make sense of this?

 Context: I'm putting together our Ansible roles (#33715) that should
 replicate what TPA will give us when we move things to a TPA host after
 we're convinced it's ready for deployment and we know what the specs will
 be, but I'm having trouble generalising even from just the Onionoo and
 Exit Scanner setups.

 I'd like to be able to set some variables, like what usernames exist, what
 groups exist, and what paths will exist and should be used for stuff, and
 then let this role set that up. The service specific (e.g. Onionoo or Exit
 Scanner) roles will then run equally on our AWS dev instances and the
 production TPA instance.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs