Re: [tor-bugs] #31051 [Internal Services/Tor Sysadmin Team]: bungei running out of space

[Tor Bug Tracker & Wiki]

#31051: bungei running out of space
-+-
 Reporter:  anarcat  |  Owner:  anarcat
 Type:  defect   | Status:  closed
 Priority:  Medium   |  Milestone:
Component:  Internal Services/Tor Sysadmin Team  |Version:
 Severity:  Normal   | Resolution:  fixed
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+-
Changes (by anarcat):

 * status:  accepted => closed
 * resolution:   => fixed


Comment:

 all good now:

 {{{
 root@bungei:~# lvextend vg_bulk/backups-pg -L +2T
   Size of logical volume vg_bulk/backups-pg changed from 1,00 TiB (262144
 extents) to 3,00 TiB (786432 extents).
   Logical volume vg_bulk/backups-pg successfully resized.
 root@bungei:~# resize2fs /dev/mapper/vg_bulk-backups--pg
 resize2fs 1.44.5 (15-Dec-2018)
 Filesystem at /dev/mapper/vg_bulk-backups--pg is mounted on
 /srv/backups/pg; on-line resizing required
 old_desc_blocks = 128, new_desc_blocks = 384
 The filesystem on /dev/mapper/vg_bulk-backups--pg is now 805306368 (4k)
 blocks long.

 root@bungei:~# df -h
 Sys. de fichiersTaille Utilisé Dispo Uti% Monté sur
 udev   63G   0   63G   0% /dev
 tmpfs  13G1,3G   12G  11% /run
 /dev/mapper/croot  20G2,6G   17G  14% /
 tmpfs  63G 72K   63G   1% /dev/shm
 tmpfs 5,0M   0  5,0M   0% /run/lock
 tmpfs  63G   0   63G   0%
 /sys/fs/cgroup
 tmpfs 4,0G   0  4,0G   0% /tmp
 /dev/md127488M140M  319M  31% /boot
 /dev/mapper/vg_bulk-backups--pg   3,0T854G  2,0T  30%
 /srv/backups/pg
 /dev/mapper/vg_bulk-backups--bacula30T 13T   17T  43%
 /srv/backups/bacula
 tmpfs  13G   0   13G   0% /run/user/0
 }}}

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #31051 [Internal Services/Tor Sysadmin Team]: bungei running out of space

[Tor Bug Tracker & Wiki]

#31051: bungei running out of space
-+-
 Reporter:  anarcat  |  Owner:  anarcat
 Type:  defect   | Status:
 |  accepted
 Priority:  Medium   |  Milestone:
Component:  Internal Services/Tor Sysadmin Team  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+-

Comment (by anarcat):

 i actually grew the *wrong* filesystem (`/srv/backups/bacula` instead of
 `.../pg`), so i'll have to go at it again.

 {{{
 root@bungei:~# df -h
 Sys. de fichiersTaille Utilisé Dispo Uti% Monté sur
 udev   63G   0   63G   0% /dev
 tmpfs  13G1,3G   12G  11% /run
 /dev/mapper/croot  20G2,9G   16G  16% /
 tmpfs  63G 72K   63G   1% /dev/shm
 tmpfs 5,0M   0  5,0M   0% /run/lock
 tmpfs  63G   0   63G   0%
 /sys/fs/cgroup
 tmpfs 4,0G   0  4,0G   0% /tmp
 /dev/md127488M140M  319M  31% /boot
 /dev/mapper/vg_bulk-backups--pg  1007G854G  103G  90%
 /srv/backups/pg
 /dev/mapper/vg_bulk-backups--bacula30T 13T   17T  43%
 /srv/backups/bacula
 tmpfs  13G   0   13G   0% /run/user/0
 tmpfs  13G   0   13G   0%
 /run/user/1529
 }}}

 i don't think it's a problem that bacula is bigger, that said... shrinking
 it seems unnecessary. i'll just give a few more TBs to pg already (say
 2TB).

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #31051 [Internal Services/Tor Sysadmin Team]: bungei running out of space

[Tor Bug Tracker & Wiki]

#31051: bungei running out of space
-+-
 Reporter:  anarcat  |  Owner:  anarcat
 Type:  defect   | Status:
 |  accepted
 Priority:  Medium   |  Milestone:
Component:  Internal Services/Tor Sysadmin Team  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+-
Changes (by anarcat):

 * status:  reopened => accepted
 * owner:  tpa => anarcat


--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #31051 [Internal Services/Tor Sysadmin Team]: bungei running out of space

[Tor Bug Tracker & Wiki]

#31051: bungei running out of space
-+-
 Reporter:  anarcat  |  Owner:  tpa
 Type:  defect   | Status:
 |  reopened
 Priority:  Medium   |  Milestone:
Component:  Internal Services/Tor Sysadmin Team  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+-
Changes (by anarcat):

 * status:  closed => reopened
 * resolution:  fixed =>


Comment:

 this is a problem again today.

 {{{
 root@bungei:~# df -h
 Sys. de fichiersTaille Utilisé Dispo Uti% Monté sur
 udev   63G   0   63G   0% /dev
 tmpfs  13G1,3G   12G  11% /run
 /dev/mapper/croot  20G2,9G   16G  16% /
 tmpfs  63G 72K   63G   1% /dev/shm
 tmpfs 5,0M   0  5,0M   0% /run/lock
 tmpfs  63G   0   63G   0%
 /sys/fs/cgroup
 tmpfs 4,0G   0  4,0G   0% /tmp
 /dev/md127488M140M  319M  31% /boot
 /dev/mapper/vg_bulk-backups--pg  1007G854G  103G  90%
 /srv/backups/pg
 /dev/mapper/vg_bulk-backups--bacula26T 13T   13T  50%
 /srv/backups/bacula
 tmpfs  13G   0   13G   0% /run/user/0
 }}}

 performing an online resize to add 10TB now.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #31051 [Internal Services/Tor Sysadmin Team]: bungei running out of space

[Tor Bug Tracker & Wiki]

#31051: bungei running out of space
-+
 Reporter:  anarcat  |  Owner:  tpa
 Type:  defect   | Status:  closed
 Priority:  Medium   |  Milestone:
Component:  Internal Services/Tor Sysadmin Team  |Version:
 Severity:  Normal   | Resolution:  fixed
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+
Changes (by anarcat):

 * status:  new => closed
 * resolution:   => fixed


Comment:

 i think this is done. the files can be removed directly from the
 filesystem, and stuff in the database gets pruned semi-automatically.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #31051 [Internal Services/Tor Sysadmin Team]: bungei running out of space

[Tor Bug Tracker & Wiki]

#31051: bungei running out of space
-+-
 Reporter:  anarcat  |  Owner:  tpa
 Type:  defect   | Status:  new
 Priority:  Medium   |  Milestone:
Component:  Internal Services/Tor Sysadmin Team  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+-

Comment (by anarcat):

 i don't quite understand how to delete old entries in the database and/or
 the storage. not "behind bacula's back" as that sounds ominous... but i
 think the rest of arlgirdense can be removed now that archive-01 is up and
 running. i just don't know how.

 see also #30880 for that documentation.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #31051 [Internal Services/Tor Sysadmin Team]: bungei running out of space

[Tor Bug Tracker & Wiki]

#31051: bungei running out of space
-+-
 Reporter:  anarcat  |  Owner:  tpa
 Type:  defect   | Status:  new
 Priority:  Medium   |  Milestone:
Component:  Internal Services/Tor Sysadmin Team  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+-

Comment (by weasel):

 manually, and behind bacula's back, removed one of arlgirdense's old
 fulls.  We can reconsider growing things once we have migrated properly to
 the new archive host.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

[tor-bugs] #31051 [Internal Services/Tor Sysadmin Team]: bungei running out of space

[Tor Bug Tracker & Wiki]

#31051: bungei running out of space
-+-
 Reporter:  anarcat  |  Owner:  tpa
 Type:  defect   | Status:  new
 Priority:  Medium   |  Milestone:
Component:  Internal Services/Tor Sysadmin Team  |Version:
 Severity:  Normal   |   Keywords:
Actual Points:   |  Parent ID:
   Points:   |   Reviewer:
  Sponsor:   |
-+-
 we got this warning from nagios this weekend:

 {{{
 DISK WARNING - free space: /srv/backups/bacula 1886723 MB (9% inode=99%)
 }}}

 indeed, the main storage is full:

 {{{
 Sys. de fichiersTaille Utilisé Dispo Uti% Monté sur
 /dev/mapper/vg_bulk-backups--bacula20T 18T  1,6T  93%
 /srv/backups/bacula
 }}}

 There's plenty of free space on the VG, however:

 {{{
   VG  #PV #LV #SN Attr   VSize  VFree
   vg_bulk   1   2   0 wz--n- 72,60t 51,60t
 }}}

 So this could quickly be resolved by just resizing that partition. But
 it's interesting to note that the vast majority of storage in use comes
 from the static content. I suspect there's a lot of duplication around
 those and we could save space in backups by being a little smarter about
 this:

 {{{
 ncdu 1.13 ~ Use the arrow keys to navigate, press ? for help
 --- /srv/backups/bacula
 
---
 4,3 TiB [ 25,0% ##] /arlgirdense.torproject.org
 4,3 TiB [ 24,5% # ] /staticiforme.torproject.org
 2,1 TiB [ 11,9%   ] /archive-01.torproject.org
 1,1 TiB [  6,2% ##] /perdulce.torproject.org
   871,0 GiB [  4,9% # ] /colchicifolium.torproject.org
   806,1 GiB [  4,5% # ] /Catalog
   554,4 GiB [  3,1% # ] /corsicum.torproject.org
   375,1 GiB [  2,1%   ] /gillii.torproject.org
   237,5 GiB [  1,3%   ] /eugeni.torproject.org
   209,0 GiB [  1,2%   ] /hetzner-nbg1-01.torproject.org
 }}}

 Above is the top-ten in disk usage. It's interesting to note that
 archive01 is only half of its partners. We should also note that
 arlgirdense is likely going to go away eventually, because archive-01 will
 replace it.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs