Re: [Toolserver-l] [Toolserver-announce] Maintenance

[River Tarnell]

Note: crontabs from nightshade have been saved as 
$HOME/crontab.before_nightshade_reinstall.  You should review this for 
any necessary changes, then install it:

  $ crontab $HOME/crontab.before_nightshade_reinstall

Until you do this, your crontab will not run.

You may want to review 
 before installing 
the new crontab.

- river/

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

[Toolserver-l] Maintenance

[River Tarnell]

Hi,

The previous announced maintenance has been completed successfully.  In 
case of problems, open an issue in the TS project in JIRA.

For problems relating to the conversion of nightshade to Solaris (e.g. 
missing software), open an issue in the TS project and include 
"[solaris]" at the start of the subject line.  Due to the disruptive 
nature of this maintenance, such requests will be given priority.  
(However, I'll be away for at least the next 12 hours, so I won't be 
able to resolve them immediately.)

- river.

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Compressing stats files better (was; Re: /mnt/user-store is full)

[River Tarnell]

emijrp:
> We can compress to xz while the new disks arrive. I read that it is about 24
> TB, so, we can revert to gzip in the future.

PS: Yes, we are installing 24TB of disks (12x 2TB disks), but this is 
raw space.  To begin with, a 2TB disk is only 1,862 GB of real space.  
Assuming we configure the disks as RAID 50 with two 6-disk legs, that 
gives (1862*(6-1))*2 = 18,620 GB (18.2TB) usable space.  We will reserve 
some of this for internal use (such as backups), so the total amount 
available to users will be less than that.

When budgeting for this upgrade, we assumed 5TB would be used for 
user-store.  In reality, there should be a lot more than this (we 
originally planned to use 1TB disks); but it won't be a full 24TB.

- river.

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Compressing stats files better (was; Re: /mnt/user-store is full)

[River Tarnell]

emijrp:
> We can compress to xz while the new disks arrive. I read that it is about 24
> TB, so, we can revert to gzip in the future.

I suggested xz pretty much as an emergency fix; we don't want to delete 
the files, but they do take up a lot of space.

I don't mind switching back to gzip when we have more space; in fact, I 
already suggested this to schutz in private.  However, if xz works okay 
(and it should, since the interface is identical to gzip) we may as well 
stay with it.  

If it turns out to be too slow, we could consider 7z or something else 
(rzip, bzip2, ..., or maybe even just gzip -9).

- river.

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Compressing stats files better (was; Re: /mnt/user-store is full)

[emijrp]

Hi Frederic, thanks for your work. Have you tested 7z?

We can compress to xz while the new disks arrive. I read that it is about 24
TB, so, we can revert to gzip in the future.

2011/1/2 Frédéric Schütz 

> On 12/28/2010 03:52 PM, emijrp wrote:
>
>  >> Well, there's 986 Gigs in /mnt/user-store/stats
>  >
> > Please, do not delete that files, they are important for stats.
>
> Following a suggestion made by River, I am currently recompressing all
> files using the newly-installed "xz" program. Based on an arbitrary
> sample (all files from 1 January 2011), we can expect a size reduction
> of about 25% -- or about 250 Gb freed in total. The compression time,
> however, is quite long, and I haven't measured how the decompression
> time compares with gzip.
>
> Let me know if the xz format causes any problem; I have started
> recompressing the older files, which are not used so often, but if this
> is ok with all users, I may start in the future recompressing the new
> files as soon as they are downloaded.
>
> Hopefully, with the new storage space, we'll be able to store these
> files in a more convenient way (not only raw files as they are now, but
> also the same information in a better format).
>
> Frédéric
>
> ___
> Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/toolserver-l
> Posting guidelines for this list:
> https://wiki.toolserver.org/view/Mailing_list_etiquette
>
___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

[Toolserver-l] Compressing stats files better (was; Re: /mnt/user-store is full)

[Frédéric Schütz]

On 12/28/2010 03:52 PM, emijrp wrote:

 >> Well, there's 986 Gigs in /mnt/user-store/stats
 >
> Please, do not delete that files, they are important for stats.

Following a suggestion made by River, I am currently recompressing all 
files using the newly-installed "xz" program. Based on an arbitrary 
sample (all files from 1 January 2011), we can expect a size reduction 
of about 25% -- or about 250 Gb freed in total. The compression time, 
however, is quite long, and I haven't measured how the decompression 
time compares with gzip.

Let me know if the xz format causes any problem; I have started 
recompressing the older files, which are not used so often, but if this 
is ok with all users, I may start in the future recompressing the new 
files as soon as they are downloaded.

Hopefully, with the new storage space, we'll be able to store these 
files in a more convenient way (not only raw files as they are now, but 
also the same information in a better format).

Frédéric

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette