Re: [Toolserver-l] /mnt/user-store is full

[emijrp]

Please, do not delete that files, they are important for stats.

2010/12/23 Q 

> On 12/22/2010 5:13 PM, Colin Marquardt wrote:
> > 2010/12/22 Aryeh Gregor 
> > 
> >:
> >> Yes.  It's spent almost the whole time so far in
> >> /aux0/user-store/osm_hillshading, which looks to be millions of tiny
> >> files split up over tens of thousands of directories.
>
>
> Well, there's 986 Gigs in /mnt/user-store/stats
>
> ___
> Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/toolserver-l
> Posting guidelines for this list:
> https://wiki.toolserver.org/view/Mailing_list_etiquette
>
___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] /mnt/user-store is full

[emijrp]

WOW, that is great!

BTW I have deleted some temp files of my own in /mnt/user-store (about 27
GB).

2010/12/24 Daniel Kinzler 

> FYI, we have ordered a new array with 24 TB of space for stats, user store,
> etc.
> We hope to get it installed in January. Things will get better soon.
>
> -- daniel
>
> On 23.12.2010 02:33, Q wrote:
> > On 12/22/2010 5:13 PM, Colin Marquardt wrote:
> >> 2010/12/22 Aryeh Gregor 
> >> 
> >:
> >>> Yes.  It's spent almost the whole time so far in
> >>> /aux0/user-store/osm_hillshading, which looks to be millions of tiny
> >>> files split up over tens of thousands of directories.
> >
> >
> > Well, there's 986 Gigs in /mnt/user-store/stats
> >
> > ___
> > Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
> > https://lists.wikimedia.org/mailman/listinfo/toolserver-l
> > Posting guidelines for this list:
> https://wiki.toolserver.org/view/Mailing_list_etiquette
>
>
> ___
> Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/toolserver-l
> Posting guidelines for this list:
> https://wiki.toolserver.org/view/Mailing_list_etiquette
>
___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] /mnt/user-store is full

[River Tarnell]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Aryeh Gregor:
> I'm looking at this, but I have no brighter idea than just running du
> on it, which is taking a very long time.

I wrote a tool called summdisk for this, which produces reports on per-user 
disk usage: 



Unfortunately it still takes a very long time to run.  According to df, there 
are 136,110,408 inodes used on the volume[0]; I wonder if people who currently 
create large numbers of small files could save some accounting space by 
aggregating them into larger blocks, like OSM's meta-tiles.

- river.

[0] of which ~127m or 93% are used by a single user
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (NetBSD)

iEYEARECAAYFAk0YgfUACgkQIXd7fCuc5vIv8ACguhvsRsc+lRi6hGOaEpWR/Xfg
cxYAn2sjtEkeeG+uoWrvTvH/5fQQ4U54
=WWcS
-END PGP SIGNATURE-

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] /mnt/user-store is full

[DaB.]

Hello,
At Friday 24 December 2010 13:43:34 DaB. wrote:
> FYI, we have ordered a new array with 24 TB of space for stats, user store,
> etc. We hope to get it installed in January. Things will get better soon.

but this should not stop people to look into the user-store and remove old 
data which they need anymore ;-). (like the 7th dump of enwp of the same age).

Sincerly,
DaB.

-- 
Userpage: [[:w:de:User:DaB.]] — PGP: 2B255885


signature.asc
Description: This is a digitally signed message part.
___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] /mnt/user-store is full

[Daniel Kinzler]

FYI, we have ordered a new array with 24 TB of space for stats, user store, etc.
We hope to get it installed in January. Things will get better soon.

-- daniel

On 23.12.2010 02:33, Q wrote:
> On 12/22/2010 5:13 PM, Colin Marquardt wrote:
>> 2010/12/22 Aryeh Gregor :
>>> Yes.  It's spent almost the whole time so far in
>>> /aux0/user-store/osm_hillshading, which looks to be millions of tiny
>>> files split up over tens of thousands of directories.
> 
> 
> Well, there's 986 Gigs in /mnt/user-store/stats
> 
> ___
> Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/toolserver-l
> Posting guidelines for this list: 
> https://wiki.toolserver.org/view/Mailing_list_etiquette


___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] /mnt/user-store is full

[Q]

On 12/22/2010 5:13 PM, Colin Marquardt wrote:
> 2010/12/22 Aryeh Gregor :
>> Yes.  It's spent almost the whole time so far in
>> /aux0/user-store/osm_hillshading, which looks to be millions of tiny
>> files split up over tens of thousands of directories.


Well, there's 986 Gigs in /mnt/user-store/stats

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] /mnt/user-store is full

[Colin Marquardt]

2010/12/22 Aryeh Gregor :
> Yes.  It's spent almost the whole time so far in
> /aux0/user-store/osm_hillshading, which looks to be millions of tiny
> files split up over tens of thousands of directories.

This has been like that for months, and never was a problem so far.
It's also unlikely that this directory has increased in size lately,
unless someone apart from me has added stuff there.

Cheers
  Colin

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] /mnt/user-store is full

[Aryeh Gregor]

On Wed, Dec 22, 2010 at 3:36 PM, Gerald A  wrote:
> It looks like a huge volume, so it makes sense that it would take some time
> to complete.

Yes.  It's spent almost the whole time so far in
/aux0/user-store/osm_hillshading, which looks to be millions of tiny
files split up over tens of thousands of directories.

> Are quotas enabled on the volume? That might give you a quick snapshot of
> who the biggest user is.

Not as far as I can tell.  quota -v on users who have lots of files
there doesn't return any results.

> You could try running the du in a subdirectory, which might give you some
> more ideas about big directories. (You can try: du -k | sort -rn).

I could, but it doesn't seem like it would be much faster than just
waiting for a du on the whole thing to complete.

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] /mnt/user-store is full

[Gerald A]

Hey,

On Wed, Dec 22, 2010 at 3:29 PM, Aryeh Gregor

> wrote:

>
> I'm looking at this, but I have no brighter idea than just running du
> on it, which is taking a very long time.  So if any other roots have a
> better idea to figure out what's going on and/or fix it, feel free to
> kill my du process (running as root) and delete /tmp/userstore-du on
> hemlock.  (I'm also not quite sure what I'd do if I did figure out the
> culprit, since I don't want to delete users' data without their
> permission unless it's clearly useless.)
>

It looks like a huge volume, so it makes sense that it would take some time
to complete.

Are quotas enabled on the volume? That might give you a quick snapshot of
who the biggest user is.

You could try running the du in a subdirectory, which might give you some
more ideas about big directories. (You can try: du -k | sort -rn).

I'm not a root, so I can't help in the problem analysis except by writing
here. :)

Gerald
___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] /mnt/user-store is full

[Aryeh Gregor]

On Wed, Dec 22, 2010 at 8:31 AM, Darkdadaah  wrote:
> Hi, it looks like the user-store is completely full:
>
>>df
> hemlock:/aux0/user-store
>  3904398336 3904398336 0 100% /mnt/user-store
>
>
> It requires cleaning and sorting (especially the dumps that are all over the
> place), but I especially wonder what is taking so much place ?
>
> I guess I'll have to use my home quota for now.

I'm looking at this, but I have no brighter idea than just running du
on it, which is taking a very long time.  So if any other roots have a
better idea to figure out what's going on and/or fix it, feel free to
kill my du process (running as root) and delete /tmp/userstore-du on
hemlock.  (I'm also not quite sure what I'd do if I did figure out the
culprit, since I don't want to delete users' data without their
permission unless it's clearly useless.)

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

[Toolserver-l] /mnt/user-store is full

[Darkdadaah]

Hi, it looks like the user-store is completely full:

>df
hemlock:/aux0/user-store
 3904398336 3904398336 0 100% /mnt/user-store


It requires cleaning and sorting (especially the dumps that are all
over the place), but I especially wonder what is taking so much place
?
I guess I'll have to use my home quota for now.

Darkdadaah
___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette