[CentOS] lots of small files in a folder on Linux centos

2011-07-24 Thread yonatan pingle
Hello, I have a rather annoying issue on going with one of my centos virtual servers. the server hosts a website using apache and mysql ,there are three persons involved with keeping the site up and running. and i am his root due to the fact he does not know anything with about Linux. there is an p

[CentOS] lots of small files in a folder on Linux centos

2011-07-24 Thread R P Herrold
On Sun, 24 Jul 2011, yonatan pingle wrote: > the coder is not tech savvy as one might expect, so it's > really hard for me to explain the issue of having lots of > files in one folder to the site owner or to the coder. I do not expect coders to remain 'not tech savvy' If the coder is not willi

[CentOS] lots of small files in a folder on Linux centos

2011-07-24 Thread R P Herrold
On Sun, 24 Jul 2011, Always Learning wrote: > If the pictures are named sequentially, why not store then at a 100 per > directory structure something like this > > /pix/0/00/pix1.jpg > > /pix/0/26/pix02614.jpg > > /pix/6/72/pix67255.jpg Go read Knuth One does not do that because then one is

[CentOS] lots of small files in a folder on Linux centos

2011-07-24 Thread R P Herrold
On Sun, 24 Jul 2011, Keith Roberts wrote: >> By using a hash, we remove those constraints, and also gain >> the virtuous effect for free of self-organizing a relatively >> level dispersion of files to the destination directories > > Not followed the whole thread, but a SQL database index of > the

[CentOS] lots of small files in a folder on Linux centos

2011-07-25 Thread R P Herrold
On Mon, 25 Jul 2011, Marc Deop wrote: > It's more than twice as fast than the previous sh script. In part this is /bin/sh v /bin/bash and using 'bashisms' matter, but yes, I did not seek to optimize a teaching throwaway > 1- m5sum the file we need ... actually the NAME of the file, to make i

Re: [CentOS] lots of small files in a folder on Linux centos

2011-07-24 Thread Eero Volotinen
2011/7/24 yonatan pingle : > Hello, > I have a rather annoying issue on going with one of my centos virtual servers. > the server hosts a website using apache and mysql ,there are three > persons involved with keeping the site up and running. > and i am his root due to the fact he does not know any

Re: [CentOS] lots of small files in a folder on Linux centos

2011-07-24 Thread Alexander Dalloz
Am 24.07.2011 13:03, schrieb Eero Volotinen: > 2011/7/24 yonatan pingle : >> uploads]# ls | wc -l >> 3123 > I assume that you are using ext3 or ext4 filesystems? Both ext3 and > ext4 slows down, if there is too much files in same directory. > XFS-fs is solution to fix this problem. > Eero Serio

Re: [CentOS] lots of small files in a folder on Linux centos

2011-07-24 Thread yonatan pingle
On Sun, Jul 24, 2011 at 2:19 PM, Alexander Dalloz wrote: > Am 24.07.2011 13:03, schrieb Eero Volotinen: >> 2011/7/24 yonatan pingle : > >>> uploads]# ls | wc -l >>> 3123 > >> I assume that you are using ext3 or ext4 filesystems? Both ext3 and >> ext4 slows down, if there is too much files in same

Re: [CentOS] lots of small files in a folder on Linux centos

2011-07-24 Thread Ljubomir Ljubojevic
yonatan pingle wrote: > On Sun, Jul 24, 2011 at 2:19 PM, Alexander Dalloz wrote: >> Am 24.07.2011 13:03, schrieb Eero Volotinen: >>> 2011/7/24 yonatan pingle : uploads]# ls | wc -l 3123 >>> I assume that you are using ext3 or ext4 filesystems? Both ext3 and >>> ext4 slows down, if there

Re: [CentOS] lots of small files in a folder on Linux centos

2011-07-24 Thread Ryan Wagoner
On Sun, Jul 24, 2011 at 7:52 AM, yonatan pingle wrote: > Hi, Alexander > good suggestions, ill monitor I/O and mysql code, sounds like a code > related issue and not a centos issue after all. > > it runs on ext3  ,i could only guess how to code deals with the dir, > as it seems to be the site buil

Re: [CentOS] lots of small files in a folder on Linux centos

2011-07-24 Thread yonatan pingle
>> >> > > Do you have cahcing turned on in CMS? That could help. > > -- > > Ljubomir Ljubojevic > (Love is in the Air) > PL Computers > Serbia, Europe > > Google is the Mother, Google is the Father, and traceroute is your > trusty Spiderman... > StarOS, Mikrotik and CentOS/RHEL/Linux consultant > _

Re: [CentOS] lots of small files in a folder on Linux centos

2011-07-24 Thread yonatan pingle
>> RHCT | RHCSA | CCNA1 > > If you are using phpMyAdmin the status page will aid you in tuning > mySQL. Look for values in red. The description will usually tell you > what to adjust to improve performance. > > Ryan > ___ > CentOS mailing list > CentOS@ce

Re: [CentOS] lots of small files in a folder on Linux centos

2011-07-24 Thread Ryan Wagoner
On Sun, Jul 24, 2011 at 8:40 AM, yonatan pingle wrote: > im good with mysqltuner.pl, > as it seems there are slow queries on mysql and i have adjusted all > values in my.cnf according to the application needs. > > looks like it's all in the code and the way the CMS handles the files > from that up

Re: [CentOS] lots of small files in a folder on Linux centos

2011-07-24 Thread yonatan pingle
On Sun, Jul 24, 2011 at 3:43 PM, Ryan Wagoner wrote: > On Sun, Jul 24, 2011 at 8:40 AM, yonatan pingle > wrote: >> im good with mysqltuner.pl, >> as it seems there are slow queries on mysql and i have adjusted all >> values in my.cnf according to the application needs. >> >> looks like it's all i

Re: [CentOS] lots of small files in a folder on Linux centos

2011-07-24 Thread John R. Dennison
On Sun, Jul 24, 2011 at 03:53:46PM +0300, yonatan pingle wrote: > > Yes Ryan, that exactly what i have done. > he will get the log shortly and i will get some not free beer. While I'm all for mysql optimization it's clearly evident from an earlier posting that your disks are thrashing with insane

Re: [CentOS] lots of small files in a folder on Linux centos

2011-07-24 Thread Diego Sanchez
2011/7/24 yonatan pingle : > > there is no caching system, its a " home made" CMS. > > You can use an accelerator too. http://en.wikipedia.org/wiki/PHP_accelerator http://en.wikipedia.org/wiki/List_of_PHP_accelerators Please, make a big backup before this! (I nevever had a problem, but... why te

Re: [CentOS] lots of small files in a folder on Linux centos

2011-07-24 Thread yonatan pingle
On Sun, Jul 24, 2011 at 4:02 PM, John R. Dennison wrote: > On Sun, Jul 24, 2011 at 03:53:46PM +0300, yonatan pingle wrote: >> >> Yes Ryan, that exactly what i have done. >> he will get the log shortly and i will get some not free beer. > > While I'm all for mysql optimization it's clearly evident

Re: [CentOS] lots of small files in a folder on Linux centos

2011-07-24 Thread yonatan pingle
On Sun, Jul 24, 2011 at 5:13 PM, R P Herrold wrote: > On Sun, 24 Jul 2011, yonatan pingle wrote: > >> the coder is not tech savvy as one might expect, so it's >> really hard for me to explain the issue of having lots of >> files in one folder to the site owner or to the coder. > > I do not expect

Re: [CentOS] lots of small files in a folder on Linux centos

2011-07-24 Thread Always Learning
> On Sun, Jul 24, 2011 at 5:13 PM, R P Herrold wrote: > > then, we look to the leading letter of the hask, to design our > > egg carton bins. We place pix1.jpg in directory: ./f/ and > > pix2.jpg in directory ./1/ and pix3.jpg in directory > > ./b/ and so forth -- if the directories

Re: [CentOS] lots of small files in a folder on Linux centos

2011-07-24 Thread Marian Marinov
On Sunday 24 July 2011 22:48:23 Always Learning wrote: > > On Sun, Jul 24, 2011 at 5:13 PM, R P Herrold wrote: > > > then, we look to the leading letter of the hask, to design our > > > egg carton bins. We place pix1.jpg in directory: ./f/ and > > > pix2.jpg in directory ./1/ and pix3

Re: [CentOS] lots of small files in a folder on Linux centos

2011-07-24 Thread Keith Roberts
On Sun, 24 Jul 2011, R P Herrold wrote: > By using a hash, we remove those constraints, and also gain > the virtuous effect for free of self-organizing a relatively > level dispersion of files to the destination directories Not followed the whole thread, but a SQL database index of the actual pi

Re: [CentOS] lots of small files in a folder on Linux centos

2011-07-24 Thread Always Learning
On Sun, 2011-07-24 at 16:33 -0400, R P Herrold wrote: > On Sun, 24 Jul 2011, Always Learning wrote: > > > If the pictures are named sequentially, why not store then at a 100 per > > directory structure something like this > > > > /pix/0/00/pix1.jpg > > > > /pix/0/26/pix02614.jpg > > > > /pix

Re: [CentOS] lots of small files in a folder on Linux centos

2011-07-24 Thread Always Learning
On Sun, 2011-07-24 at 17:50 -0400, R P Herrold wrote: > On Sun, 24 Jul 2011, Keith Roberts wrote: > > >> By using a hash, we remove those constraints, and also gain > >> the virtuous effect for free of self-organizing a relatively > >> level dispersion of files to the destination directories > >

Re: [CentOS] lots of small files in a folder on Linux centos

2011-07-24 Thread Rajagopal Swaminathan
Greetings, On Sun, Jul 24, 2011 at 2:59 PM, yonatan pingle wrote: > Hello, > after looking into the website folders, i have found one folder which > from my point of view is one of the causes for the server loads. > hmm... does mount -noatime -noadirtime help speed it up? -- Regards, Rajag

Re: [CentOS] lots of small files in a folder on Linux centos

2011-07-24 Thread John R. Dennison
On Mon, Jul 25, 2011 at 06:38:33AM +0530, Rajagopal Swaminathan wrote: > > hmm... does mount -noatime -noadirtime help speed it up? Just an FYI: noatime is a superset that includes noadirtime. John -- You can safely assume you've cre

Re: [CentOS] lots of small files in a folder on Linux centos

2011-07-24 Thread Les Mikesell
On 7/24/11 4:08 PM, Keith Roberts wrote: > On Sun, 24 Jul 2011, R P Herrold wrote: > >> By using a hash, we remove those constraints, and also gain >> the virtuous effect for free of self-organizing a relatively >> level dispersion of files to the destination directories > > Not followed the whole

Re: [CentOS] lots of small files in a folder on Linux centos

2011-07-25 Thread Marc Deop
On Sunday 24 July 2011 10:13:30 R P Herrold wrote: > #!/bin/sh > # > CANDIDATES="pix1.jpg pix2.jpg pix3.jpg" > for i in `echo "${CANDIDATES}"`; do > HASH=`echo "$i" | md5sum - | awk {'print $1'}` > echo "$i${HASH}" > done I know it absolutelly has nothing to d

Re: [CentOS] lots of small files in a folder on Linux centos

2011-07-25 Thread Lamar Owen
On Sunday, July 24, 2011 05:29:23 AM yonatan pingle wrote: ... > lately the server is under-preforming and load averages are high, > mysql service keeps crashing and the server is hitting max memory > usage ( so i added ram .. ) , > after looking into the website folders, i have found one folder wh