There is a big difference with working with small files (around 16kb)
and big files (2mb). Performance is much better with big files. Witch is
too bad for me ;(
On 8/11/12 2:15 AM, Gandalf Corvotempesta wrote:
What do you mean with "small files"? 16k ? 160k? 16mb?
Do you know any workaround or any other software for this?
Mee too i'm trying to create a clustered storage for many
small file
2012/8/10 Philip Poten <philip.po...@gmail.com
<mailto:philip.po...@gmail.com>>
Hi Ivan,
that's because Gluster has really bad "many small files" performance
due to it's architecture.
On all stat() calls (which rsync is doing plenty of), all replicas are
being checked for integrity.
regards,
Philip
2012/8/10 Ivan Dimitrov <dob...@amln.net <mailto:dob...@amln.net>>:
> So I stopped a node to check the BIOS and after it went up, the
rebalance
> kicked in. I was looking for those kind of speeds on a normal
write. The
> rebalance is much faster than my rsync/cp.
>
>
https://dl.dropbox.com/u/282332/Screen%20Shot%202012-08-10%20at%202.04.09%20PM.png
>
> Best Regards
> Ivan Dimitrov
>
>
> On 8/10/12 1:23 PM, Ivan Dimitrov wrote:
>>
>> Hello
>> What am I doing wrong?!?
>>
>> I have a test setup with 4 identical servers with 2 disks each in
>> distribute-replicate 2. All servers are connected to a GB switch.
>>
>> I am experiencing really slow speeds at anything I do. Slow
write, slow
>> read, not to mention random write/reads.
>>
>> Here is an example:
>> random-files is a directory with 32768 files with average size
16kb.
>> [root@gltclient]:~# rsync -a /root/speedtest/random-files/
>> /home/gltvolume/
>> ^^ This will take more than 3 hours.
>>
>> On any of the servers if I do "iostat" the disks are not loaded
at all:
>>
>>
https://dl.dropbox.com/u/282332/Screen%20Shot%202012-08-10%20at%201.08.54%20PM.png
>>
>> This is similar result for all servers.
>>
>> Here is an example of simple "ls" command on the content.
>> [root@gltclient]:~# unalias ls
>> [root@gltclient]:~# /usr/bin/time -f "%e seconds" ls
/home/gltvolume/ | wc
>> -l
>> 2.81 seconds
>> 5393
>>
>> almost 3 seconds to display 5000 files?!?! When they are
32,000, the ls
>> will take around 35-45 seconds.
>>
>> This directory is on local disk:
>> [root@gltclient]:~# /usr/bin/time -f "%e seconds" ls
>> /root/speedtest/random-files/ | wc -l
>> 1.45 seconds
>> 32768
>>
>> [root@gltclient]:~# /usr/bin/time -f "%e seconds" cat
/home/gltvolume/*
>> >/dev/null
>> 190.50 seconds
>>
>> [root@gltclient]:~# /usr/bin/time -f "%e seconds" du -sh
/home/gltvolume/
>> 126M /home/gltvolume/
>> 75.23 seconds
>>
>>
>> Here is the volume information.
>>
>> [root@glt1]:~# gluster volume info
>>
>> Volume Name: gltvolume
>> Type: Distributed-Replicate
>> Volume ID: 16edd852-8d23-41da-924d-710b753bb374
>> Status: Started
>> Number of Bricks: 4 x 2 = 8
>> Transport-type: tcp
>> Bricks:
>> Brick1: 1.1.74.246:/home/sda3
>> Brick2: glt2.network.net:/home/sda3
>> Brick3: 1.1.74.246:/home/sdb1
>> Brick4: glt2.network.net:/home/sdb1
>> Brick5: glt3.network.net:/home/sda3
>> Brick6: gltclient.network.net:/home/sda3
>> Brick7: glt3.network.net:/home/sdb1
>> Brick8: gltclient.network.net:/home/sdb1
>> Options Reconfigured:
>> performance.io-thread-count: 32
>> performance.cache-size: 256MB
>> cluster.self-heal-daemon: on
>>
>>
>> [root@glt1]:~# gluster volume status all detail
>> Status of volume: gltvolume
>>
>>
------------------------------------------------------------------------------
>> Brick : Brick 1.1.74.246:/home/sda3
>> Port : 24009
>> Online : Y
>> Pid : 1479
>> File System : ext4
>> Device : /dev/sda3
>> Mount Options : rw,noatime
>> Inode Size : 256
>> Disk Space Free : 179.3GB
>> Total Disk Space : 179.7GB
>> Inode Count : 11968512
>> Free Inodes : 11901550
>>
>>
------------------------------------------------------------------------------
>> Brick : Brick glt2.network.net:/home/sda3
>> Port : 24009
>> Online : Y
>> Pid : 1589
>> File System : ext4
>> Device : /dev/sda3
>> Mount Options : rw,noatime
>> Inode Size : 256
>> Disk Space Free : 179.3GB
>> Total Disk Space : 179.7GB
>> Inode Count : 11968512
>> Free Inodes : 11901550
>>
>>
------------------------------------------------------------------------------
>> Brick : Brick 1.1.74.246:/home/sdb1
>> Port : 24010
>> Online : Y
>> Pid : 1485
>> File System : ext4
>> Device : /dev/sdb1
>> Mount Options : rw,noatime
>> Inode Size : 256
>> Disk Space Free : 228.8GB
>> Total Disk Space : 229.2GB
>> Inode Count : 15269888
>> Free Inodes : 15202933
>>
>>
------------------------------------------------------------------------------
>> Brick : Brick glt2.network.net:/home/sdb1
>> Port : 24010
>> Online : Y
>> Pid : 1595
>> File System : ext4
>> Device : /dev/sdb1
>> Mount Options : rw,noatime
>> Inode Size : 256
>> Disk Space Free : 228.8GB
>> Total Disk Space : 229.2GB
>> Inode Count : 15269888
>> Free Inodes : 15202933
>>
>>
------------------------------------------------------------------------------
>> Brick : Brick glt3.network.net:/home/sda3
>> Port : 24009
>> Online : Y
>> Pid : 28963
>> File System : ext4
>> Device : /dev/sda3
>> Mount Options : rw,noatime
>> Inode Size : 256
>> Disk Space Free : 179.3GB
>> Total Disk Space : 179.7GB
>> Inode Count : 11968512
>> Free Inodes : 11906058
>>
>>
------------------------------------------------------------------------------
>> Brick : Brick gltclient.network.net:/home/sda3
>> Port : 24009
>> Online : Y
>> Pid : 3145
>> File System : ext4
>> Device : /dev/sda3
>> Mount Options : rw,noatime
>> Inode Size : 256
>> Disk Space Free : 179.3GB
>> Total Disk Space : 179.7GB
>> Inode Count : 11968512
>> Free Inodes : 11906058
>>
>>
------------------------------------------------------------------------------
>> Brick : Brick glt3.network.net:/home/sdb1
>> Port : 24010
>> Online : Y
>> Pid : 28969
>> File System : ext4
>> Device : /dev/sdb1
>> Mount Options : rw,noatime
>> Inode Size : 256
>> Disk Space Free : 228.8GB
>> Total Disk Space : 229.2GB
>> Inode Count : 15269888
>> Free Inodes : 15207375
>>
>>
------------------------------------------------------------------------------
>> Brick : Brick gltclient.network.net:/home/sdb1
>> Port : 24010
>> Online : Y
>> Pid : 3151
>> File System : ext4
>> Device : /dev/sdb1
>> Mount Options : rw,noatime
>> Inode Size : 256
>> Disk Space Free : 228.8GB
>> Total Disk Space : 229.2GB
>> Inode Count : 15269888
>> Free Inodes : 15207375
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users@gluster.org <mailto:Gluster-users@gluster.org>
>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users@gluster.org <mailto:Gluster-users@gluster.org>
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
Gluster-users@gluster.org <mailto:Gluster-users@gluster.org>
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
