[Gluster-users] ls gives Input/Output error

[James Whitwell]

Hi,

I've been getting an "Input/Output error" from 'ls' on a new install of
glusterfs 3.4.2 on 32-bit Ubuntu 14.04 on our production servers.  On our
test servers (provisioned with Vagrant using the same version of Ubuntu,
but 64-bit) I don't get the error.  I can't seem to see anything in the
logs that points to the error, but hopefully someone can point me in the
right direction.

jams@node01:~$ sudo gluster
gluster> volume create s5 node01:/gfs/data2/s5 force
volume create: s5: success: please start the volume to access data
gluster> volume start s5
volume start: s5: success
gluster> exit
jams@node01:~$ sudo mkdir /gfs/mnt/s5
jams@node01:~$ sudo mount -t glusterfs node01:/s5 /gfs/mnt/s5
jams@node01:~$ mount | grep s5
node01:/s5 on /gfs/mnt/s5 type fuse.glusterfs
(rw,default_permissions,allow_other,max_read=131072)
jams@node01:~$ cd /gfs/mnt/s5
jams@node01:/gfs/mnt/s5$ ls
jams@node01:/gfs/mnt/s5$ sudo touch a b c d e f g h i j k l m n o p q r s t
u v w x y z
jams@node01:/gfs/mnt/s5$ ls -l
ls: reading directory .: Input/output error
total 0
jams@node01:/gfs/mnt/s5$

Running an strace on 'ls -l' shows the error coming from this:

openat(AT_FDCWD, ".",
O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY|O_CLOEXEC) = 3
getdents64(3, 0x9b04ee4, 32768) = -1 EIO (Input/output error)

If I remove a couple of files it starts working again:

jams@node01:/gfs/mnt/s5$ sudo rm z
jams@node01:/gfs/mnt/s5$ ls
ls: reading directory .: Input/output error
jams@node01:/gfs/mnt/s5$ sudo rm y
jams@node01:/gfs/mnt/s5$ ls
ls: reading directory .: Input/output error
jams@node01:/gfs/mnt/s5$ sudo rm x
jams@node01:/gfs/mnt/s5$ ls
a  b  c  d  e  f  g  h  i  j  k  l  m  n  o  p  q  r  s  t  u  v  w
jams@node01:/gfs/mnt/s5$

My /var/log/glusterfs/gfs-mnt-s5.log looks like this:

[2015-04-30 05:14:58.414575] I [glusterfsd.c:1910:main]
0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.2
(/usr/sbin/glusterfs --volfile-id=/s5 --volfile-server=node01 /gfs/mnt/s5)
[2015-04-30 05:14:58.417227] I [socket.c:3480:socket_init] 0-glusterfs: SSL
support is NOT enabled
[2015-04-30 05:14:58.417287] I [socket.c:3495:socket_init] 0-glusterfs:
using system polling thread
[2015-04-30 05:14:58.421834] I [socket.c:3480:socket_init] 0-s5-client-0:
SSL support is NOT enabled
[2015-04-30 05:14:58.421873] I [socket.c:3495:socket_init] 0-s5-client-0:
using system polling thread
[2015-04-30 05:14:58.421903] I [client.c:2154:notify] 0-s5-client-0: parent
translators are ready, attempting connect on transport
Given volfile:
+--+
  1: volume s5-client-0
  2: type protocol/client
  3: option password 76ac8223-0c60-40e5-958e-22a186477e95
  4: option username fa437814-33f8-4711-a672-8da1c914a39d
  5: option transport-type tcp
  6: option remote-subvolume /gfs/data2/s5
  7: option remote-host node01
  8: end-volume
  9:
 10: volume s5-dht
 11: type cluster/distribute
 12: subvolumes s5-client-0
 13: end-volume
 14:
 15: volume s5-write-behind
 16: type performance/write-behind
 17: subvolumes s5-dht
 18: end-volume
 19:
 20: volume s5-read-ahead
 21: type performance/read-ahead
 22: subvolumes s5-write-behind
 23: end-volume
 24:
 25: volume s5-io-cache
 26: type performance/io-cache
 27: subvolumes s5-read-ahead
 28: end-volume
 29:
 30: volume s5-quick-read
 31: type performance/quick-read
 32: subvolumes s5-io-cache
 33: end-volume
 34:
 35: volume s5-open-behind
 36: type performance/open-behind
 37: subvolumes s5-quick-read
 38: end-volume
 39:
 40: volume s5-md-cache
 41: type performance/md-cache
 42: subvolumes s5-open-behind
 43: end-volume
 44:
 45: volume s5
 46: type debug/io-stats
 47: option count-fop-hits off
 48: option latency-measurement off
 49: subvolumes s5-md-cache
 50: end-volume

+--+
[2015-04-30 05:14:58.422818] I [rpc-clnt.c:1676:rpc_clnt_reconfig]
0-s5-client-0: changing port to 49154 (from 0)
[2015-04-30 05:14:58.422887] W [socket.c:514:__socket_rwv] 0-s5-client-0:
readv failed (No data available)
[2015-04-30 05:14:58.423616] I
[client-handshake.c:1659:select_server_supported_programs] 0-s5-client-0:
Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-04-30 05:14:58.423836] I
[client-handshake.c:1456:client_setvolume_cbk] 0-s5-client-0: Connected to
172.16.100.1:49154, attached to remote volume '/gfs/data2/s5'.
[2015-04-30 05:14:58.423851] I
[client-handshake.c:1468:client_setvolume_cbk] 0-s5-client-0: Server and
Client lk-version numbers are not same, reopening the fds
[2015-04-30 05:14:58.427357] I [fuse-bridge.c:4769:fuse_graph_setup]
0-fuse: switched to graph 0
[2015-04-30 05:14:58.427561] I
[client-handshake.c:450:client_set_lk_version_cbk] 0-s5-client-0: Server lk
version = 1
[2015-04-30 05:14:58.427600] I [fuse-bridge.c:3724:fuse_init]
0-glusterfs-fuse: FU

Re: [Gluster-users] Poor performance with small files

[Ron Trompert]

Hi Ben,

Thanks for the info.

Cheers,

Ron


On 29/04/15 21:03, Ben Turner wrote:
> - Original Message -
>> From: "Ron Trompert" 
>> To: gluster-users@gluster.org
>> Sent: Wednesday, April 29, 2015 1:25:59 PM
>> Subject: [Gluster-users] Poor performance with small files
>>
>> Hi,
>>
>> We run gluster as storage solution for our Owncloud-based sync and share
>> service. At the moment we have about 30 million files in the system
>> which addup to a little more than  30TB. Most of these files are as you
>> may expect very small, i.e. in the 100KB ball park. For about a year
>> everything ran perfectly fine. We run 3.6.2 by the way.
> 
> Upgrade to 3.6.3 and set client.event-threads and server.event-threads to at 
> least 4:
> 
> "Previously, epoll thread did socket even-handling and the same thread was 
> used for serving the client or processing the response received from the 
> server. Due to this, other requests were in a queue untill the current epoll 
> thread completed its operation. With multi-threaded epoll, events are 
> distributed that improves the performance due the parallel processing of 
> requests/responses received."
> 
> Here are the guidelines for tuning them:
> 
> https://access.redhat.com/documentation/en-US/Red_Hat_Storage/3/html/Administration_Guide/Small_File_Performance_Enhancements.html
> 
> In my testing with epoll threads at 4 I saw a between a 15% and 50% increase 
> depending on the workload.
> 
> There are several smallfile perf enhancements in the works:
> 
> *http://www.gluster.org/community/documentation/index.php/Features/Feature_Smallfile_Perf
> 
> *Lookup unhashed is the next feature and should be ready with 3.7(correct me 
> if I am wrong).  
> 
> *If you are using RAID 6 you may want to do some testing with RAID 10 or 
> JBOD, but the benefits here only come into play with alot of concurrent 
> access(30+ processes / threads working with different files).
> 
> *Tiering may help here if you want to add some SSDs, this is also a 3.7 
> feature.
> 
> HTH!
> 
> -b
> 
>>
>> Now we are trying to commission new hardware. We have done this by
>> adding the new nodes to our cluster and using the add-brick and
>> remove-brick procedure to get the data to the new nodes. In a week we
>> have migrated only 8.5TB this way. What are we doing wrong here? Is
>> there a way to improve the gluster performance on small files?
>>
>> I have another question. If you want to setup a gluster that will
>> contain lots of very small files. What would be a good practice to set
>> things up in terms configuration, sizes of bricks related tot memory and
>> number of cores, number of brick per node etc.?
>>
>>
>>
>> Best regards and thanks in advance,
>>
>> Ron
>>
>>
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http://www.gluster.org/mailman/listinfo/gluster-users
>>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Large gluster read performance overhead?

[Raghuram BK]

Resending if anyone has ideas..

On Tue, Apr 28, 2015 at 3:32 PM, Raghuram BK  wrote:

>
> I've been trying to figure out the performance overhead of gluster over
> the underlying filesystem and find the difference to be quite stark. Is
> this normal? Can something be done about it? I've tried to eliminate the
> network overhead by doing everything locally and eliminate the effect of
> caching by forcing hits to the hard drives.. Here's what I did :
>
> 1. Force the underlying filesystem(ZFS) to always read from disk
> 2. Create the underlying storage (zfs create frzpool/normal/d1)
> 3. Create a gluster distributed volume with only one brick on the local
> machine. (gluster volume create g1
> fractalio-pri.fractalio.lan:/frzpool/normal/d1 force)
> 4. Start it (gluster volume start g1)
> 5. Check the volume info :
>
> [root@fractalio-pri fractalio]# gluster v info g1
>
> Volume Name: g1
> Type: Distribute
> Volume ID: e50f13d2-cb98-47f4-8113-3f15b4b6306a
> Status: Started
> Number of Bricks: 1
> Transport-type: tcp
> Bricks:
> Brick1: fractalio-pri.fractalio.lan:/frzpool/normal/d1
>
> 6. Mount it (mount -t glusterfs localhost:/g1 /mnt/g1)
> 7. Populate a test file into the volume :
>
> [root@fractalio-pri fractalio]# dd if=/dev/zero of=/mnt/g1/ddfile1 bs=1M
> count=2000
> 2000+0 records in
> 2000+0 records out
> 2097152000 bytes (2.1 GB) copied, 8.4938 s, 247 MB/s
>
> 8. Read the file from the gluster mount :
>
> [root@fractalio-pri fractalio]# dd if=/mnt/g1/ddfile1 of=/dev/zero bs=1M
> 2000+0 records in
> 2000+0 records out
> 2097152000 bytes (2.1 GB) copied, 84.4174 s, 24.8 MB/s
>
> 9. Read the file directly from the underlying storage :
>
> [root@fractalio-pri fractalio]# dd if=/frzpool/normal/d1/ddfile1
> of=/dev/zero bs=1M
> 2000+0 records in
> 2000+0 records out
> 2097152000 bytes (2.1 GB) copied, 24.722 s, 84.8 MB/s
>
>
> The throughput comes down from 84.8MB/s to 24.8MB/s, a 240% overhead?!
>



-- 

*Fractalio Data, India*

Mobile: +91 96635 92022

Email: r...@fractalio.com 
Web: www.fractalio.com
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] NFS vs native fuse

[Dan Mons]

Specific to Linux, the NFS client uses standard filesystem caching
which has a few pros and cons of it's own.

Native GlusterFS uses up application space RAM and is a hard-set
number that you must define.  In our studio, our standard rollout is a
32GB RAM workstation, so native GlusterFS clients are told to use
quite a bit of RAM for cache.  When we connect up smaller VMs, often
they have less RAM overall than we assign just for cache, and as such
use NFS instead.

-Dan


Dan Mons - R&D Sysadmin
Cutting Edge
http://cuttingedge.com.au


On 29 April 2015 at 20:01, Kingsley  wrote:
> On Tue, 2015-04-28 at 17:09 -0700, Dave Warren wrote:
>> Bandwidth is also a consideration, the FUSE client will upload multiple
>> copies based on the replica setting for the volume, so if the client is
>> connected at 100Mb/s or over wifi, and the servers are cross-connected
>> on a 10Gb/s backplane, having the client upload multiple copies vs
>> having the NFS server handle the replicas may have an impact on very
>> large files.
>>
>> Finally, NFS seems to have a lighter CPU footprint on the client, at the
>> possible cost of higher server CPU load, although this is anecdotal
>> (from my own experience), and probably a mixed bag.
>
> Oh I think I get it - the Gluster server daemons are also NFS server
> daemons, so mounting via NFS you're still talking to a Gluster daemon on
> the server ... and when writing to the server, the servers then handle
> the replication themselves?
>
> I thought people were just putting the path to a brick in /etc/exports
> and then just using nfsd that came with the OS, which I presume would
> then break things.
>
> In that case, would this be a reasonable summary of mounting via NFS
> instead of using the native fuse client?
>
> NFS pros:
>   * faster
>   * lighter weight on client resources (CPU, bandwidth)
>   * available to clients that have a standard NFS client but cannot
> install gluster-fuse
>
> NFS cons:
>   * more load on servers
>   * writes complete on the client before the data is replicated
> (which is faster, but less secure)
>   * if the server that the client happens to be connected to goes
> down, the client loses access to the volume (whereas the fuse
> client recovers and continues writing to the remaining notes)
>
> Does that seem about right?
>
> --
> Cheers,
> Kingsley.
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] VM hangs when one of the glusterfs server started

[Jay]

Hello guys

 

I've encountered a problem, i have a glusterfs cluster of 3 servers, 1
volume,which is a replicate volume:

 

[root@sy-host-1 ~]# gluster v info

 

Volume Name: img

Type: Replicate

Volume ID: 93e23097-1899-4396-9fca-cf592f4c7a25

Status: Started

Number of Bricks: 1 x 3 = 3

Transport-type: tcp

Bricks:

Brick1: 192.168.60.13:/ytxt/glusterfs

Brick2: 192.168.60.14:/ytxt/glusterfs

Brick3: 192.168.60.15:/ytxt/glusterfs

 

The volume is used as a shared filesystem for kvm, but when i was doing some
maintenance work on one of the servers, say, sy-host-2, i stopped glusterd
daemon, after the maintenance work done , i started glusterd daemon, but as
soon as the process was up, some of the vm was hanging, can not ssh to the
vm. I was thinking if there was something to do with glusterd daemon,so i
tried to stop the daemon just started, bang~~, everything was back to normal

 

As soon as sy-host-2 started the glusterd daemon, sy-host-1 showed up
something in the log(/var/log/glusterfs/glusterd.log):

 

[2015-04-28 06:08:35.342054] E
[client-handshake.c:1496:client_query_portmap_cbk] 0-img-client-1: failed to
get the port number for remote subvolume. Please run 'gluster volume status'
on server to see if brick process is running.

[2015-04-28 06:08:35.342223] I [client.c:2215:client_rpc_notify]
0-img-client-1: disconnected from img-client-1. Client process will keep
trying to connect to glusterd until brick's port is available

 

I ran the command "gluster volume status" on sy-host-2, nothing but normal

 

Any help is great appreciated

 

--

Jay

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Poor performance of NFS client for large writes compared to native client

[Behrooz Shafiee]

Hi,

 I was comparing GlusterFS native and NFS clients and I noticed, NFS client
is significantly slower for large writes. I wrote about 200, 1GB files
using a 1MB block sizes and NFS throughput was almost half of native
client. Can anyone explain why is that?

Thanks,
-- 
Behrooz
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Poor performance with small files

[Sander Zijlstra]

Hi Ben,

Thanks for the extensive response….

3.6.3 has been released recently hasn’t it?… we just upgraded to 3.6.2 two 
weeks ago because our new glusterfs servers were freshly installed and indeed 
got the latest packages available at the time of installation.

As the remove-brick operation is still running, can we stop this process 
without problems?… In the RedHat documents it’s stated as a “technology 
preview” but in the github documentation for glusterfs it’s stated that the 
data migration will simply stop and leave the data as is, at the time of 
stopping the remove-brick.

Met vriendelijke groet / kind regards,

Sander Zijlstra


> On 29 Apr 2015, at 21:03, Ben Turner  wrote:
> 
> - Original Message -
>> From: "Ron Trompert" 
>> To: gluster-users@gluster.org
>> Sent: Wednesday, April 29, 2015 1:25:59 PM
>> Subject: [Gluster-users] Poor performance with small files
>> 
>> Hi,
>> 
>> We run gluster as storage solution for our Owncloud-based sync and share
>> service. At the moment we have about 30 million files in the system
>> which addup to a little more than  30TB. Most of these files are as you
>> may expect very small, i.e. in the 100KB ball park. For about a year
>> everything ran perfectly fine. We run 3.6.2 by the way.
> 
> Upgrade to 3.6.3 and set client.event-threads and server.event-threads to at 
> least 4:
> 
> "Previously, epoll thread did socket even-handling and the same thread was 
> used for serving the client or processing the response received from the 
> server. Due to this, other requests were in a queue untill the current epoll 
> thread completed its operation. With multi-threaded epoll, events are 
> distributed that improves the performance due the parallel processing of 
> requests/responses received."
> 
> Here are the guidelines for tuning them:
> 
> https://access.redhat.com/documentation/en-US/Red_Hat_Storage/3/html/Administration_Guide/Small_File_Performance_Enhancements.html
> 
> In my testing with epoll threads at 4 I saw a between a 15% and 50% increase 
> depending on the workload.
> 
> There are several smallfile perf enhancements in the works:
> 
> *http://www.gluster.org/community/documentation/index.php/Features/Feature_Smallfile_Perf
> 
> *Lookup unhashed is the next feature and should be ready with 3.7(correct me 
> if I am wrong).
> 
> *If you are using RAID 6 you may want to do some testing with RAID 10 or 
> JBOD, but the benefits here only come into play with alot of concurrent 
> access(30+ processes / threads working with different files).
> 
> *Tiering may help here if you want to add some SSDs, this is also a 3.7 
> feature.
> 
> HTH!
> 
> -b
> 
>> 
>> Now we are trying to commission new hardware. We have done this by
>> adding the new nodes to our cluster and using the add-brick and
>> remove-brick procedure to get the data to the new nodes. In a week we
>> have migrated only 8.5TB this way. What are we doing wrong here? Is
>> there a way to improve the gluster performance on small files?
>> 
>> I have another question. If you want to setup a gluster that will
>> contain lots of very small files. What would be a good practice to set
>> things up in terms configuration, sizes of bricks related tot memory and
>> number of cores, number of brick per node etc.?
>> 
>> 
>> 
>> Best regards and thanks in advance,
>> 
>> Ron
>> 
>> 
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http://www.gluster.org/mailman/listinfo/gluster-users
>> 
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Disastrous performance with rsync to mounted Gluster volume.

[Joe Julian]

Yes, the (kernel) nfs client caches a lot more so, as long as the 
destination hasn't changed, you should be good. If it has, it's possible 
to compare your source with stale target metadata which can, in theory, 
lead to unexpected results.


On 04/29/2015 03:20 PM, David Robinson wrote:

Joe,

I switched my mounts from FUSE to NFS and the rsync is traversing the 
filesystem 5x faster.


Previously I had:

rsync -axv --numeric-ids --inplace -e 
\"/root/hpnssh_for_backup/bin/ssh -p222 -ax -oNoneSwitch=yes 
-oNoneEnabled=yes\" --delete --whole-file gfsib02b:/homegfs 
/backup/backup.0


where gfsib02b is one of the gluster nodes and homegfs is a FUSE 
mounted volume.  /backup is a FUSE mount on my backup machine.



I changed this to:

rsync -axv --numeric-ids --inplace -e 
\"/root/hpnssh_for_backup/bin/ssh -p222 -ax -oNoneSwitch=yes 
-oNoneEnabled=yes\" --delete --whole-file gfsib02b:/homegfs_nfs 
/backup/backup.0

where gfsib02b:/homegfs_nfs is an NFS mount of the same volume.
[root@gfs02b homegfs_nfs]# mount | grep homegfs
gfsib02b.corvidtec.com:/homegfs.tcp on /homegfs type fuse.glusterfs 
(rw,allow_other,max_read=131072)
gfsib02b.corvidtec.com:/homegfs on /homegfs_nfs type nfs 
(rw,vers=3,intr,bg,rsize=32768,wsize=32768,addr=10.200.71.2)


Is a 5x speed difference expected between the FUSE and NFS mounts?  
Note that very little data is being transferred.  And, when data is 
being transferred, the transfer rate seems fine.  My issue with 
gluster and rsync was that it seemed to be taking an extremely long 
time to figure out what to transfer and this seems to largely be 
mitigated by using NFS instead of FUSE on the server side (side where 
data to be backed up resides).  I am still using gluster for the 
backup machine as this seemed to have a very small affect on the timing.


David


-- Original Message --
From: "Joe Julian" 
To: gluster-users@gluster.org
Sent: 4/27/2015 6:56:04 PM
Subject: Re: [Gluster-users] Disastrous performance with rsync to 
mounted Gluster volume.





On 04/27/2015 03:00 PM, Ernie Dunbar wrote:

On 2015-04-27 14:09, Joe Julian wrote:


I've also noticed that if I increase the count of those writes, the
transfer speed increases as well:

2097152 bytes (2.1 MB) copied, 0.036291 s, 57.8 MB/s
root@backup:/home/webmailbak# dd if=/dev/zero of=/mnt/testfile
count=2048 bs=1024; sync
2048+0 records in
2048+0 records out
2097152 bytes (2.1 MB) copied, 0.0362724 s, 57.8 MB/s
root@backup:/home/webmailbak# dd if=/dev/zero of=/mnt/testfile
count=2048 bs=1024; sync
2048+0 records in
2048+0 records out
2097152 bytes (2.1 MB) copied, 0.0360319 s, 58.2 MB/s
root@backup:/home/webmailbak# dd if=/dev/zero of=/mnt/testfile
count=10240 bs=1024; sync
10240+0 records in
10240+0 records out
10485760 bytes (10 MB) copied, 0.127219 s, 82.4 MB/s
root@backup:/home/webmailbak# dd if=/dev/zero of=/mnt/testfile
count=10240 bs=1024; sync
10240+0 records in
10240+0 records out
10485760 bytes (10 MB) copied, 0.128671 s, 81.5 MB/s

This is correct, there is overhead that happens with small files and
the smaller the file the less throughput you get. That said, since
files are smaller you should get more files / second but less MB /
second. I have found that when you go under 16k changing files size
doesn't matter, you will get the same number of 16k files / second as
you do 1 k files.

 The overhead happens regardless. You just notice it more when you're
doing it a lot more frequently.



Well, it would be helpful to know what specifically rsync is trying 
to do when it's sitting there making overhead, and whether it's 
possible to tell rsync to avoid doing it, and just copy files 
instead (which it does quite quickly).


I suppose technically speaking, it's an rsync-specific question, but 
it's all about making rsync and glusterfs play nice, and we pretty 
much all need to know that!


Yes, that's very rsync specific. Rsync not only checks the files 
metadata, but it also does a hash comparison.


Each lookup() of each file requires a lookup from *each* replica 
server. Lookup's are triggered on open, or even fstat. Since rsync 
requests the stat of every file for comparison, this requires a 
little extra network time. The client queries all the replica in case 
one is out of date to ensure it's returning accurate results (and 
heal if a replica needs it).


After bulding a list of files that differ between the source and the 
target, rsync copies the file to a temporary filename. After 
completing the temporary file, rsync then renames the temporary to 
the target filename. This has the disadvantage of putting the target 
file on the "wrong" dht subvolume because the hash for the temporary 
filename is different from the target filename.


rsync's network transfer is over ssh by default, so you're also 
hitting encryption and buffer overhead.


The optimum process for *initially* copying large numbers of files to 
gluster would be to blindly copy a list of files from the source to 
the target without re

Re: [Gluster-users] Disastrous performance with rsync to mounted Gluster volume.

[David Robinson]

Joe,

I switched my mounts from FUSE to NFS and the rsync is traversing the 
filesystem 5x faster.


Previously I had:

rsync -axv --numeric-ids --inplace -e \"/root/hpnssh_for_backup/bin/ssh 
-p222 -ax -oNoneSwitch=yes -oNoneEnabled=yes\" --delete --whole-file 
gfsib02b:/homegfs /backup/backup.0


where gfsib02b is one of the gluster nodes and homegfs is a FUSE mounted 
volume.  /backup is a FUSE mount on my backup machine.



I changed this to:

rsync -axv --numeric-ids --inplace -e \"/root/hpnssh_for_backup/bin/ssh 
-p222 -ax -oNoneSwitch=yes -oNoneEnabled=yes\" --delete --whole-file 
gfsib02b:/homegfs_nfs /backup/backup.0

where gfsib02b:/homegfs_nfs is an NFS mount of the same volume.
[root@gfs02b homegfs_nfs]# mount | grep homegfs
gfsib02b.corvidtec.com:/homegfs.tcp on /homegfs type fuse.glusterfs 
(rw,allow_other,max_read=131072)
gfsib02b.corvidtec.com:/homegfs on /homegfs_nfs type nfs 
(rw,vers=3,intr,bg,rsize=32768,wsize=32768,addr=10.200.71.2)


Is a 5x speed difference expected between the FUSE and NFS mounts?  Note 
that very little data is being transferred.  And, when data is being 
transferred, the transfer rate seems fine.  My issue with gluster and 
rsync was that it seemed to be taking an extremely long time to figure 
out what to transfer and this seems to largely be mitigated by using NFS 
instead of FUSE on the server side (side where data to be backed up 
resides).  I am still using gluster for the backup machine as this 
seemed to have a very small affect on the timing.


David


-- Original Message --
From: "Joe Julian" 
To: gluster-users@gluster.org
Sent: 4/27/2015 6:56:04 PM
Subject: Re: [Gluster-users] Disastrous performance with rsync to 
mounted Gluster volume.





On 04/27/2015 03:00 PM, Ernie Dunbar wrote:

On 2015-04-27 14:09, Joe Julian wrote:


I've also noticed that if I increase the count of those writes, the
transfer speed increases as well:

2097152 bytes (2.1 MB) copied, 0.036291 s, 57.8 MB/s
root@backup:/home/webmailbak# dd if=/dev/zero of=/mnt/testfile
count=2048 bs=1024; sync
2048+0 records in
2048+0 records out
2097152 bytes (2.1 MB) copied, 0.0362724 s, 57.8 MB/s
root@backup:/home/webmailbak# dd if=/dev/zero of=/mnt/testfile
count=2048 bs=1024; sync
2048+0 records in
2048+0 records out
2097152 bytes (2.1 MB) copied, 0.0360319 s, 58.2 MB/s
root@backup:/home/webmailbak# dd if=/dev/zero of=/mnt/testfile
count=10240 bs=1024; sync
10240+0 records in
10240+0 records out
10485760 bytes (10 MB) copied, 0.127219 s, 82.4 MB/s
root@backup:/home/webmailbak# dd if=/dev/zero of=/mnt/testfile
count=10240 bs=1024; sync
10240+0 records in
10240+0 records out
10485760 bytes (10 MB) copied, 0.128671 s, 81.5 MB/s

This is correct, there is overhead that happens with small files and
the smaller the file the less throughput you get. That said, since
files are smaller you should get more files / second but less MB /
second. I have found that when you go under 16k changing files size
doesn't matter, you will get the same number of 16k files / second as
you do 1 k files.

 The overhead happens regardless. You just notice it more when you're
doing it a lot more frequently.



Well, it would be helpful to know what specifically rsync is trying to 
do when it's sitting there making overhead, and whether it's possible 
to tell rsync to avoid doing it, and just copy files instead (which it 
does quite quickly).


I suppose technically speaking, it's an rsync-specific question, but 
it's all about making rsync and glusterfs play nice, and we pretty 
much all need to know that!


Yes, that's very rsync specific. Rsync not only checks the files 
metadata, but it also does a hash comparison.


Each lookup() of each file requires a lookup from *each* replica 
server. Lookup's are triggered on open, or even fstat. Since rsync 
requests the stat of every file for comparison, this requires a little 
extra network time. The client queries all the replica in case one is 
out of date to ensure it's returning accurate results (and heal if a 
replica needs it).


After bulding a list of files that differ between the source and the 
target, rsync copies the file to a temporary filename. After completing 
the temporary file, rsync then renames the temporary to the target 
filename. This has the disadvantage of putting the target file on the 
"wrong" dht subvolume because the hash for the temporary filename is 
different from the target filename.


rsync's network transfer is over ssh by default, so you're also hitting 
encryption and buffer overhead.


The optimum process for *initially* copying large numbers of files to 
gluster would be to blindly copy a list of files from the source to the 
target without reading the target. If copying across a network, 
maximizing the packet size is also advantageous. I've found tar ( + pv 
if you want to monitor throughput) + netcat (or socat) to be much 
faster than rsync.

___
Gluster-users mailing list
Gluster-us

Re: [Gluster-users] client is terrible with large amount of small files

[Ben Turner]

- Original Message -
> From: "gjprabu" 
> To: "A Ghoshal" 
> Cc: gluster-users@gluster.org, gluster-users-boun...@gluster.org
> Sent: Wednesday, April 29, 2015 9:07:07 AM
> Subject: Re: [Gluster-users] client is terrible with large amount of small 
> files
> 
> Hi Ghoshal,
> 
> Please find the details below.
> 
> A) Glusterfs version
> glusterfs 3.6.2

Upgrade to 3.6.3 and set client.event-threads and server.event-threads to at 
least 4.  Here is a guide on tuning MT epoll:

https://access.redhat.com/documentation/en-US/Red_Hat_Storage/3/html/Administration_Guide/Small_File_Performance_Enhancements.html

> 
> B) volume configuration (gluster v  info)
> gluster volume info
> 
> 
> Volume Name: integvol
> Type: Replicate
> Volume ID: b8f3a19e-59bc-41dc-a55a-6423ec834492
> Status: Started
> Number of Bricks: 1 x 3 = 3
> Transport-type: tcp
> Bricks:
> Brick1: integ-gluster2:/srv/sdb1/brick
> Brick2: integ-gluster1:/srv/sdb1/brick
> Brick3: integ-gluster3:/srv/sdb1/brick
>
> 
> C) host linux version
> CentOS release 6.5 (Final)

Are your bricks on LVM?  Are you using thinp?  If so update to the latest 
kernel as thinp perf was really bad in 6.5 and early 6.6 kernels.

> 
> D) details about the kind of network you use to connect your servers making
> up your storage pool.
> We are connecting LAN to LAN there is no special network configuration done
> 
> Frome client we use to mount like below
> mount -t glusterfs gluster1:/integvol /mnt/gluster/
> 
> 
> Regards
> Prabu
> 
> 
> 
>  On Wed, 29 Apr 2015 17:58:16 +0530 A Ghoshal wrote
> 
> 
> 
> 
> Performance would largely depend upon setup. While I cannot think of any
> setup that would cause write to be this slow, if would help if you share the
> following details:
> 
> A) Glusterfs version
> B) volume configuration (gluster v  info)
> C) host linux version
> D) details about the kind of network you use to connect your servers making
> up your storage pool.
> 
> Thanks,
> Anirban
> 
> 
> 
> From: gjprabu < gjpr...@zohocorp.com >
> To: < gluster-users@gluster.org >
> Date: 04/29/2015 05:52 PM
> Subject: Re: [Gluster-users] client is terrible with large amount of small
> files
> Sent by: gluster-users-boun...@gluster.org
> 
> 
> 
> 
> Hi Team,
> 
> If anybody know the solution please share us.
> 
> Regards
> Prabu
> 
> 
> 
>  On Tue, 28 Apr 2015 19:32:40 +0530 gjprabu < gjpr...@zohocorp.com >
> wrote 
> Hi Team,
> 
> We are using glusterfs newly and testing data transfer part in client using
> fuse.glusterfs file system but it is terrible with large amount of small
> files (Large amount of small file 150MB of size it's writing around 18min).
> I can able copy small files and syncing between the server brick are working
> fine but it is terrible with large amount of small files.
> 
> if anybody please share the solution for the above issue.
> 
> Regards
> Prabu
> 
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
> 
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
> 
> 
> =-=-=
> Notice: The information contained in this e-mail
> message and/or attachments to it may contain
> confidential or privileged information. If you are
> not the intended recipient, any dissemination, use,
> review, distribution, printing or copying of the
> information contained in this e-mail message
> and/or attachments to it are strictly prohibited. If
> you have received this communication in error,
> please notify us by reply e-mail or telephone and
> immediately and permanently delete the message
> and any attachments. Thank you
> 
> 
> 
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Poor performance with small files

[Ben Turner]

- Original Message -
> From: "Ron Trompert" 
> To: gluster-users@gluster.org
> Sent: Wednesday, April 29, 2015 1:25:59 PM
> Subject: [Gluster-users] Poor performance with small files
> 
> Hi,
> 
> We run gluster as storage solution for our Owncloud-based sync and share
> service. At the moment we have about 30 million files in the system
> which addup to a little more than  30TB. Most of these files are as you
> may expect very small, i.e. in the 100KB ball park. For about a year
> everything ran perfectly fine. We run 3.6.2 by the way.

Upgrade to 3.6.3 and set client.event-threads and server.event-threads to at 
least 4:

"Previously, epoll thread did socket even-handling and the same thread was used 
for serving the client or processing the response received from the server. Due 
to this, other requests were in a queue untill the current epoll thread 
completed its operation. With multi-threaded epoll, events are distributed that 
improves the performance due the parallel processing of requests/responses 
received."

Here are the guidelines for tuning them:

https://access.redhat.com/documentation/en-US/Red_Hat_Storage/3/html/Administration_Guide/Small_File_Performance_Enhancements.html

In my testing with epoll threads at 4 I saw a between a 15% and 50% increase 
depending on the workload.

There are several smallfile perf enhancements in the works:

*http://www.gluster.org/community/documentation/index.php/Features/Feature_Smallfile_Perf

*Lookup unhashed is the next feature and should be ready with 3.7(correct me if 
I am wrong).  

*If you are using RAID 6 you may want to do some testing with RAID 10 or JBOD, 
but the benefits here only come into play with alot of concurrent access(30+ 
processes / threads working with different files).

*Tiering may help here if you want to add some SSDs, this is also a 3.7 feature.

HTH!

-b

> 
> Now we are trying to commission new hardware. We have done this by
> adding the new nodes to our cluster and using the add-brick and
> remove-brick procedure to get the data to the new nodes. In a week we
> have migrated only 8.5TB this way. What are we doing wrong here? Is
> there a way to improve the gluster performance on small files?
> 
> I have another question. If you want to setup a gluster that will
> contain lots of very small files. What would be a good practice to set
> things up in terms configuration, sizes of bricks related tot memory and
> number of cores, number of brick per node etc.?
> 
> 
> 
> Best regards and thanks in advance,
> 
> Ron
> 
> 
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
> 
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Poor performance with small files

[Ron Trompert]

Hi,

We run gluster as storage solution for our Owncloud-based sync and share
service. At the moment we have about 30 million files in the system
which addup to a little more than  30TB. Most of these files are as you
may expect very small, i.e. in the 100KB ball park. For about a year
everything ran perfectly fine. We run 3.6.2 by the way.

Now we are trying to commission new hardware. We have done this by
adding the new nodes to our cluster and using the add-brick and
remove-brick procedure to get the data to the new nodes. In a week we
have migrated only 8.5TB this way. What are we doing wrong here? Is
there a way to improve the gluster performance on small files?

I have another question. If you want to setup a gluster that will
contain lots of very small files. What would be a good practice to set
things up in terms configuration, sizes of bricks related tot memory and
number of cores, number of brick per node etc.?



Best regards and thanks in advance,

Ron


___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Questions on turning on Gluster volume profile

[Bharathram Sivasubramanian]

We came across the gluster volume profile link from the gluster.org website 
(http://www.gluster.org/community/documentation/index.php/Gluster_3.2:_Start_Profiling).

We are currently running a distributed, replicated gluster volume (16 bricks in 
total, replication between 2 bricks) and we would like to turn on the gluster 
volume profile to collect some I/O statistics about the volume. We have a few 
questions about the profile as follows:

-Will the system be adversely impacted if we turn on the volume profile 
(i.e. could the profile itself cause the volume to hang and/or increase the 
read/write latency or any other negative effects)?

-Do the volume profile add any additional logs to the brick and volume 
gluster logs (apart from the screen output of the statistics)? And if yes, can 
you quantify how much additional logging in terms of MB per day will be added?

-Would you recommend that we leave the profile turned on and running 
for lengthy periods of time (like for several days at a stretch), or is it 
meant to be turned on for only short periods of time?
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Updated website needs your help for content!

[Tuomas Kuosmanen]

> * Tigert is working on some new GlusterFS website layout
>   ideas.  Preview here: https://glusternew-tigert.rhcloud.com
> 
>   He'll start a mailing list thread about it shortly.

That said, here goes.

I put something up, it's just a skeleton so far, so it does look empty.

I would love to have some help with the content, not limited to:

  * What is Gluster? In one, two and five sentences? :-)

  * How would be the best way to install? We need to write the
instructions, and those should be simple and still sensible
so that you get a real, functional setup in the end that you
can use. Maybe some example use cases, like making an office
NAS with replication, or something? 

I guess the install guide is something people are working on
already, so that could help on this part.

I am good with pixels and try to write good content too, but I lack
the knowledge of Gluster and the good stuff we want to tell people
about it.

Also, test it with handheld / mobile devices and tablets etc - it should
adapt to screen size changes nicely. The pain of using a ruby framework 
to build the site does give us some nice tools to deal with them in 
return.

//Tuomas
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] REMINDER: Weekly Gluster Community meeting is in 30 mins!

[Justin Clift]

On 29 Apr 2015, at 12:30, Justin Clift  wrote:
> Reminder!!!
> 
> The weekly Gluster Community meeting is in 30 mins, in
> #gluster-meeting on IRC.
> 
> This is a completely public meeting, everyone is encouraged
> to attend and be a part of it. :)

Thanks for everyone for attending!

* 3.6.3 has been released and announced (thanks raghu!)

  3.6.4beta1 will be available in a week or so.

* 3.7.0beta1 (tarball) has been released.  We're still working
  on packages for it. ;)

  This will go into Fedora Rawhide too.

* 3.5.4beta1 will likely be ready by the start of next week.

* Tigert is working on some new GlusterFS website layout
  ideas.  Preview here: https://glusternew-tigert.rhcloud.com

  He'll start a mailing list thread about it shortly.

Meeting log:

  
https://meetbot.fedoraproject.org/gluster-meeting/2015-04-29/gluster-meeting.2015-04-29-12.01.html

Regards and best wishes,

Justin Clift

--
GlusterFS - http://www.gluster.org

An open source, distributed file system scaling to several
petabytes, and handling thousands of clients.

My personal twitter: twitter.com/realjustinclift

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] [Gluster-devel] Gluster Benchmark Kit

[Ben Turner]

- Original Message -
> From: "M S Vishwanath Bhat" 
> To: "Benjamin Turner" 
> Cc: "Kiran Patil" , gluster-users@gluster.org, "Gluster 
> Devel" ,
> btur...@redhat.com
> Sent: Wednesday, April 29, 2015 3:20:12 AM
> Subject: Re: [Gluster-devel] Gluster Benchmark Kit
> 
> On 28 April 2015 at 01:03, Benjamin Turner  wrote:
> 
> > Hi Kiran, thanks for the feedback!  I already put up a repo on githib:
> >
> > https://github.com/bennyturns/gluster-bench
> >
> > On my TODO list is:
> >
> > -The benchmark is currently RHEL / RHGS(Red Hat Gluster Storage) specific,
> > I want to make things work with at least non paid RPM distros and Ubuntu.
> > -Other filesystems(like you mentioned)
> > -No LVM and non thinp config options.
> > -EC, tiering, snapshot capabilities.
> >
> > I'll probably fork things and have a Red Hat specific version and an
> > upstream version.  As soon as I have everything working on Centos I'll let
> > the list know and we can enhance things to do whatever we need.  I always
> > thought it would be interesting if we had a page where people could submit
> > their benchmark data and the HW / config used.  Having a standard tool /
> > tool set will help there.
> >
> 
> Ben,
> 
> Do you think it is a good Idea (or is it possible) to integrate these with
> distaf? (https://github.com/gluster/distaf)

When I made this my goal was for someone to be able to build a cluster from 
scratch and run the full benchmark suite with only like 3-4 commands.  I 
specifically didn't use distaf to cut down on complexity, because the test 
tools were all already multinode capable, and I am already maintaining the 
rhs-system-init script I figured I would just reuse that for the setup.  That 
said I have been seeing more and more reasons to move things to distaf(FIO is 
not multi node capable, gather profiling data from each server, etc), and I was 
kicking around the idea of DISTAF-ifing it.  Let me fix everything to work with 
Centos/Fedora and get it into git.  From there we can look at distaf-ifing 
things, but I agree that is the way things should go.

-b
 
> That would enable us to choose workloads suitable to each scenario for
> single (set of) tests.
> 
> Best Regards,
> Vishwanath
> 
> 
> >
> > -b
> >
> >
> > On Mon, Apr 27, 2015 at 3:31 AM, Kiran Patil  wrote:
> >
> >> Hi,
> >>
> >> I came across "Gluster Benchmark Kit" while reading [Gluster-users]
> >> Disastrous performance with rsync to mounted Gluster volume thread.
> >>
> >> http://54.82.237.211/gluster-benchmark/gluster-bench-README
> >>
> >> http://54.82.237.211/gluster-benchmark
> >>
> >> The Kit includes tools such as iozone, smallfile and fio.
> >>
> >> This Kit is not documented and need to baseline this tool for Gluster
> >> Benchmark testing.
> >>
> >> The community is going to benefit by adopting and extending it as per
> >> their needs and the kit should be hosted on Github.
> >>
> >> The init.sh script in the Kit contains only XFS filesystem which can be
> >> extended to BTRFS and ZFS.
> >>
> >> Thanks Ben Turner for sharing it.
> >>
> >> Kiran.
> >>
> >> ___
> >> Gluster-devel mailing list
> >> gluster-de...@gluster.org
> >> http://www.gluster.org/mailman/listinfo/gluster-devel
> >>
> >>
> >
> > ___
> > Gluster-devel mailing list
> > gluster-de...@gluster.org
> > http://www.gluster.org/mailman/listinfo/gluster-devel
> >
> >
> 
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] client is terrible with large amount of small files

[gjprabu]

Hi Ghoshal,
 
 Please find the details below.

A) Glusterfs version 
glusterfs 3.6.2 

B) volume configuration (gluster v  info)
gluster volume info
 

Volume Name: integvol
Type: Replicate
Volume ID: b8f3a19e-59bc-41dc-a55a-6423ec834492
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: integ-gluster2:/srv/sdb1/brick
Brick2: integ-gluster1:/srv/sdb1/brick
Brick3: integ-gluster3:/srv/sdb1/brick

C) host linux version 
CentOS release 6.5 (Final)

D) details about the kind of network you use to connect your servers making up 
your storage pool. 
   We are connecting LAN to LAN there is no special network configuration done

Frome client we use to mount like below
mount -t glusterfs gluster1:/integvol /mnt/gluster/


Regards
Prabu





 On Wed, 29 Apr 2015 17:58:16 +0530 A Ghoshal 
wrote  

Performance would largely depend upon setup. While I cannot think of any setup 
that would cause write to be this slow, if would help if you share the 
following details: 
 
A) Glusterfs version 
B) volume configuration (gluster v  info) 
C) host linux version 
D) details about the kind of network you use to connect your servers making up 
your storage pool. 
 
Thanks, 
Anirban 
 
 
 
From:gjprabu  
To: 
Date:04/29/2015 05:52 PM 
Subject:Re: [Gluster-users] client is terrible with large amount of 
small files 
Sent by:gluster-users-boun...@gluster.org 
  
 
 
Hi Team,
 
  If anybody know the solution please share us.
  
Regards 
Prabu 
 
 

  On Tue, 28 Apr 2015 19:32:40 +0530 gjprabu  
wrote   
Hi Team,
 
  We are using glusterfs newly and testing data transfer part in client 
using fuse.glusterfs file system but it is terrible with large amount of small 
files (Large amount of small file 150MB of size it's writing around 18min). I 
can able copy small files and syncing between the server brick are working fine 
but it is terrible with large amount of small files. 
 
  if anybody please share the solution for the above issue.
   
Regards 
Prabu 
 
___ 
 Gluster-users mailing list 
 Gluster-users@gluster.org 
 http://www.gluster.org/mailman/listinfo/gluster-users 
 
___
 Gluster-users mailing list
 Gluster-users@gluster.org
 http://www.gluster.org/mailman/listinfo/gluster-users 
=-=-=
 Notice: The information contained in this e-mail
 message and/or attachments to it may contain 
 confidential or privileged information. If you are 
 not the intended recipient, any dissemination, use, 
 review, distribution, printing or copying of the 
 information contained in this e-mail message 
 and/or attachments to it are strictly prohibited. If 
 you have received this communication in error, 
 please notify us by reply e-mail or telephone and 
 immediately and permanently delete the message 
 and any attachments. Thank you
 




___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] client is terrible with large amount of small files

[A Ghoshal]

Performance would largely depend upon setup. While I cannot think of any 
setup that would cause write to be this slow, if would help if you share 
the following details:

A) Glusterfs version
B) volume configuration (gluster v  info)
C) host linux version
D) details about the kind of network you use to connect your servers 
making up your storage pool.

Thanks,
Anirban



From:   gjprabu 
To: 
Date:   04/29/2015 05:52 PM
Subject:Re: [Gluster-users] client is terrible with large amount 
of small files
Sent by:gluster-users-boun...@gluster.org



Hi Team,

  If anybody know the solution please share us.

Regards
Prabu



 On Tue, 28 Apr 2015 19:32:40 +0530 gjprabu  
wrote  
Hi Team,

  We are using glusterfs newly and testing data transfer part in 
client using fuse.glusterfs file system but it is terrible with large 
amount of small files (Large amount of small file 150MB of size it's 
writing around 18min). I can able copy small files and syncing between the 
server brick are working fine but it is terrible with large amount of 
small files. 

  if anybody please share the solution for the above issue.
 
Regards
Prabu

___ 
Gluster-users mailing list 
Gluster-users@gluster.org 
http://www.gluster.org/mailman/listinfo/gluster-users

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users
=-=-=
Notice: The information contained in this e-mail
message and/or attachments to it may contain 
confidential or privileged information. If you are 
not the intended recipient, any dissemination, use, 
review, distribution, printing or copying of the 
information contained in this e-mail message 
and/or attachments to it are strictly prohibited. If 
you have received this communication in error, 
please notify us by reply e-mail or telephone and 
immediately and permanently delete the message 
and any attachments. Thank you


___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] client is terrible with large amount of small files

[gjprabu]

Hi Team,

  If anybody know the solution please share us.

Regards
Prabu





 On Tue, 28 Apr 2015 19:32:40 +0530 gjprabu  
wrote  

Hi Team,

  We are using glusterfs newly and testing data transfer part in client 
using fuse.glusterfs file system but it is terrible with large amount of small 
files (Large amount of small file 150MB of size it's writing around 18min). I 
can able copy small files and syncing between the server brick are working fine 
but it is terrible with large amount of small files. 

  if anybody please share the solution for the above issue.
  
Regards
Prabu




___ 
Gluster-users mailing list 
Gluster-users@gluster.org 
http://www.gluster.org/mailman/listinfo/gluster-users




___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] REMINDER: Weekly Gluster Community meeting is in 30 mins!

[Justin Clift]

Reminder!!!

The weekly Gluster Community meeting is in 30 mins, in
#gluster-meeting on IRC.

This is a completely public meeting, everyone is encouraged
to attend and be a part of it. :)

To add Agenda items
***

Just add them to the main text of the Etherpad, and be at
the meeting. :)

 https://public.pad.fsfe.org/p/gluster-community-meetings

Regards and best wishes,

Justin Clift

--
GlusterFS - http://www.gluster.org

An open source, distributed file system scaling to several
petabytes, and handling thousands of clients.

My personal twitter: twitter.com/realjustinclift

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Poor performance small files

[Ron Trompert]

Hi,

We run gluster as storage solution for our Owncloud-based sync and share
service. At the moment we have about 30 million files in the system
which addup to a little more than  30TB. Most of these files are as you
may expect very small, i.e. in the 100KB ball park. For about a year
everything ran perfectly fine. We run 3.6.2 by the way.

Now we are trying to commission new hardware. We have done this by
adding the new nodes to our cluster and using the add-brick and
remove-brick procedure to get the data to the new nodes. In a week we
have migrated only 8.5TB this way. What are we doing wrong here? Is
there a way to improve the gluster performance on small files?

I have another question. If you want to setup a gluster that will
contain lots of very small files. What would be a good practice to set
things up in terms configuration, sizes of bricks related tot memory and
number of cores, number of brick per node etc.?



Best regards and thanks in advance,

Ron
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Shared IP address

[Niels de Vos]

On Wed, Apr 29, 2015 at 11:34:31AM +0200, Sharad Shukla wrote:
> Hi Susant,
> 
> I have installed Glusterfs in 2 machines which I want to use for
> establishing cluster. I am using CentOs 6.6. The gluster volume is set up
> and running fine. I am manually creating the files onto the mounted volume
> and they are replicating..
> 
> So far looks like everything is working. Now I have configured a shared ip
> address for reaching the replicated volume by using the following command:
> 
> gluster volume set [VOLUME] auth.allow [IP ADDRESS]
> 
> This IP address is visible to me under "gluster volume info", but I am
> unable to ping this ip address. I am getting the message that "Host is
> unreachable"

This option allows connecting *from* that IP-address to the volume. It
is not an option that automatically configures a virtual ip-address for
the trusted storage pool of your Gluster servers.

> I need this ip address reachable so that I can connect my application which
> needs to use this shared ip to connect to cluster.

You would need to use a cluster manager that handles IP-addresses and
their relocation on failure. Or a load-balancer that redirects the
IP-address to the actual servers.

An other option is to use the "backup-volfile-servers" or
"backupvolfile-server" mount option. That way you an specify a fall-back
server during the mount process. After the mounting, the client will
talk to all the bricks in the volume directly, and not use the virtual
IP-address anymore.

HTH,
Niels


pgpzvINsX7Zed.pgp
Description: PGP signature
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] NFS vs native fuse

[Kingsley]

On Tue, 2015-04-28 at 17:09 -0700, Dave Warren wrote:
> Bandwidth is also a consideration, the FUSE client will upload multiple 
> copies based on the replica setting for the volume, so if the client is 
> connected at 100Mb/s or over wifi, and the servers are cross-connected 
> on a 10Gb/s backplane, having the client upload multiple copies vs 
> having the NFS server handle the replicas may have an impact on very 
> large files.
> 
> Finally, NFS seems to have a lighter CPU footprint on the client, at the 
> possible cost of higher server CPU load, although this is anecdotal 
> (from my own experience), and probably a mixed bag.

Oh I think I get it - the Gluster server daemons are also NFS server
daemons, so mounting via NFS you're still talking to a Gluster daemon on
the server ... and when writing to the server, the servers then handle
the replication themselves?

I thought people were just putting the path to a brick in /etc/exports
and then just using nfsd that came with the OS, which I presume would
then break things.

In that case, would this be a reasonable summary of mounting via NFS
instead of using the native fuse client?

NFS pros:
  * faster
  * lighter weight on client resources (CPU, bandwidth)
  * available to clients that have a standard NFS client but cannot
install gluster-fuse

NFS cons:
  * more load on servers
  * writes complete on the client before the data is replicated
(which is faster, but less secure)
  * if the server that the client happens to be connected to goes
down, the client loses access to the volume (whereas the fuse
client recovers and continues writing to the remaining notes)

Does that seem about right?

-- 
Cheers,
Kingsley.

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Shared IP address

[Alex Crow]

On 29/04/15 10:34, Sharad Shukla wrote:

Hi Susant,

I have installed Glusterfs in 2 machines which I want to use for 
establishing cluster. I am using CentOs 6.6. The gluster volume is set 
up and running fine. I am manually creating the files onto the mounted 
volume and they are replicating..


So far looks like everything is working. Now I have configured a 
shared ip address for reaching the replicated volume by using the 
following command:


gluster volume set[VOLUME]auth.allow[IP ADDRESS]

This IP address is visible to me under "gluster volume info", but I am 
unable to ping this ip address. I am getting the message that "Host is 
unreachable"


I need this ip address reachable so that I can connect my application 
which needs to use this shared ip to connect to cluster.


This application is a little urgent for me. I would really appreciate 
your help..


Thanks
Sharad


Hi,

That command is just setting an acl for that IP to connect to the 
gluster daemon.


Normally you don't need to add any IP addresses as glusterfs clients 
will know about both servers as soon as they connect to one of them. If 
you're using Samba or NFS the preferred option is to use CTDB to spread 
a pool of virtual IP addresses over the servers.


Cheers

Alex




--
This message has been scanned for viruses and
dangerous content by *MailScanner* , and is
believed to be clean.


___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users


--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
"Transact" is operated by Integrated Financial Arrangements plc. 29
Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608
5300. (Registered office: as above; Registered in England and Wales
under number: 3727592). Authorised and regulated by the Financial
Conduct Authority (entered on the Financial Services Register; no. 190856).

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Shared IP address

[Sharad Shukla]

Hi Susant,

I have installed Glusterfs in 2 machines which I want to use for
establishing cluster. I am using CentOs 6.6. The gluster volume is set up
and running fine. I am manually creating the files onto the mounted volume
and they are replicating..

So far looks like everything is working. Now I have configured a shared ip
address for reaching the replicated volume by using the following command:

gluster volume set [VOLUME] auth.allow [IP ADDRESS]

This IP address is visible to me under "gluster volume info", but I am
unable to ping this ip address. I am getting the message that "Host is
unreachable"

I need this ip address reachable so that I can connect my application which
needs to use this shared ip to connect to cluster.

This application is a little urgent for me. I would really appreciate your
help..

Thanks
Sharad
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] [Gluster-devel] Gluster Benchmark Kit

[M S Vishwanath Bhat]

On 28 April 2015 at 01:03, Benjamin Turner  wrote:

> Hi Kiran, thanks for the feedback!  I already put up a repo on githib:
>
> https://github.com/bennyturns/gluster-bench
>
> On my TODO list is:
>
> -The benchmark is currently RHEL / RHGS(Red Hat Gluster Storage) specific,
> I want to make things work with at least non paid RPM distros and Ubuntu.
> -Other filesystems(like you mentioned)
> -No LVM and non thinp config options.
> -EC, tiering, snapshot capabilities.
>
> I'll probably fork things and have a Red Hat specific version and an
> upstream version.  As soon as I have everything working on Centos I'll let
> the list know and we can enhance things to do whatever we need.  I always
> thought it would be interesting if we had a page where people could submit
> their benchmark data and the HW / config used.  Having a standard tool /
> tool set will help there.
>

Ben,

Do you think it is a good Idea (or is it possible) to integrate these with
distaf? (https://github.com/gluster/distaf)

That would enable us to choose workloads suitable to each scenario for
single (set of) tests.

Best Regards,
Vishwanath


>
> -b
>
>
> On Mon, Apr 27, 2015 at 3:31 AM, Kiran Patil  wrote:
>
>> Hi,
>>
>> I came across "Gluster Benchmark Kit" while reading [Gluster-users]
>> Disastrous performance with rsync to mounted Gluster volume thread.
>>
>> http://54.82.237.211/gluster-benchmark/gluster-bench-README
>>
>> http://54.82.237.211/gluster-benchmark
>>
>> The Kit includes tools such as iozone, smallfile and fio.
>>
>> This Kit is not documented and need to baseline this tool for Gluster
>> Benchmark testing.
>>
>> The community is going to benefit by adopting and extending it as per
>> their needs and the kit should be hosted on Github.
>>
>> The init.sh script in the Kit contains only XFS filesystem which can be
>> extended to BTRFS and ZFS.
>>
>> Thanks Ben Turner for sharing it.
>>
>> Kiran.
>>
>> ___
>> Gluster-devel mailing list
>> gluster-de...@gluster.org
>> http://www.gluster.org/mailman/listinfo/gluster-devel
>>
>>
>
> ___
> Gluster-devel mailing list
> gluster-de...@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel
>
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users