[Gluster-users] Need help with optimizing GlusterFS for Apache
Our webserver is configured as such: The actual website files, php, html ,css and so on. Or on a dedicated non-glusterfs ext4 partition. However, the website access Videos and especially image files on a gluster mounted directory. The write performance for our backend gluster storage is not that important. Since it only comes into play when someone uploads a video or image. However, the files are being read from constantly and randomly. Especially all the small image files which get read everytime someone loads our website. It is quite a busy site, with somewhere along the line of 500 apache accesses per second during our busy hours. I've been taking a look at performance.cache-size and translators. But I'm not quite sure what the ideal settings would be. Can anyone help me out with some settings to try out, based on experience they've had? Thanks. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Confused about Geo Replication..
On 10/24/2011 12:44 PM, Robert Krig wrote: On 10/24/2011 11:11 AM, Venky Shankar wrote: On 10/24/2011 02:16 PM, Robert Krig wrote: I've been reading the documentation, but I'm a bit stumped as to how to setup geo-replication with glusterfs. The documentation mentions that you use an existing glusterfs volume to start geo replication. But what kind of volume? Do you just create a standard replicated volume with a replica count of 1? Geo-rep only needs the volume name. You can create volume of any type (plain distribute, replicate, distribute-replicate etc...) and use it as the master volume for geo-replication. Suppose you have a distribute-replicate volume called 'vol-dr' then you start geo-rep by: # gluster volume geo-replication vol-dr slave start So if I want to just create a simple 2 server masterslave setup, what would be the best choice for the initial Gluster volume? Replicated with a single node? ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users I've been trying a few things out, it seems for geo-replication you need a third node. Or am I overlooking something? I would like to try to setup a A-B geo-replication setup. One node being replicated by a slave node. But if I try to create a replicated volume, glusterfs insists on at least two bricks. Or should I try creating a distributed gluster volume with just one brick? ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] Confused about Geo Replication..
I've been reading the documentation, but I'm a bit stumped as to how to setup geo-replication with glusterfs. The documentation mentions that you use an existing glusterfs volume to start geo replication. But what kind of volume? Do you just create a standard replicated volume with a replica count of 1? Could somebody outline the steps for geo-replication? Also, does geo-replication perform better than replication? In our setup we need redundancy, but more as a fail-safe backup option, rather than a load balancing setup. We just want to be able to quickly switch to our backup server, if our primary server should go down for any reason. Would a geo-replication setup with GlusterFS have better read and write performance, since the slave can lag behind a bit? ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Confused about Geo Replication..
On 10/24/2011 11:11 AM, Venky Shankar wrote: On 10/24/2011 02:16 PM, Robert Krig wrote: I've been reading the documentation, but I'm a bit stumped as to how to setup geo-replication with glusterfs. The documentation mentions that you use an existing glusterfs volume to start geo replication. But what kind of volume? Do you just create a standard replicated volume with a replica count of 1? Geo-rep only needs the volume name. You can create volume of any type (plain distribute, replicate, distribute-replicate etc...) and use it as the master volume for geo-replication. Suppose you have a distribute-replicate volume called 'vol-dr' then you start geo-rep by: # gluster volume geo-replication vol-dr slave start So if I want to just create a simple 2 server masterslave setup, what would be the best choice for the initial Gluster volume? Replicated with a single node? ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Optimal XFS formatting?
Try using ext4 if you can. Small file read performance will be MUCH better than xfs. On the other hand, you might wanna run some benchmark tests which resemble your workload, to compare xfs vs ext4 both with and without glusterfs. On 10/20/2011 03:36 PM, Sabuj Pattanayek wrote: Hi, I've seen that EXT4 has better random I/O performance than XFS, especially on small reads and writes. For large sequential reads and writes XFS is a little bit better. For really large sequential reads and write EXT4 and XFS are about the same. I used to format XFS using this: mkfs.xfs -l size=64m (notes from http://everything2.com/title/Filesystem+performance+tweaking+with+XFS+on+Linux) but realized that it doesn't seem to effect performance for me. You should definitely try mounting with this : mount -t xfs -o rw,noatime,nodiratime,logbufs=8 HTH, Sabuj On Thu, Oct 20, 2011 at 8:18 AM, Gerald Brandt g...@majentis.com wrote: Hi, Are there any 'optimal' settings for XFS formatting under GlusterFS? The storage will be used for Virtual Disk storage, virtual disk size from 8GB to 100 GB in size. One of the VM's (separate gluster volume) will be running MSSQL server (4K reads and writes). The other will be running file servers, etc). Thanks, Gerald ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Need help with optimizing GlusterFS for Apache
On 10/17/2011 05:16 PM, Robert Krig wrote: Our webserver is configured as such: The actual website files, php, html ,css and so on. Or on a dedicated non-glusterfs ext4 partition. However, the website access Videos and especially image files on a gluster mounted directory. The write performance for our backend gluster storage is not that important. Since it only comes into play when someone uploads a video or image. However, the files are being read from constantly and randomly. Especially all the small image files which get read everytime someone loads our website. It is quite a busy site, with somewhere along the line of 500 apache accesses per second during our busy hours. I've been taking a look at performance.cache-size and translators. But I'm not quite sure what the ideal settings would be. Can anyone help me out with some settings to try out, based on experience they've had? Thanks. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users I think I'm going to have to abandon GlusterFS for our Image files. The performance is abysmal. I've tried all sorts of settings, but at some point the http process just keeps spawning more and more processess because clients are waiting because the directory can't be read, since glusterfs is busy. We're not even reaching 500 apache requests per second and already apache locks up. I'm pretty sure it can't be the hardware, since we're talking about a 12 Core Hyperthreading Xeon CPU, with 48GB of ram and 30TB of storage in a hardware Raid. I realise that GlusterFS is not ideal for many small files, but this is beyond ridiculous. It certainly doesn't help that the documentation doesn't even properly explain how to activate different translators, or where exactly to edit them by hand in the config files. If anyone has any suggestions, I'd be happy to hear them. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] Need help with optimizing GlusterFS for Apache
Our webserver is configured as such: The actual website files, php, html ,css and so on. Or on a dedicated non-glusterfs ext4 partition. However, the website access Videos and especially image files on a gluster mounted directory. The write performance for our backend gluster storage is not that important. Since it only comes into play when someone uploads a video or image. However, the files are being read from constantly and randomly. Especially all the small image files which get read everytime someone loads our website. It is quite a busy site, with somewhere along the line of 500 apache accesses per second during our busy hours. I've been taking a look at performance.cache-size and translators. But I'm not quite sure what the ideal settings would be. Can anyone help me out with some settings to try out, based on experience they've had? Thanks. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Sync Check
On 09/29/2011 10:49 AM, Tobias Matthaeus wrote: Hello everyone, is there any method to check, if the bricks are in sync? Thanks and Regards Tobi I usually compare the sizes of df of each brick. However, that doesn't necessarily mean that they are in perfect sync. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Is gluster suitable and production ready foremail/webservers?
On 09/26/2011 07:34 AM, Emmanuel Noobadmin wrote: I've been leaning towards actually deploying gluster in one of my projects for a while and finally a probable candidate project came up. However, researching into the specific use case, it seems that gluster isn't really suitable for load profiles that deal with lots of concurrent small files. e.g. http://www.techforce.com.br/news/linux_blog/glusterfs_tuning_small_files http://rackerhacker.com/2010/12/02/keep-web-servers-in-sync-with-drbd-and-ocfs2/ http://bugs.gluster.com/show_bug.cgi?id=2869 http://gluster.org/pipermail/gluster-users/2011-June/007970.html The first two are rather old so maybe the situation has changed. But the bug report and mailing list issue in June ring alarm bells. Is gluster really unsuited for this kind of workload or have things improved since then? ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users I guess the question to ask here is, do you need a lot of read/write performance for your application, or is redundancy and synchronisation more important? In my own tests I used rsync to transfer 14TB of data to our new two glusterfs storage nodes. The data was composed of about 500GB of small jpegs and the rest was video files. As you can guess, rsync is not so good with lots of small files, at least not THAT many small files, so with a 10Gigabit ethernet connection, on the small files we got about 10-30 megabytes per second. Once we got to the big files, we managed about 100-150megabytes /per second. Definitely not the maximum the system was capable of, but then again, these weren't ideal testing conditions. A simple dd if=/dev/zero | pv | dd of=/storage/testfile.dmp on a locally mounted glusterfsmount resulted in about 200-250megabytes /s. Of course an iperf between the two nodes resulted in a maximum network speed of around 5 gigabits/s. Of course, regardless of what other people might have experienced. Your best bet ist to test it with your own equipment. There are so many variables between differing distros, kernels, optimisations, and hardware, it's hard to guarantee any kind of minimum performance. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Is gluster suitable and production ready foremail/webservers?
On 09/26/2011 03:04 PM, Emmanuel Noobadmin wrote: As you can guess, rsync is not so good with lots of small files, at least not THAT many small files, so with a 10Gigabit ethernet connection, on the small files we got about 10-30 megabytes per second. 10~30MB/s is more than OK for me. However, you're on 10G while my client has a budget I need to work within so bonded 1G with VLAN is probably the best I can do. Any idea/data on how much an impact that might make? I forgot to mention that our 10gigabit was also a shared vlan. We have a dedicated external ip and virtual internal one on a single 10gbe Ethernet Interface. However, I don't how much of an impact it would make with just a 1gbit vlan. I have only just begun using glusterfs, and this is my first server using 10gbe ethernet, so it might be that there is still some performance gain that is available through some tuning. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] Self Heal fails...
Hi there. I'm new to GlusterFS. I'm currently evaluating it for production usage. I have two Storage Servers which use JFS as a filesystem for the underlying export. The setup is supposed to be replicated. I've been experimenting with various settings for benchmarking and such, as well as trying out different failure scenarios. Anyways, the export directory on node 1 is out of sync with node 2. So I mounted the storage volume via glusterfs client on node1 in another directory. The fuse mounted directory is /storage As per Manual I tried doing the find gluster-mount -noleaf -print0 | xargs --null stat /dev/null dance, however the logs throw a bunch of errors: # [2011-09-16 18:29:33.759729] E [client3_1-fops.c:1216:client3_1_inodelk_cbk] 0-GLSTORAGE-client-0: error [2011-09-16 18:29:33.759747] I [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-GLSTORAGE-client-0: remote operation failed: Invalid argument [2011-09-16 18:29:33.759942] E [afr-self-heal-metadata.c:672:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-GLSTORAGE-replicate-0: Non Blocking metadata inodelks failed for /. [2011-09-16 18:29:33.759961] E [afr-self-heal-metadata.c:674:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-GLSTORAGE-replicate-0: Metadata self-heal failed for /. [2011-09-16 18:29:33.760167] W [rpc-common.c:64:xdr_to_generic] (--/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x7d) [0x7f4702a751ad] (--/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5) [0x7f4702a74de5] (--/usr/local/lib/glusterfs/3.2.3/xlator/protocol/client.so(client3_1_entrylk_cbk+0x52) [0x7f46ff88a572]))) 0-xdr: XDR decoding failed [2011-09-16 18:29:33.760200] E [client3_1-fops.c:1292:client3_1_entrylk_cbk] 0-GLSTORAGE-client-0: error [2011-09-16 18:29:33.760215] I [client3_1-fops.c:1303:client3_1_entrylk_cbk] 0-GLSTORAGE-client-0: remote operation failed: Invalid argument [2011-09-16 18:29:33.760417] E [afr-self-heal-entry.c:2292:afr_sh_post_nonblocking_entry_cbk] 0-GLSTORAGE-replicate-0: Non Blocking entrylks failed for /. [2011-09-16 18:29:33.760447] E [afr-self-heal-common.c:1554:afr_self_heal_completion_cbk] 0-GLSTORAGE-replicate-0: background meta-data entry self-heal failed on / [2011-09-16 18:29:33.760808] I [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-GLSTORAGE-client-0: remote operation failed: Invalid argument ### Is this normal? The directory in question already has 150GB of data, so the find command is still running. Will it be ok once it finishes? from what I understand from the manual, the files should repair as the find process runs, or did I misinterpret that? If self heal should fail, is there a failsafe method to ensure that both nodes are in sync again? ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Self Heal fails...
On 09/16/2011 06:36 PM, Robert Krig wrote: Hi there. I'm new to GlusterFS. I'm currently evaluating it for production usage. I have two Storage Servers which use JFS as a filesystem for the underlying export. The setup is supposed to be replicated. I've been experimenting with various settings for benchmarking and such, as well as trying out different failure scenarios. Anyways, the export directory on node 1 is out of sync with node 2. So I mounted the storage volume via glusterfs client on node1 in another directory. The fuse mounted directory is /storage As per Manual I tried doing the find gluster-mount -noleaf -print0 | xargs --null stat /dev/null dance, however the logs throw a bunch of errors: # [2011-09-16 18:29:33.759729] E [client3_1-fops.c:1216:client3_1_inodelk_cbk] 0-GLSTORAGE-client-0: error [2011-09-16 18:29:33.759747] I [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-GLSTORAGE-client-0: remote operation failed: Invalid argument [2011-09-16 18:29:33.759942] E [afr-self-heal-metadata.c:672:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-GLSTORAGE-replicate-0: Non Blocking metadata inodelks failed for /. [2011-09-16 18:29:33.759961] E [afr-self-heal-metadata.c:674:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-GLSTORAGE-replicate-0: Metadata self-heal failed for /. [2011-09-16 18:29:33.760167] W [rpc-common.c:64:xdr_to_generic] (--/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x7d) [0x7f4702a751ad] (--/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5) [0x7f4702a74de5] (--/usr/local/lib/glusterfs/3.2.3/xlator/protocol/client.so(client3_1_entrylk_cbk+0x52) [0x7f46ff88a572]))) 0-xdr: XDR decoding failed [2011-09-16 18:29:33.760200] E [client3_1-fops.c:1292:client3_1_entrylk_cbk] 0-GLSTORAGE-client-0: error [2011-09-16 18:29:33.760215] I [client3_1-fops.c:1303:client3_1_entrylk_cbk] 0-GLSTORAGE-client-0: remote operation failed: Invalid argument [2011-09-16 18:29:33.760417] E [afr-self-heal-entry.c:2292:afr_sh_post_nonblocking_entry_cbk] 0-GLSTORAGE-replicate-0: Non Blocking entrylks failed for /. [2011-09-16 18:29:33.760447] E [afr-self-heal-common.c:1554:afr_self_heal_completion_cbk] 0-GLSTORAGE-replicate-0: background meta-data entry self-heal failed on / [2011-09-16 18:29:33.760808] I [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-GLSTORAGE-client-0: remote operation failed: Invalid argument ### Is this normal? The directory in question already has 150GB of data, so the find command is still running. Will it be ok once it finishes? from what I understand from the manual, the files should repair as the find process runs, or did I misinterpret that? If self heal should fail, is there a failsafe method to ensure that both nodes are in sync again? ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users Well, the find process has finished in the meantime, and as expected, it didn't fix anything. here are the last few lines of the client mount log: ## 2011-09-16 18:48:45.287954] I [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-GLSTORAGE-client-0: remote operation failed: Invalid argument [2011-09-16 18:48:45.288394] I [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-GLSTORAGE-client-0: remote operation failed: Invalid argument [2011-09-16 18:48:45.288921] I [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-GLSTORAGE-client-0: remote operation failed: Invalid argument [2011-09-16 18:48:45.289535] I [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-GLSTORAGE-client-0: remote operation failed: Invalid argument [2011-09-16 18:48:45.290063] I [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-GLSTORAGE-client-0: remote operation failed: Invalid argument [2011-09-16 18:48:45.290649] I [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-GLSTORAGE-client-0: remote operation failed: Invalid argument [2011-09-16 18:48:45.291126] I [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-GLSTORAGE-client-0: remote operation failed: Invalid argument [2011-09-16 20:14:52.289901] W [rpc-common.c:64:xdr_to_generic] (--/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x7d) [0x7f4702a751ad] (--/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5) [0x7f4702a74de5] (--/usr/local/lib/glusterfs/3.2.3/xlator/protocol/client.so(client3_1_statfs_cbk+0x71) [0x7f46ff88b741]))) 0-xdr: XDR decoding failed [2011-09-16 20:14:52.289928] E [client3_1-fops.c:624:client3_1_statfs_cbk] 0-GLSTORAGE-client-0: error [2011-09-16 20:14:52.289939] I [client3_1-fops.c:637:client3_1_statfs_cbk] 0-GLSTORAGE-client-0: remote operation failed: Invalid argument
Re: [Gluster-users] Self Heal fails...
I'm using GlusterFS version 3.2.3 built from the sources of the gluster.org website. I think I've found a way. I've shutdown my volume, detached the peers and basically recreated my storage volume from scratch. This time I started the setup with probing a peer from the node that had the up to date data in its underlying storage directory. Then I created the Volume again from scratch, this time entering node2:/export first and then node1:/export. Then I mounted the Gluster Volume locally and am currently running the find one liner on it. Judging from the logs, it seems to be rebuilding. I'm just wondering if there is perhaps a more elegant way to force a resync. It would be nice if there was a feature or a command, so that you can say: ok Node2, you are the main source, node1 listen to what node2 has to say. On 09/16/2011 08:31 PM, Burnash, James wrote: Hi Robert. Can you tell us what version you are running? That helps nail down if this is a known bug in a specific version. James Burnash Unix Engineer Knight Capital Group -Original Message- From: gluster-users-boun...@gluster.org [mailto:gluster-users-boun...@gluster.org] On Behalf Of Robert Krig Sent: Friday, September 16, 2011 2:17 PM To: gluster-users@gluster.org Subject: Re: [Gluster-users] Self Heal fails... On 09/16/2011 06:36 PM, Robert Krig wrote: Hi there. I'm new to GlusterFS. I'm currently evaluating it for production usage. I have two Storage Servers which use JFS as a filesystem for the underlying export. The setup is supposed to be replicated. I've been experimenting with various settings for benchmarking and such, as well as trying out different failure scenarios. Anyways, the export directory on node 1 is out of sync with node 2. So I mounted the storage volume via glusterfs client on node1 in another directory. The fuse mounted directory is /storage As per Manual I tried doing the find gluster-mount -noleaf -print0 | xargs --null stat /dev/null dance, however the logs throw a bunch of errors: ## ### [2011-09-16 18:29:33.759729] E [client3_1-fops.c:1216:client3_1_inodelk_cbk] 0-GLSTORAGE-client-0: error [2011-09-16 18:29:33.759747] I [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-GLSTORAGE-client-0: remote operation failed: Invalid argument [2011-09-16 18:29:33.759942] E [afr-self-heal-metadata.c:672:afr_sh_metadata_post_nonblocking_inodelk _cbk] 0-GLSTORAGE-replicate-0: Non Blocking metadata inodelks failed for /. [2011-09-16 18:29:33.759961] E [afr-self-heal-metadata.c:674:afr_sh_metadata_post_nonblocking_inodelk _cbk] 0-GLSTORAGE-replicate-0: Metadata self-heal failed for /. [2011-09-16 18:29:33.760167] W [rpc-common.c:64:xdr_to_generic] (--/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x7d) [0x7f4702a751ad] (--/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5) [0x7f4702a74de5] (--/usr/local/lib/glusterfs/3.2.3/xlator/protocol/client.so(client3_1 _entrylk_cbk+0x52) [0x7f46ff88a572]))) 0-xdr: XDR decoding failed [2011-09-16 18:29:33.760200] E [client3_1-fops.c:1292:client3_1_entrylk_cbk] 0-GLSTORAGE-client-0: error [2011-09-16 18:29:33.760215] I [client3_1-fops.c:1303:client3_1_entrylk_cbk] 0-GLSTORAGE-client-0: remote operation failed: Invalid argument [2011-09-16 18:29:33.760417] E [afr-self-heal-entry.c:2292:afr_sh_post_nonblocking_entry_cbk] 0-GLSTORAGE-replicate-0: Non Blocking entrylks failed for /. [2011-09-16 18:29:33.760447] E [afr-self-heal-common.c:1554:afr_self_heal_completion_cbk] 0-GLSTORAGE-replicate-0: background meta-data entry self-heal failed on / [2011-09-16 18:29:33.760808] I [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-GLSTORAGE-client-0: remote operation failed: Invalid argument ## # Is this normal? The directory in question already has 150GB of data, so the find command is still running. Will it be ok once it finishes? from what I understand from the manual, the files should repair as the find process runs, or did I misinterpret that? If self heal should fail, is there a failsafe method to ensure that both nodes are in sync again? ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users Well, the find process has finished in the meantime, and as expected, it didn't fix anything. here are the last few lines of the client mount log: ## 2011-09-16 18:48:45.287954] I [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-GLSTORAGE-client-0: remote operation failed: Invalid argument [2011-09-16 18:48:45.288394] I [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-GLSTORAGE-client-0: remote operation failed: Invalid argument [2011-09-16 18:48:45.288921
