Re: [Gluster-devel] Upcalls Infrastructure

[Soumya Koduri]

Hi,

Below patches include preliminary upcall framework support but has only 
'cache_invalidation' use-case handled.


http://review.gluster.org/#/c/9534/
http://review.gluster.org/#/c/9535/
http://review.gluster.org/#/c/9536/

Kindly review the changes.

Lease_Lock support changes will be submitted in the new patches after 
addressing the proposed changes discussed in the earlier mail.


Thanks,
Soumya


On 02/19/2015 12:30 PM, Soumya Koduri wrote:

Hi,

We recently have uncovered few issues with respect to lease_locks
support and had discussions around the same. Thanks to everyone involved.

So the new changes proposed in the design (in addition to the ones
discussed in the earlier mail) are -

* Earlier, in case if a client takes a lease-lock and a conflicting fop
is requested by another client, RECALL_LEASE CBK event will be sent to
the first client and till the first client unlocks the LEASE_LOCK, we
send EDELAY/ERETRY error to the conflicting fops. This works for
protocol clients (like NFS/SMB) which keep retrying on receiving that
error but not for FUSE clients or any of the other auxiliary services
(like rebalance/self-heal/quota) which will error-out immediately.

So to resolve that, we choose to block the fops based on the flags
passed (by default 'BLOCK' or 'NON_BLOCK' incase of protocol clients).
The blocking will be done in the same way as current locks xlator does
to block lock requests (maintain a queue of call stubs and wake them up
once the LEASE_LOCK is released/recalled).

* Earlier, when a lease_lk request comes, the upcall xlator maps it to
POSIX lock for the entire file before granting it. And incase if the
same client takes an fcntl lock, it will be merged with the earlier lock
taken and unlock of either of the locks will result in the loss of lock
state.

To avoid that, we plan to define a new lk_entry (LEASE_LOCK) in the
'locks' xlator to store lease_locks and add support to not merge it with
the locks of any other type.

* In addition, before granting lease_lock, we now check if there are
existing open-fds on the file with the conflicting access requested. If
yes, lease_lock will not be granted.

* While sending RECALL_LEASE CBK event, a new timer event will be
registered to notify in case of recall timeout so that we can purge
lease locks forcefully and wake up blocked fops.

* Few Enhancements which may be considered,
 *   to start with upcall entries are maintained in a linked list.
We may change it to RBT tree for performance improvement.
 *   store Upcall entries in inode/fd_ctxt for faster lookup.

Thanks,
Soumya

On 01/22/2015 02:31 PM, Soumya Koduri wrote:

Hi,

I have updated the feature page with more design details and the
dependencies/limitations this support has.

http://www.gluster.org/community/documentation/index.php/Features/Upcall-infrastructure#Dependencies



Kindly check the same and provide your inputs.

Few of them which may be addressed for 3.7 release are -

*AFR/EC*
 - Incase of replica bricks maintained by AFR, the upcalls state is
maintained and processed on all the replica bricks. This will result in
duplicate notifications sent by all those bricks incase of
non-idempotent fops.
 - Hence we need support on AFR to filter out such duplicate
callback notifications. Similar support is needed for EC as well.
 - One of the approaches suggested by the AFR team is to cache the
upcall notifications received for around 1min (their current lifetime)
to detect & filter out the duplicate notifications sent by the replica
bricks.


*Cleanup during network disconnect - protocol/server*
- At present, incase of network disconnects between the
glusterfs-server and the client, the protocol/server looks up the fd
table associated with that client and sends 'flush' op for each of those
fds to cleanup the locks associated with it.

- We need similar support to flush the lease locks taken. Hence,
while granting the lease-lock, we plan to associate that upcall_entry
with the corresponding fd_ctx or inode_ctx so that they can be easily
tracked if needed to be cleaned up. Also it will help in faster lookup
of the upcall entries while trying to process the fops using the same
fd/inode.

Note: Above cleanup is done for the upcall state associated with only
lease-locks. For the other entries maintained (for eg:, for
cache-invalidations), the reaper thread (which will be used to cleanup
the expired entries in this xlator) will clean-up those states as well
once they get expired.

*Replay of the lease-locks state*
   - At present, replay of locks by the client xlator (after network
disconnect and reconnect) seems to have been disabled.
   - But when it is being enabled, we need to add support to replay
lease-locks taken as well.
   - Till then, this will be considered as a limitation and will be
documented as suggested by KP.

Thanks,
Soumya


On 12/16/2014 09:36 AM, Krishnan Parthasarathi wrote:


- Is there a new connection from glusterfsd (upcall xlat

Re: [Gluster-devel] A new approach to solve Geo-replication ACTIVE/PASSIVE switching in distributed replicate setup!

[Kotresh Hiremath Ravishankar]

Hi All,

The logic discussed in previous mail thread is not feasible.  So in order to 
solve
the Active/Passive switching in geo-replication, following new idea is thought 
off.

1. Have a shared storage, a glusterfs management volume specific to 
geo-replication.

2. Use fcntl lock on a file stored on above said shared volume. There will be 
one file
   per replica set.

Each worker tries to lock the file on shared storage, who ever wins will be 
ACTIVE.
With this, we are able to solve the problem but there is an issue when the 
shared
storage goes down (if it is replica, when all replicas goes down). In that case,
the lock state is lost.

But if we use sanlock, as ovirt uses, I think the above problem of lock state 
being
lost could be solved ?
https://fedorahosted.org/sanlock/

If anybody have used sanlocks, is it a good option in this respect ?
Please share your thoughts, suggestions on this.


Thanks and Regards,
Kotresh H R

- Original Message -
> From: "Kotresh Hiremath Ravishankar" 
> To: gluster-devel@gluster.org
> Sent: Monday, December 22, 2014 10:53:34 AM
> Subject: [Gluster-devel] A new approach to solve Geo-replication 
> ACTIVE/PASSIVE switching in distributed replicate
> setup!
> 
> Hi All,
> 
> Current Desgin and its limitations:
> 
> Geo-replication syncs changes across geography using changelogs
> captured
>   by changelog translator. Changelog translator sits on server side just
>   above posix
>   translator. Hence, in distributed replicated setup, both replica pairs
>   collect
>   changelogs w.r.t their bricks. Geo-replication syncs the changes using only
>   one
>   brick among the replica pair at a time, calling it as "ACTIVE" and other
>   non syncing
>   brick as "PASSIVE".
>
>  Let's consider below example of distributed replicated setup where
>  NODE-1 as b1 and its replicated brick b1r is in NODE-2
> 
> NODE-1 NODE-2
>   b1b1r
> 
>   At the beginning, geo-replication chooses to sync changes from NODE-1:b1
>   and
>   NODE-2:b1r will be "PASSIVE". The logic depends on virtual getxattr
>   'trusted.glusterfs.node-uuid' which always returns first up subvolume i.e.,
>   NODE-1.
>   When NODE-1 goes down, the above xattr returns NODE-2 and that is made
>   'ACTIVE'.
>   But when NODE-1 comes back again, the above xattr returns NODE-1 and it is
>   made
>   'ACTIVE' again. So for a brief interval of time, if NODE-2 had not finished
>   processing
>the changelog, both NODE-2 and NODE-1 will be ACTIVE causing rename race
>as below.
>
>https://bugzilla.redhat.com/show_bug.cgi?id=1140183
> 
> 
> SOLUTION:
>Don't make NODE-2 'PASSIVE' when NODE-1 comes back again untill NODE-2
>goes down.
> 
> 
> APPROACH TO SOLVE WHICH I CAN THINK OF:
> 
> Have a distributed store of a file, which captures the bricks which are
> active.
> When a NODE goes down, the file is updated with it's replica bricks making
> sure, at any point in time, the file has all the bricks to be made active.
> Geo-replication worker process is made 'ACTIVE' only if it is in the file.
> 
>  Implementation can be in two ways:
> 
>   1. Have a distributed store for above implementation. This needs to be
>   thought
>  of as distributed store is not in place in glusterd yet.
> 
>   2. Other solution is to store in a file similar to existing glusterd global
>  configuration file (/var/lib/glusterd/options). When this file is
>  updated,
>  version number is incremented. When the node which is gone down, comes
>  up,
>  gets this file from peers if it's version number is less that of peers.
> 
> 
> I did a POC with second approach storing list of active bricks
> 'NodeUUID:brickpath'
> in options file itself. It seems to work fine except the bug in glusterd
> where the
> daemons are getting spawned before the node gets 'options' file from other
> node during
> handshake.
> 
> CHANGES IN GLUSTERD:
> When a node goes down, all the other nodes are notified through
> glusterd_peer_rpc_notify,
>   where, it needs to find the replicas of the node which went down and update
>   the global
>   file.
> 
> PROBLEMS/LIMITATIONS WITH THIS APPRAOCH:
> 1. If glusterd is killed and the node is still up, this makes the other
> replica 'ACTIVE'.
>So both replica bricks will be syncing at this point of time which is
>not expected.
> 
> 2. If the single brick process is killed, it's replica brick is not made
> 'ACTIVE'.
> 
> 
> Glusterd/AFR folks,
> 
> 1. Do you see a better approach other than above to solve this issue?
> 2. Is this approach feasible? If yes, how can I handle the problems
> mentioned above ?
> 3. Is this approach feasible from scalability point of view since
> complete list of active
>brick path is stored and read by gsyncd ?
> 3. Does this approach fits into three way rep

Re: [Gluster-devel] [Gluster-users] High CPU Usage - Glusterfsd

[Justin Clift]

On 22 Feb 2015, at 20:47, ML mail  wrote:
> Dear Ben,
> 
> Very interesting answer from yours of how to find out where the bottleneck 
> is. These commands and paramters (iostat, sar) should maybe be documented on 
> the Gluster wiki.

Good idea.  Any interest in creating a wiki account and getting it
started? :)

(yes, this is my transparent attempt to get more people doing stuff
on the wiki :>)

Regards and best wishes,

Justin Clift

--
GlusterFS - http://www.gluster.org

An open source, distributed file system scaling to several
petabytes, and handling thousands of clients.

My personal twitter: twitter.com/realjustinclift

___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] High CPU Usage - Glusterfsd

[ML mail]

Dear Ben,

Very interesting answer from yours of how to find out where the bottleneck is. 
These commands and paramters (iostat, sar) should maybe be documented on the 
Gluster wiki.

I have a question for you, in order to better use my CPU cores (6 cores per 
node) I was wondering if I should create two bricks per node even if these two 
bricks would be pointing to the same RAID array. What do you think? Would there 
be a performance gain in my case and is that recommended? I read that optimally 
the second brick, if on the same server, should be on another RAID or HBA 
controller.

Best regards
ML





On Sunday, February 22, 2015 7:40 PM, Ben England  wrote:
Renchu, 

I didn't see anything about average file size and read/write mix.  One example 
of how to observe both of these, as well as latency and throughput - on server 
run these commands:

# gluster volume profile your-volume start
# gluster volume profile your-volume info > /tmp/dontcare
# sleep 60
# gluster volume profile your-volume info > profile-for-last-minute.log

There is also a "gluster volume top" command that may be of use to you in 
understanding what your users are doing with Gluster.

Also you may want to run "top -H" and see whether any threads in either 
glusterfsd or smbd are at or near 100% CPU - if so, you really are hitting a 
CPU bottleneck.  Looking at process CPU utilization can be deceptive, since a 
process may include multiple threads.  "sar -n DEV 2" will show you network 
utilization, and "iostat -mdx /dev/sd? 2" on your server will show block device 
queue depth (latter two tools require sysstat rpm).  Together these can help 
you to understand what kind of bottleneck you are seeing.

I don't see how many "bricks" are in your Gluster volume but it sounds like you 
have only one glusterfsd/server.   If you have idle cores on your servers, you 
can harness more CPU power by using multiple bricks/server, which results in 
multiple glusterfsd processes on each server, allowing greater parallelism.
For example, you can do this by presenting individual disk drives as bricks 
rather than RAID volumes.

Let us know if these suggestions helped

-ben england

- Original Message -
> From: "Renchu Mathew" 
> To: gluster-us...@gluster.org
> Cc: gluster-devel@gluster.org
> Sent: Sunday, February 22, 2015 7:09:09 AM
> Subject: [Gluster-devel] High CPU Usage - Glusterfsd
> 
> 
> 
> Dear all,
> 
> 
> 
> I have implemented glusterfs storage on my company – 2 servers with
> replicate. But glustherfsd shows more than 100% CPU utilization most of the
> time. So it is so slow to access the gluster volume. My setup is two
> glusterfs servers with replication. The gluster volume (almost 10TB of data)
> is mounted on another server (glusterfs native client) and using samba share
> for the network users to access those files. Is there any way to reduce the
> processor usage on these servers? Please give a solution ASAP since the
> users are complaining about the poor performance. I am using glusterfs
> version 3.6.
> 
> 
> 
> Regards
> 
> 
> 
> Renchu Mathew | Sr. IT Administrator
> 
> 
> 
> 
> 
> 
> 
> CRACKNELL DUBAI | P.O. Box 66231 | United Arab Emirates | T +971 4 3445417 |
> F +971 4 3493675 | M +971 50 7386484
> 
> ABU DHABI | DUBAI | LONDON | MUSCAT | DOHA | JEDDAH
> 
> EMAIL ren...@cracknell.com | WEB www.cracknell.com
> 
> 
> 
> This email, its content and any files transmitted with it are intended solely
> for the addressee(s) and may be legally privileged and/or confidential. If
> you are not the intended recipient please let us know by email reply and
> delete it from the system. Please note that any views or opinions presented
> in this email do not necessarily represent those of the company. Email
> transmissions cannot be guaranteed to be secure or error-free as information
> could be intercepted, corrupted, lost, destroyed, arrive late or incomplete,
> or contain viruses. The company therefore does not accept liability for any
> errors or omissions in the contents of this message which arise as a result
> of email transmission.
> 
> 
> 
> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel

> 
___
Gluster-users mailing list
gluster-us...@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Object Quota feature proposal for GlusterFS-3.7

[Luis Pabon]

Quick question, 
  Is this feature still necessary since gluster-swift is now SwiftOnFile and is 
deployed as a storage policy of Swift which uses the Swift container database?

- Luis

- Original Message -
From: "Prashanth Pai" 
To: "Vijaikumar M" 
Cc: "gluster-devel@gluster.org >> Gluster Devel" 
Sent: Tuesday, February 17, 2015 9:47:20 PM
Subject: Re: [Gluster-devel] Object Quota feature proposal for GlusterFS-3.7

This might be helpful for  better integration with Swift. Swift maintains 
accounting metadata per account and per container. From glusterfs perspective, 
accounts are (represented as) first level directories in root and containers 
are second level directories. So accounts contain containers.

Each account has following metadata:
X-Account-Object-Count
X-Account-Bytes-Used
X-Account-Container-Count

Each container has following metadata:
X-Container-Object-Count
X-Container-Bytes-Used

I've got a few questions:
Will there be separate count stored for files vs directories ?
Does the count represent entire sub tree or just the immediate directory 
contents ?
Would a getxattr on a dir for count aggregate counts from all distributed 
bricks ?

Thanks.

Regards,
 -Prashanth Pai

- Original Message -
From: "Vijaikumar M" 
To: "gluster-devel@gluster.org >> Gluster Devel" 
Sent: Tuesday, February 17, 2015 5:22:34 PM
Subject: [Gluster-devel] Object Quota feature proposal for GlusterFS-3.7

Hi All, 

We are proposing files/objects quota feature for GlusterFS-3.7. 

Here is the feature page: 
http://www.gluster.org/community/documentation/index.php/Features/Object_Count 


'Object Quotas' is an enhancement to the existing 'File Usage Quotas' and has 
the following benefits: 


* Easy to query number of objects present in a volume. 
* Can serve as an accounting mechanism for quota enforcement based on 
number of Inodes. 
* This interface will be useful for integration with OpenStack Swift and 
Ceilometer. 

We can set the following Quota object limits, similar to file usage: 


* 1) Directory level - limit the number of files at the directory level 
* 2) Volume level - limit the number of files at the volume level 



Looking forward for Question/Feedback. 

Thanks, 
Vijay 


___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] High CPU Usage - Glusterfsd

[Ben England]

Renchu, 

I didn't see anything about average file size and read/write mix.  One example 
of how to observe both of these, as well as latency and throughput - on server 
run these commands:

# gluster volume profile your-volume start
# gluster volume profile your-volume info > /tmp/dontcare
# sleep 60
# gluster volume profile your-volume info > profile-for-last-minute.log

There is also a "gluster volume top" command that may be of use to you in 
understanding what your users are doing with Gluster.

Also you may want to run "top -H" and see whether any threads in either 
glusterfsd or smbd are at or near 100% CPU - if so, you really are hitting a 
CPU bottleneck.  Looking at process CPU utilization can be deceptive, since a 
process may include multiple threads.  "sar -n DEV 2" will show you network 
utilization, and "iostat -mdx /dev/sd? 2" on your server will show block device 
queue depth (latter two tools require sysstat rpm).  Together these can help 
you to understand what kind of bottleneck you are seeing.

I don't see how many "bricks" are in your Gluster volume but it sounds like you 
have only one glusterfsd/server.   If you have idle cores on your servers, you 
can harness more CPU power by using multiple bricks/server, which results in 
multiple glusterfsd processes on each server, allowing greater parallelism.
For example, you can do this by presenting individual disk drives as bricks 
rather than RAID volumes.

Let us know if these suggestions helped

-ben england

- Original Message -
> From: "Renchu Mathew" 
> To: gluster-us...@gluster.org
> Cc: gluster-devel@gluster.org
> Sent: Sunday, February 22, 2015 7:09:09 AM
> Subject: [Gluster-devel] High CPU Usage - Glusterfsd
> 
> 
> 
> Dear all,
> 
> 
> 
> I have implemented glusterfs storage on my company – 2 servers with
> replicate. But glustherfsd shows more than 100% CPU utilization most of the
> time. So it is so slow to access the gluster volume. My setup is two
> glusterfs servers with replication. The gluster volume (almost 10TB of data)
> is mounted on another server (glusterfs native client) and using samba share
> for the network users to access those files. Is there any way to reduce the
> processor usage on these servers? Please give a solution ASAP since the
> users are complaining about the poor performance. I am using glusterfs
> version 3.6.
> 
> 
> 
> Regards
> 
> 
> 
> Renchu Mathew | Sr. IT Administrator
> 
> 
> 
> 
> 
> 
> 
> CRACKNELL DUBAI | P.O. Box 66231 | United Arab Emirates | T +971 4 3445417 |
> F +971 4 3493675 | M +971 50 7386484
> 
> ABU DHABI | DUBAI | LONDON | MUSCAT | DOHA | JEDDAH
> 
> EMAIL ren...@cracknell.com | WEB www.cracknell.com
> 
> 
> 
> This email, its content and any files transmitted with it are intended solely
> for the addressee(s) and may be legally privileged and/or confidential. If
> you are not the intended recipient please let us know by email reply and
> delete it from the system. Please note that any views or opinions presented
> in this email do not necessarily represent those of the company. Email
> transmissions cannot be guaranteed to be secure or error-free as information
> could be intercepted, corrupted, lost, destroyed, arrive late or incomplete,
> or contain viruses. The company therefore does not accept liability for any
> errors or omissions in the contents of this message which arise as a result
> of email transmission.
> 
> 
> 
> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel
> 
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] OOM issue in openstack Cinder - GlusterFS CI env

[Niels de Vos]

Hi Deepak and others,

On Sat, Feb 21, 2015 at 10:00:18PM +0530, Deepak Shetty wrote:
> Hi All,
>   I am looking for some help from glusterfs side for the Out of Memory
> (OOM) issue
> we are seeing when using GlusterFS as a storage backend for openstack
> Cinder (block storage service)

This seems to be using the FUSE mount points for the VM images, right?

> openstack has a upstream CI env managed by openstack infra team, where
> we added a new job that creates a devstack env (openstack all in one for
> newbies) and configures the block service (Cinder) with GlusterFS as
> storage backend. Once setup, the CI job runs openstack tempest (Integration
> test suite of openstack) that does API level testing of the whole openstack
> env.
> 
> As part of that testing, ~1.5 to 2 hours into the run, the tempest job
> (VM) hits OOM and the kernel oom-killer kills the process with the max
> memory to reduce memory pressure.
> 
> The tempest job is based on CentOS 7 and uses glusterfs 3.6.2 as the
> storage backend for openstack Cinder
> 
> The openstack-dev thread @
> *http://thread.gmane.org/gmane.comp.cloud.openstack.devel/46861
> * has
> details including links to the logs captured from the tempest jobs.

http://fungi.yuggoth.org/tmp/logs2.tar seems to be the most useful.

> Per the openstack infra folks, they have other jobs based off CentOS7 that
> doesn't hit this issue, the only change we are adding is configuring cinder
> with glusterfs when this happens, so right now glusterfs is in the
> spotlight for causing this.

Well, there definitely is something wrong on the Gluster side. After the
'blackout' between 17:47 - 18:17, this is logged in
logs/screen-c-vol.txt.gz:

  mount -t glusterfs 127.0.0.1:/vol1 
/opt/stack/data/cinder/mnt/f4c6ad7e3bba4ad1195b3b538efab64a" returned: 32

32 as exit code from "mount" means "mount failure".

The mountpoint would log details in
logs/glusterfs/opt-stack-data-cinder-mnt-f4c6ad7e3bba4ad1195b3b538efab64a.log.
At 18:00 in that file, there is this message:

  0-vol1-client-0: server 104.130.30.179:49152 has not responded in the last 42 
seconds, disconnecting.

Its a one-brick volume, so this error is pretty fatal. This error does
not cause an unmount, but the mountpoint will return errors. Subsequent
mount tries will fail, because the mountpoint is not unmounted yet.
Tests to see if the target directory can be used to mount a volume, will
fail, causing the mount process to return error 32.

A little later, the (still) existing fuse mount client manages to
connect to the brick again:

  0-vol1-client-0: changing port to 49152 (from 0)

The log of the brick does not contain anything useful for this time :-/

> I am looking thru the logs trying to co-relate syslog, dstat, tempest info
> to figure the state of the system and what was happening at and before the
> OOM to get any clues, but wanted to start this thread in gluster-devel to
> see if others can pitch in with their ideas to accelerate the debug and
> help root cause.

I have not yet found any pointer to what could have taken place between
17:47-18:17. It seems that network I/O stopped (ping timeout in FUSE).

One of the few logs that I came across which contains some messages
during this 'blackout', is logs/screen-n-cond.txt.gz. But I assume these
messages are triggered by the underlying network issues.


You mention that there was a OOM? Which logs contain some details about
that? Also, could you explain what test (workload) was running in the
VM(s?) on the FUSE mountpoint?

In logs/screen-dstat.txt.gz there seems to be a process called
"cinder-backup" that is running around that time. From the details of
dstat, I guess that this process causes the reading of files from the
Gluster Volume. When the system becomes responsive again, dstat has 11G
as "dsk/total read", it jumped from ~400M in 30 minutes. If this amount
of data is not flushed but kept in memory, I don't doubt that an OOM
could be triggered.

Do you know where this backup is written to? If we can setup a
reproducer environment that emulates the workload, things would become
much easier for debugging :)

One of the 1st things I would try, is to configure the kernel to flush
dirty memory more aggressively ('sysctl' values for 'vm.*' options). An
other thing that may help to identify the memory usage, is capturing the
output of 'slabtop --once' in a 1-minute interval.

Possibly you could run the same test on a higher spec'd (more RAM?)
machine, and capture more details. Even if there is no OOM, the memory
usage pattern should look similar and might aid with the investigations.

HTH,
Niels

> Also pasting relevant part of the chat log I had with infra folks ...
> 
> Feb 20 21:46:28 deepakcs: you are at 70% wait time at the
> end of that
> 
> Feb 20 21:46:37 so your io system is just gone bonkers
> 
> Feb 20 21:47:14  sdague: that would explain why the console login
> 

[Gluster-devel] Sharding - Inode write fops - recoverability from failures - design

[Krutika Dhananjay]

Hi, 

Please find the design doc for one of the problems in sharding which Pranith 
and I are trying to solve and its solution @ 
http://review.gluster.org/#/c/9723/1 . 
Reviews and feedback are much appreciated. 

Thanks, 
Krutika 
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

[Gluster-devel] High CPU Usage - Glusterfsd

[Renchu Mathew]

Dear all,

I have implemented glusterfs storage on my company - 2 servers with replicate. 
But glustherfsd shows more than 100% CPU utilization most of the time. So it is 
so slow to access the gluster volume. My setup is two glusterfs servers with 
replication. The gluster volume (almost 10TB of data) is mounted on another 
server (glusterfs native client) and using samba share for the network users to 
access those files. Is there any way to reduce the processor usage on these 
servers? Please give a solution ASAP since the users are complaining about the 
poor performance. I am using glusterfs version 3.6.

Regards

Renchu Mathew  |  Sr. IT Administrator

[Description: Description: cid:image002.jpg@01CF53EA.2A4680F0]

CRACKNELL  DUBAI   |  P.O. Box 66231  |   United Arab Emirates  |  T +971 4 
3445417  |  F +971 4 3493675 |  M +971 50 7386484
ABU DHABI | DUBAI | LONDON | MUSCAT | DOHA | JEDDAH
EMAIL ren...@cracknell.com | WEB www.cracknell.com

This email, its content and any files transmitted with it are intended solely 
for the addressee(s) and may be legally privileged and/or confidential. If you 
are not the intended recipient please let us know by email reply and delete it 
from the system. Please note that any views or opinions presented in this email 
do not necessarily represent those of the company. Email transmissions cannot 
be guaranteed to be secure or error-free as information could be intercepted, 
corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The 
company therefore does not accept liability for any errors or omissions in the 
contents of this message which arise as a result of email transmission.

___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel