[Gluster-users] Gluster News - September

[Vijay Bellur]

Gluster news for September is now available at:

http://blog.gluster.org/2015/10/gluster-news-september-2015/

We will be moving to bi-weekly news updates from October and intend 
using the new template discussed in these lists for further updates.


Regards,
Vijay
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] [Gluster-devel] Heketi Release 1 Available for use

[Atin Mukherjee]

+gluster-users

-Atin
Sent from one plus one
On Oct 3, 2015 7:32 AM, "Luis Pabon"  wrote:

> Hi all,
>   Release 1 of Heketi is now available for use.
>
>   Heketi is a service used to manage the lifecycle of GlusterFS volumes.
> With Heketi, cloud services like OpenStack Manila, Kubernetes, and
> OpenShift can dynamically provision GlusterFS volumes with any of the
> supported durability types.  Heketi will automatically determine the
> location for bricks across the cluster, making sure to place bricks and its
> replicas across different failure domains.  Heketi also supports any number
> of GlusterFS clusters, allowing cloud services to provide network file
> storage without being limited to a single GlusterFS cluster.
>
> Website:  https://github.com/heketi/heketi
> Documentation:  https://github.com/heketi/heketi/wiki
> Demo:  https://github.com/heketi/heketi/wiki/Demo
> Download:  https://github.com/heketi/heketi/releases
>
> Questions can be asked over email, using Github issues, or using Gitter (
> https://gitter.im/heketi/heketi).
>
> Please try it out and let us know if you find any issues or have any
> comments.
>
> If you find a bug, checkout
> https://github.com/heketi/heketi/wiki/Filing-a-bug.
>
> Thank you!
>
> - Luis
>
> ___
> Gluster-devel mailing list
> gluster-de...@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] [IMPORTANT, PLEASE READ] replace-brick problem with all releases till now

[Steve Dainard]

On Thu, Oct 1, 2015 at 2:24 AM, Pranith Kumar Karampuri
 wrote:
> hi,
>  In releases till now from day-1 with replication, there is a corner
> case bug which can wipe out the all the bricks in that replica set  when the
> disk/brick(s) are  replaced.
>
> Here are the steps that could lead to that situation:
> 0) Clients are operating on the volume and are actively pumping data.
> 1) Execute replace-brick command (OR) take down the brick that needs the
> disk replaced or re-formatted. and bring the brick back up.

So the better course of action would be to remove-brick  replica
 start, replace the disk, and then add-brick  replica 
? Perhaps it would be wise to un-peer the host before adding the brick
back?

Is there any chance that adding a 3rd replica to a 2 replica cluster
with active client writes could cause the same issue?

On 3.7.3 I recently lost 2 of 3 bricks all the way down to the XFS
filesystem being corrupted, but I blamed that on the disk controller
which was doing a raid0 pass-through on 2 hosts, but not the new 3rd
host. This occurred after some time though, and client writes were
being blocked while the 3rd brick was being added.

> 2) Client creates a file/directory just on the root of the brick which
> succeeds on the new brick but fails on the bricks that have been online
> (May be because the file existed on the bricks that are good copies)
> 3) Now when self-heal is triggered from the Client/self-heal-daemon it
> thinks the just replaced brick is the correct directory and deletes the
> files/directories from the bricks that have the actual data.
>
> I have been working on afr for almost 4 years now and never saw any user
> complain about this problem. We were working on a document for an  official
> way to replace brick/disk but it never occurred to us that this  could
> happen until recently. I am going to get a proper document by end of this
> week on replacing the bricks/disks in a safe way. And will keep you  guys
> posted about fixes to prevent this from happening entirely.
>
> Pranith
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] samba-vfs-glusterfs share problems

[Steve Dainard]

Hi Diego,

Awesome, works - much appreciated.

As far as I can search this isn't listed anywhere on the gluster.org
docs, but there is a link to a blog hosted here:
https://lalatendumohanty.wordpress.com/2014/02/11/using-glusterfs-with-samba-and-samba-vfs-plugin-for-glusterfs-on-fedora-20/

And it is also documented here:
https://www.mankier.com/8/vfs_glusterfs for future searches.

On Thu, Oct 1, 2015 at 2:53 PM, Diego Remolina  wrote:
> On all your shares where you use vfs objects = glusterfs also add the option:
>
> kernel share modes = No
>
> Then restart samba.
>
> Here is one of my example shares:
>
> [Projects]
>path = /projects
>browseable = yes
>write list = @Staff,root,@Admin,@Managers
>writeable = yes
>guest ok = no
>create mask = 660
>directory mask = 770
>kernel share modes = No
>vfs objects = glusterfs
>glusterfs:loglevel = 7
>glusterfs:logfile = /var/log/samba/glusterfs-projects.log
>glusterfs:volume = export
>
> HTH,
>
> Diego
>
> On Thu, Oct 1, 2015 at 4:15 PM, Steve Dainard  wrote:
>> samba-vfs-glusterfs-4.1.12-23.el7_1.x86_64
>> gluster 3.6.6
>>
>> I've shared a gluster volume using samba vfs with the options:
>> vfs objects = glusterfs
>> glusterfs:volume = test
>> path = /
>>
>> I can do the following:
>> (Windows client):
>> -Create new directory
>> -Create new file -- an error pops up "Unable to create the file 'New
>> Test Document.txt' The system cannot find the file specified.' BUT the
>> file is created anyways and shows up in the directory immediately
>> -Create a new sub directory in the above directory
>> -Delete or rename any file
>> (Linux client)
>> -Create new directory
>> -Create a new file under sub-directory -- "Device or resource busy"
>> BUT the file is created anyways - must refresh to see file in GUI
>> -Delete or rename any file
>>
>> Can not do:
>> (Windows client)
>> -Edit the new txt file in notepad -- file opens but popup says 'The
>> process cannot access the file because it is being used by another
>> process.'
>> -Create a new file in the above new directory -- "Unable to create the
>> file 'New Text Document.txt' The system cannot find the file
>> specified", refreshing the directory doesn't list the file, but if
>> samba is restarted the client can see the file.
>> (Linux client)
>> -Edit the new file -- "Unexpected error: Device or resource busy"
>>
>>
>> I've also shared the same volume via a fuse mount, and have no issues
>> performing any file operations.
>>
>> smb.conf shares:
>>
>> [gluster-test-fuse]
>> comment = exporting gluster filesystem via fuse mount
>> path = /mnt/gluster/test
>> read only = no
>> writable = yes
>> write list = +users
>> browseable = yes
>> guest ok = yes
>> create mask = 0660
>> ;directory mask = 0770
>> force directory mode = 0770
>> force group = users
>>
>> [gluster-test]
>> comment = exporting gluster filesystem via gfapi
>> vfs objects = glusterfs
>> glusterfs:volume = test
>> glusterfs:logfile = /var/log/samba/glusterfs-test.%M.log
>> glusterfs:loglevel = 7
>> path = /
>> writable = yes
>> write list = +users
>> browseable = yes
>> guest ok = yes
>> create mask = 0660
>> ;directory mask = 0770
>> force directory mode = 0770
>> force group = users
>>
>> Other things of note:
>>
>> I have a default ACL on the gluster volume, as well as setgid.
>>
>> Anyone else get samba vfs to work properly? I've opened a bugzilla
>> here: https://bugzilla.redhat.com/show_bug.cgi?id=1268092
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http://www.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] 3.7 client with 3.5 server

[Sarunas Burdulis]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Hello,

Is it possible to have version 3.5 volume (Ubuntu 15.04 server) mounted
with 3.7 client (Ubuntu 15.10)? Any mount options that would enable
this? Currently, with only 'defaults' option in /etc/fstab, mounting fails:

[2015-10-02 18:06:17.658297] W [socket.c:642:__socket_rwv] 0-glusterfs:
readv on 129.170.28.78:24007 failed (No data available)

[2015-10-02 18:06:17.659013] E [rpc-clnt.c:362:saved_frames_unwind] (-->
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x19a)[0x7f88fd43494a]
(-->
/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1cf)[0x7f88f
d20025f] (-->
/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f88fd20037e]
(-->
/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x9c)[0x7f88fd201b6c]
(--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(r
pc_clnt_notify+0x48)[0x7f88fd202328] ) 0-glusterfs: forced unwinding
frame type(GlusterFS Handshake) op(GETSPEC(2)) called at

2015-10-02 18:06:17.658034 (xid=0x1)
[2015-10-02 18:06:17.659087] E [glusterfsd-mgmt.c:1604:mgmt_getspec_cbk]
0-mgmt: failed to fetch volume file (key:/mathcluster)

Thanks!

- -- 
Sarunas Burdulis
Systems Administrator
Department of Mathematics, Dartmouth College
http://math.dartmouth.edu/~sarunas
-BEGIN PGP SIGNATURE-
Version: GnuPG v2

iQEcBAEBCAAGBQJWDspmAAoJEAcPjabqyGA9wPsH/jmD+9oNUGpluW6gGIzw3vY/
nHLz0RXHNh+46rgoLTBkMVeE+OZ3WS9VNYJqMw2NuzsArwtAYwxbbLHjgBLPdVf5
mpw1EokbgmczEKREljH7qiJHuPENJ+wA/JjUDUxvCeN6PhHqDbdYsraSfB/Q9eFj
ywgNV5r6gmVLJB3p7wCNSh/ByzNV52cAHLx9Mg3RVEj7RZ9U8v9S1aEVzQO9LtJ1
Te4546aI3yEmQYJJ1iM3OvQCvpiAjAzHlhDDaHyiOZYkzaYHP4GMxFlViDTKdJ7Y
BWVk8tWWoeIhIeakNaXIzYk+CmOsE30FPx2KZpZMJmPL+RRS1sk6UPXHTI0JD04=
=x5vd
-END PGP SIGNATURE-
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Tuning for small files

[Thibault Godouet]

Right, so what I did is:
- on one node (gluster 3.7.3), run 'gluster volume shared profile start'
- on the client mount, run the test
- on the node, run 'gluster volume shared profile info' (and copied the
output)
- finally, ran 'gluster volume profile shared stop'

I repeated this for two different tests (simple rm followed by svn
checkout, and a more complete build test), on an NFS mount and on a Fuse
mount.

To my surprise the svn checkout is actually a lot faster (3x) on the Fuse
mount than NFS.
However the build test is a lot slower on the Fuse mount (+50%, which is a
lot considering the compilation is CPU intensive, not just I/Os!).

Ben I will send you the profile outputs separately now...
On 29 Sep 2015 9:40 pm, "Ben Turner"  wrote:

> - Original Message -
> > From: "Thibault Godouet" 
> > To: "Ben Turner" 
> > Cc: hm...@t-hamel.fr, gluster-users@gluster.org
> > Sent: Tuesday, September 29, 2015 1:36:20 PM
> > Subject: Re: [Gluster-users] Tuning for small files
> >
> > Ben,
> >
> > I suspect meta-data / 'ls -l' performance is very important for my svn
> > use-case.
> >
> > Having said that, what do you mean by small file performance? I thought
> > what people meant by this was really the overhead of meta-data, with a
> 'ls
> > -l' being a sort of extreme case (pure meta-data).
> > Obviously if you also have to read and write actual data (albeit not much
> > at all per file), then the effect of meta-data overhead would get diluted
> > to a degree, bit potentially still very present.
>
> Where you run into problems with smallfiles on gluster is latency of
> sending data over the wire.  For every smallfile create there are a bunch
> of different file opetations we have to do on every file.  For example we
> will have to do at least 1 lookup per brick to make sure that the file
> doesn't exist anywhere before we create it.  We actually got it down to 1
> per brick with lookup optimize on, its 2 IIRC(maybe more?) with it
> disabled.  So the time we spend waiting for those lookups to complete adds
> to latency which lowers the number of files that can be created in a given
> period of time.  Lookup optimize was implemented in 3.7 and like I said its
> now at the optimal 1 lookup per brick on creates.
>
> The other problem with small files that we had in 3.6 is that we were
> using a single threaded event listener(epoll is what we call it).  This
> single thread would spike a CPU to 100%(called a hot thread) and glusterfs
> would become CPU bound.  The solution here was to make the event listener
> multi threaded so that we could spread the epoll load across CPUs there by
> eliminating the CPU bottleneck and allowing us to process more events in a
> given time.  FYI epoll is defaulted to 2 threads in 3.7, but I have seen
> cases where I still bottlenecked on CPU without 4 threads in my envs, so I
> usually do 4.  This was implemented in upstream 3.7 but was backported to
> RHGS 3.0.4 if you have a RH based version.
>
> Fixing these two issues lead to the performance gains I was talking about
> with smallfile creates.  You are probably thinking from a distributed FS +
> metadata server perspective(MDS) where the bottleneck is the MDS for
> smallfiles.  Since gluster doesn't have an MDS that load is transferred to
> the clients / servers and this lead to a CPU bottleneck when epoll was
> single threaded.  I think this is the piece you may have been missing.
>
> >
> > Would there be an easy way to tell how much time is spent on meta-data
> vs.
> > Data in a profile output?
>
> Yep!  Can you gather some profiling info and send it to me?
>
> >
> > One thing I wonder: do your comments apply to both native Fuse and NFS
> > mounts?
> >
> > Finally, all this brings me back to my initial question really: are there
> > any tuning recommendation of configuration tuning for my requirement
> (small
> > file read/writes on a pair of nodes with replication) beyond the thread
> > counts and lookup optimize?
> > Or are those by far the most important in this scenario?
>
> For creating a bunch of small files those are the only two that I know of
> that will have a large impact, maybe some others from the list can give
> some input on anything else we can do here.
>
> -b
>
> >
> > Thx,
> > Thibault.
> > - Original Message -
> > > From: hm...@t-hamel.fr
> > > To: aba...@magix.net
> > > Cc: gluster-users@gluster.org
> > > Sent: Monday, September 28, 2015 7:40:52 AM
> > > Subject: Re: [Gluster-users] Tuning for small files
> > >
> > > I'm also quite interested by small files performances optimization, but
> > > I'm a bit confused about the best option between 3.6/3.7.
> > >
> > > Ben Turner was saying that 3.6 might give the best performances:
> > >
> http://www.gluster.org/pipermail/gluster-users/2015-September/023733.html
> > >
> > > What kind of gain is expected (with consistent-metadata) if this
> > > regression is solved?
> >
> > Just to be clear, the issue I am talking about is metadata only(think ls
> -l

Re: [Gluster-users] glusterd crashing

[Gaurav Garg]

>> Pulling those logs now but how do I generate the core file you are asking
for?

When there is crash then core file automatically generated based on your 
*ulimit* set option. you can find location of core file in your root or current 
working directory or where ever you have set your core dump file location. core 
file gives you information regarding crash, where exactly crash happened.
you can find appropriate core file by looking at crash time in glusterd log's 
by searching "crash" keyword. you can also paste few line's just above latest 
"crash" keyword in glusterd logs.

Just for your curiosity if you willing to look where it crash then you can 
debug it by #gdb -c  glusterd

Thank you...

Regards,
Gaurav  

- Original Message -
From: "Gene Liverman" 
To: "Gaurav Garg" 
Cc: "gluster-users" 
Sent: Friday, October 2, 2015 8:28:49 PM
Subject: Re: [Gluster-users] glusterd crashing

Pulling those logs now but how do I generate the core file you are asking
for?





--
*Gene Liverman*
Systems Integration Architect
Information Technology Services
University of West Georgia
glive...@westga.edu
678.839.5492

ITS: Making Technology Work for You!




On Fri, Oct 2, 2015 at 2:25 AM, Gaurav Garg  wrote:

> Hi Gene,
>
> you have paste glustershd log. we asked you to paste glusterd log.
> glusterd and glustershd both are different process. with this information
> we can't find out why your glusterd crashed. could you paste *glusterd*
> logs (/var/log/glusterfs/usr-local-etc-glusterfs-glusterd.vol.log*) in
> pastebin (not in this mail thread) and give the link of pastebin in this
> mail thread. Can you also attach core file or you can paste backtrace of
> that core dump file.
> It will be great if you give us sos report of the node where the crash
> happen.
>
> Thanx,
>
> ~Gaurav
>
> - Original Message -
> From: "Gene Liverman" 
> To: "gluster-users" 
> Sent: Friday, October 2, 2015 4:47:00 AM
> Subject: Re: [Gluster-users] glusterd crashing
>
> Sorry for the delay. Here is what's installed:
> # rpm -qa | grep gluster
> glusterfs-geo-replication-3.7.4-2.el6.x86_64
> glusterfs-client-xlators-3.7.4-2.el6.x86_64
> glusterfs-3.7.4-2.el6.x86_64
> glusterfs-libs-3.7.4-2.el6.x86_64
> glusterfs-api-3.7.4-2.el6.x86_64
> glusterfs-fuse-3.7.4-2.el6.x86_64
> glusterfs-server-3.7.4-2.el6.x86_64
> glusterfs-cli-3.7.4-2.el6.x86_64
>
> The cmd_history.log file is attached.
> In gluster.log I have filtered out a bunch of lines like the one below due
> to make them more readable. I had a node down for multiple days due to
> maintenance and another one went down due to a hardware failure during that
> time too.
> [2015-10-01 00:16:09.643631] W [MSGID: 114031]
> [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-gv0-client-0: remote
> operation failed. Path: 
> (31f17f8c-6c96-4440-88c0-f813b3c8d364) [No such file or directory]
>
> I also filtered out a boat load of self heal lines like these two:
> [2015-10-01 15:14:14.851015] I [MSGID: 108026]
> [afr-self-heal-metadata.c:56:__afr_selfheal_metadata_do] 0-gv0-replicate-0:
> performing metadata selfheal on f78a47db-a359-430d-a655-1d217eb848c3
> [2015-10-01 15:14:14.856392] I [MSGID: 108026]
> [afr-self-heal-common.c:651:afr_log_selfheal] 0-gv0-replicate-0: Completed
> metadata selfheal on f78a47db-a359-430d-a655-1d217eb848c3. source=0 sinks=1
>
>
> [root@eapps-gluster01 glusterfs]# cat glustershd.log |grep -v 'remote
> operation failed' |grep -v 'self-heal'
> [2015-09-27 08:46:56.893125] E [rpc-clnt.c:201:call_bail] 0-glusterfs:
> bailing out frame type(GlusterFS Handshake) op(GETSPEC(2)) xid = 0x6 sent =
> 2015-09-27 08:16:51.742731. timeout = 1800 for 127.0.0.1:24007
> [2015-09-28 12:54:17.524924] W [socket.c:588:__socket_rwv] 0-glusterfs:
> readv on 127.0.0.1:24007 failed (Connection reset by peer)
> [2015-09-28 12:54:27.844374] I [glusterfsd-mgmt.c:1512:mgmt_getspec_cbk]
> 0-glusterfs: No change in volfile, continuing
> [2015-09-28 12:57:03.485027] W [socket.c:588:__socket_rwv] 0-gv0-client-2:
> readv on 160.10.31.227:24007 failed (Connection reset by peer)
> [2015-09-28 12:57:05.872973] E [socket.c:2278:socket_connect_finish]
> 0-gv0-client-2: connection to 160.10.31.227:24007 failed (Connection
> refused)
> [2015-09-28 12:57:38.490578] W [socket.c:588:__socket_rwv] 0-glusterfs:
> readv on 127.0.0.1:24007 failed (No data available)
> [2015-09-28 12:57:49.054475] I [glusterfsd-mgmt.c:1512:mgmt_getspec_cbk]
> 0-glusterfs: No change in volfile, continuing
> [2015-09-28 13:01:12.062960] W [glusterfsd.c:1219:cleanup_and_exit]
> (-->/lib64/libpthread.so.0() [0x3c65e07a51]
> -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x405e4d]
> -->/usr/sbin/glusterfs(cleanup_and_exit+0x65) [0x4059b5] ) 0-: received
> signum (15), shutting down
> [2015-09-28 13:01:12.981945] I [MSGID: 100030] [glusterfsd.c:2301:main]
> 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7.4
> (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p
> /var/lib/gluster

Re: [Gluster-users] glusterd crashing

[Gene Liverman]

Pulling those logs now but how do I generate the core file you are asking
for?





--
*Gene Liverman*
Systems Integration Architect
Information Technology Services
University of West Georgia
glive...@westga.edu
678.839.5492

ITS: Making Technology Work for You!




On Fri, Oct 2, 2015 at 2:25 AM, Gaurav Garg  wrote:

> Hi Gene,
>
> you have paste glustershd log. we asked you to paste glusterd log.
> glusterd and glustershd both are different process. with this information
> we can't find out why your glusterd crashed. could you paste *glusterd*
> logs (/var/log/glusterfs/usr-local-etc-glusterfs-glusterd.vol.log*) in
> pastebin (not in this mail thread) and give the link of pastebin in this
> mail thread. Can you also attach core file or you can paste backtrace of
> that core dump file.
> It will be great if you give us sos report of the node where the crash
> happen.
>
> Thanx,
>
> ~Gaurav
>
> - Original Message -
> From: "Gene Liverman" 
> To: "gluster-users" 
> Sent: Friday, October 2, 2015 4:47:00 AM
> Subject: Re: [Gluster-users] glusterd crashing
>
> Sorry for the delay. Here is what's installed:
> # rpm -qa | grep gluster
> glusterfs-geo-replication-3.7.4-2.el6.x86_64
> glusterfs-client-xlators-3.7.4-2.el6.x86_64
> glusterfs-3.7.4-2.el6.x86_64
> glusterfs-libs-3.7.4-2.el6.x86_64
> glusterfs-api-3.7.4-2.el6.x86_64
> glusterfs-fuse-3.7.4-2.el6.x86_64
> glusterfs-server-3.7.4-2.el6.x86_64
> glusterfs-cli-3.7.4-2.el6.x86_64
>
> The cmd_history.log file is attached.
> In gluster.log I have filtered out a bunch of lines like the one below due
> to make them more readable. I had a node down for multiple days due to
> maintenance and another one went down due to a hardware failure during that
> time too.
> [2015-10-01 00:16:09.643631] W [MSGID: 114031]
> [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-gv0-client-0: remote
> operation failed. Path: 
> (31f17f8c-6c96-4440-88c0-f813b3c8d364) [No such file or directory]
>
> I also filtered out a boat load of self heal lines like these two:
> [2015-10-01 15:14:14.851015] I [MSGID: 108026]
> [afr-self-heal-metadata.c:56:__afr_selfheal_metadata_do] 0-gv0-replicate-0:
> performing metadata selfheal on f78a47db-a359-430d-a655-1d217eb848c3
> [2015-10-01 15:14:14.856392] I [MSGID: 108026]
> [afr-self-heal-common.c:651:afr_log_selfheal] 0-gv0-replicate-0: Completed
> metadata selfheal on f78a47db-a359-430d-a655-1d217eb848c3. source=0 sinks=1
>
>
> [root@eapps-gluster01 glusterfs]# cat glustershd.log |grep -v 'remote
> operation failed' |grep -v 'self-heal'
> [2015-09-27 08:46:56.893125] E [rpc-clnt.c:201:call_bail] 0-glusterfs:
> bailing out frame type(GlusterFS Handshake) op(GETSPEC(2)) xid = 0x6 sent =
> 2015-09-27 08:16:51.742731. timeout = 1800 for 127.0.0.1:24007
> [2015-09-28 12:54:17.524924] W [socket.c:588:__socket_rwv] 0-glusterfs:
> readv on 127.0.0.1:24007 failed (Connection reset by peer)
> [2015-09-28 12:54:27.844374] I [glusterfsd-mgmt.c:1512:mgmt_getspec_cbk]
> 0-glusterfs: No change in volfile, continuing
> [2015-09-28 12:57:03.485027] W [socket.c:588:__socket_rwv] 0-gv0-client-2:
> readv on 160.10.31.227:24007 failed (Connection reset by peer)
> [2015-09-28 12:57:05.872973] E [socket.c:2278:socket_connect_finish]
> 0-gv0-client-2: connection to 160.10.31.227:24007 failed (Connection
> refused)
> [2015-09-28 12:57:38.490578] W [socket.c:588:__socket_rwv] 0-glusterfs:
> readv on 127.0.0.1:24007 failed (No data available)
> [2015-09-28 12:57:49.054475] I [glusterfsd-mgmt.c:1512:mgmt_getspec_cbk]
> 0-glusterfs: No change in volfile, continuing
> [2015-09-28 13:01:12.062960] W [glusterfsd.c:1219:cleanup_and_exit]
> (-->/lib64/libpthread.so.0() [0x3c65e07a51]
> -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x405e4d]
> -->/usr/sbin/glusterfs(cleanup_and_exit+0x65) [0x4059b5] ) 0-: received
> signum (15), shutting down
> [2015-09-28 13:01:12.981945] I [MSGID: 100030] [glusterfsd.c:2301:main]
> 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7.4
> (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p
> /var/lib/glusterd/glustershd/run/glustershd.pid -l
> /var/log/glusterfs/glustershd.log -S
> /var/run/gluster/9a9819e90404187e84e67b01614bbe10.socket --xlator-option
> *replicate*.node-uuid=416d712a-06fc-4b3c-a92f-8c82145626ff)
> [2015-09-28 13:01:13.009171] I [MSGID: 101190]
> [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread
> with index 1
> [2015-09-28 13:01:13.092483] I [graph.c:269:gf_add_cmdline_options]
> 0-gv0-replicate-0: adding option 'node-uuid' for volume 'gv0-replicate-0'
> with value '416d712a-06fc-4b3c-a92f-8c82145626ff'
> [2015-09-28 13:01:13.100856] I [MSGID: 101190]
> [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread
> with index 2
> [2015-09-28 13:01:13.103995] I [MSGID: 114020] [client.c:2118:notify]
> 0-gv0-client-0: parent translators are ready, attempting connect on
> transport
> [2015-09-28 13:01:13.114745] I [MSGID: 114020] [client.c:2118:noti

[Gluster-users] Question regarding tiering

[Ameet Pyati]

Hi,

I am trying to attach a cache tier to normal distributed volume. I am
seeing write failures when the cache brick becomes full. following are the
steps


*>> 1. create volume using hdd brick*

*root@host:~/gluster/glusterfs# gluster volume create vol
host:/data/brick1/hdd/*
*volume create: vol: success: please start the volume to access data*
*root@host:~/gluster/glusterfs# gluster volume start vol*
*volume start: vol: success*

*>> 2. mount and write one file of size 1G*

*root@host:~/gluster/glusterfs# mount -t glusterfs host:/vol /mnt*
*root@host:~/gluster/glusterfs# dd if=/dev/zero of=/mnt/file1 bs=1G count=1*
*1+0 records in*
*1+0 records out*
*1073741824 bytes (1.1 GB) copied, 1.50069 s, 715 MB/s*

*root@host:~/gluster/glusterfs# du -sh /data/brick**
*1.1G/data/brick1*
*60K /data/brick2*


*>> 3. attach ssd brick as tier*

*root@host:~/gluster/glusterfs# gluster volume attach-tier vol
host:/data/brick2/ssd/*
*Attach tier is recommended only for testing purposes in this release. Do
you want to continue? (y/n) y*
*volume attach-tier: success*
*volume rebalance: vol: success: Rebalance on vol has been started
successfully. Use rebalance status command to check status of the rebalance
process.*
*ID: dea8d1b7-f0f4-4c17-94f5-ba0e263bc561*

*root@host:~/gluster/glusterfs# gluster volume rebalance vol tier status*
*Node Promoted files   Demoted filesStatus*
*----*
*localhost00in progress*
*volume rebalance: vol: success*


*>> 4. write data to fill up cache tier*

*root@host:~/gluster/glusterfs# dd if=/dev/zero of=/mnt/file2 bs=1G count=9
oflag=direct*
*9+0 records in*
*9+0 records out*
*9663676416 bytes (9.7 GB) copied, 36.793 s, 263 MB/s*
*root@host:~/gluster/glusterfs# du -sh /data/brick**
*1.1G/data/brick1*
*9.1G/data/brick2*
*root@host:~/gluster/glusterfs# gluster volume rebalance vol tier status*
*Node Promoted files   Demoted filesStatus*
*----*
*localhost00in progress*
*volume rebalance: vol: success*
*root@host:~/gluster/glusterfs# gluster volume rebalance vol  status*
*Node Rebalanced-files  size
scanned  failures   skipped   status   run time in
secs*
*   -  ---   ---
---   ---   --- 
--*
*   localhost00Bytes
  0 0 0  in progress
112.00*
*volume rebalance: vol: success*

*root@host:~/gluster/glusterfs# dd if=/dev/zero of=/mnt/file3 bs=1G count=5
oflag=direct*
*dd: error writing â/mnt/file3â: No space left on device*
*dd: closing output file â/mnt/file3â: No space left on device*

*root@host:~/gluster/glusterfs# du -sh /data/brick**
*1.1G/data/brick1*
*9.3G/data/brick2*

* there is lot of space free in cold brick but writes are failing...*

*root@vsan18:~/gluster/glusterfs# df -h*
*. *
*.*
*/dev/sdb3   231G  1.1G  230G   1% /data/brick1*
*/dev/ssd   9.4G  9.4G  104K 100% /data/brick2*
*host:/vol 241G   11G  230G   5% /mnt*

Please let me know if I am missing something.
Is this behavior expected. shouldn't the files be re-balanced?

Thanks,
Ameet
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users