[Gluster-users] When to use striped volumes?

[Dave Fan]

Hello everyone,


We are trying to set up a Gluster-based storage for best performance. On the 
official Gluster website. It says:


Striped – Striped volumes stripes data across bricks in the volume. For best 
results, you should use striped volumes only in high concurrency environments 
accessing very large files.


Is there a rule-of-thumb on what size qualifies as "very large files" here?


Many thanks,
Dave___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Weekly Community meeting 2017-10-18

[Kaushal M]

Hi everyone,

If you haven't already heard, we will be moving to a new schedule for
the weekly community meeting, starting with tomorrows' meeting.

The meeting will be held at 1500UTC tomorrow, in #gluster-meeting on Freenode.

Add your updates and topics for discussion at
https://bit.ly/gluster-community-meetings

See you all tomorrow.

~kaushal
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] New schedule for community meetings - 1500UTC every alternate Wednesday.

[Kaushal M]

Hi All,

We recently changed the community meeting format to make it more
lively and are pleased with the nature of discussions happening since
the change. In order to foster more participation for our meetings, we
will be trying out a bi-weekly cadence and move the meeting to 1500UTC
on alternate Wednesdays. The meeting will continue to happen in
#gluster-meeting on Freenode.


We intend letting the new schedule be effective from  Jan 18. The next
community meeting will be held in #gluster-meeting on Freenode, at
1500UTC on Jan 18.

If you are a regular attendee of the community meetings and will be
inconvenienced by the altered schedule, please let us know.

Thanks,
Kaushal and Vijay
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] arbiter node sharing

[Ravishankar N]

On 01/16/2017 09:42 PM, p...@email.cz wrote:

Hello dears,

how can i share the arbiter node between two-three gluster clusters ??

I've got two clusters ( centos 7.2) with gluster (3.8) filesystem and 
I'd need to share arbiter node between them to spare server nodes.

exam:
gluster volume create SDAP1 replica 3 arbiter 1 
16.0.0.161:/GLUSTER/sdaP1/GFS 16.0.0.162:/GLUSTER/sdaP1/GFS 
16.0.0.159:/GLUSTER/1KVM12-sda1/GFS  force


but gluster peer returns error:
peer probe: failed: 16.0.0.159 is either already part of another 
cluster or having volumes configured  ( YES, it IS , I know)


So, exists any way how to fix this wish ?


No, peer probing a node that is a part of another cluster is not 
supported in gluster. All volumes need to be a part of the same trusted 
storage pool (i.e. cluster).

Regards,
Ravi


regards
Paf1



___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users


___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] [FOSDEM 2017] Announcing a Gluster Stand!

[Amye Scavarda]

On Wed, Nov 30, 2016 at 3:08 PM, Amye Scavarda  wrote:

> FOSDEM has notified us that we get a stand to promote the Gluster
> project at FOSDEM this year, February 4th and 5th, 2017.
>
> This is a call for volunteers to help staff this, please ping me off
> list if you're interested.
>
> This is in addition to the Software Defined Storage Devroom that we're
> already supporting on Sunday.
> Thanks!
>  -amye
>
> --
> Amye Scavarda | a...@redhat.com | Gluster Community Lead
>

Hi all,
If you haven't already notified me of your interest in supporting Gluster
at FOSDEM, please ping me.
We're finalizing schedule and details this week.

Thanks!
- amye

-- 
Amye Scavarda | a...@redhat.com | Gluster Community Lead
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] How to achieve linear performance as adding nodes to gluster cluster (distributed replica volume)?

[haoyuan.ge]

Hi,


I have tested 3 kinds of distributed replica volume: 4 * 2, 3* 2 and 2*2. I 
suppose 4 * 2 should achieve the best IOPS, however, their performance seems 
similar. 


I have tested with "dd if=/dev/zero of=/mnt/glusterfs/block8 bs=128M count=1" 
and "dd if=/dev/zero of=/mnt/glusterfs/block8 bs=32M count=8". All bricks are 
on virtual machines,  with same hardware: 2 cores cpu, 8G memory.


The following is my volume configuration:
Volume Name: rep4
Type: Distributed-Replicate
Volume ID: b2ad2871-cfad-4f2c-afdb-38c2c4d6239c
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: 192.168.16.145:/home/vagrant/rep4
Brick2: 192.168.16.146:/home/vagrant/rep4
Brick3: 192.168.16.82:/home/vagrant/rep4
Brick4: 192.168.16.114:/home/vagrant/rep4

Volume Name: rep6
Type: Distributed-Replicate
Volume ID: 2cbcefce-da7a-4823-aee7-432c40f3ae55
Status: Started
Number of Bricks: 3 x 2 = 6
Transport-type: tcp
Bricks:
Brick1: 192.168.16.49:/home/vagrant/rep6
Brick2: 192.168.16.114:/home/vagrant/rep6
Brick3: 192.168.16.141:/home/vagrant/rep6
Brick4: 192.168.16.145:/home/vagrant/rep6
Brick5: 192.168.16.146:/home/vagrant/rep6
Brick6: 192.168.16.82:/home/vagrant/rep6

Volume Name: rep8
Type: Distributed-Replicate
Volume ID: ea77934c-bd5d-4578-8b39-c02402d00739
Status: Started
Number of Bricks: 4 x 2 = 8
Transport-type: tcp
Bricks:
Brick1: 192.168.16.145:/home/vagrant/rep8
Brick2: 192.168.16.146:/home/vagrant/rep8
Brick3: 192.168.16.114:/home/vagrant/rep8
Brick4: 192.168.16.82:/home/vagrant/rep8
Brick5: 192.168.16.141:/home/vagrant/rep8
Brick6: 192.168.16.49:/home/vagrant/rep8
Brick7: 192.168.16.144:/home/vagrant/rep8
Brick8: 192.168.16.112:/home/vagrant/rep8



According to 
"http://moo.nac.uci.edu/~hjm/Performance_in_a_Gluster_Systemv6F.pdf;, 
> To scale out performance, enterprises need simply add additional storage 
> server nodes, and will generally see linear performance improvements.


I wonder how can I achieve  linear performance improvements? Have I tested in 
the wrong way?
--
Regards,
Haoyuan Ge___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] How to achieve linear performance as adding nodes to gluster cluster (distributed replica volume)?

[H. Ge]

Hi,

I have tested 3 kinds of distributed replica volume: 4 * 2, 3* 2 and 2*2. I
suppose 4 * 2 should achieve the best IOPS, however, their performance
seems similar.

I have tested with "*dd if=/dev/zero of=/mnt/glusterfs/block8 bs=128M
count=1*" and "*dd if=/dev/zero of=/mnt/glusterfs/block8 bs=32M count=8*".
All bricks are on virtual machines,  with same hardware: 2 cores cpu, 8G
memory.

The following is my volume configuration:
Volume Name: rep4
Type: Distributed-Replicate
Volume ID: b2ad2871-cfad-4f2c-afdb-38c2c4d6239c
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: 192.168.16.145:/home/vagrant/rep4
Brick2: 192.168.16.146:/home/vagrant/rep4
Brick3: 192.168.16.82:/home/vagrant/rep4
Brick4: 192.168.16.114:/home/vagrant/rep4
Volume Name: rep6
Type: Distributed-Replicate
Volume ID: 2cbcefce-da7a-4823-aee7-432c40f3ae55
Status: Started
Number of Bricks: 3 x 2 = 6
Transport-type: tcp
Bricks:
Brick1: 192.168.16.49:/home/vagrant/rep6
Brick2: 192.168.16.114:/home/vagrant/rep6
Brick3: 192.168.16.141:/home/vagrant/rep6
Brick4: 192.168.16.145:/home/vagrant/rep6
Brick5: 192.168.16.146:/home/vagrant/rep6
Brick6: 192.168.16.82:/home/vagrant/rep6
Volume Name: rep8
Type: Distributed-Replicate
Volume ID: ea77934c-bd5d-4578-8b39-c02402d00739
Status: Started
Number of Bricks: 4 x 2 = 8
Transport-type: tcp
Bricks:
Brick1: 192.168.16.145:/home/vagrant/rep8
Brick2: 192.168.16.146:/home/vagrant/rep8
Brick3: 192.168.16.114:/home/vagrant/rep8
Brick4: 192.168.16.82:/home/vagrant/rep8
Brick5: 192.168.16.141:/home/vagrant/rep8
Brick6: 192.168.16.49:/home/vagrant/rep8
Brick7: 192.168.16.144:/home/vagrant/rep8
Brick8: 192.168.16.112:/home/vagrant/rep8

According to "http://moo.nac.uci.edu/~hjm/Performance_in_a_Gluster_Systemv
6F.pdf",
*> To scale out performance, enterprises need simply add additional storage
server nodes, and will generally see linear performance improvements.*

I wonder how can I achieve  linear performance improvements? Have I tested
in the wrong way?

Regards
Harry(Haoyuan) Ge
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Migration path from native Gluster-NFS towards NFS-Ganesha

[Giuseppe Ragusa]

Sorry: unfortunately it seems that I explained my setup/intentions in a very 
misleading way :-)


 (English is not my native language, but I know this is a poor excuse)



What I meant is that I configured an hyperconverged oVirt+Gluster
cluster, but I added CTDB, Gluster-NFS and Samba on top (well, on the
bottom actually, since they too run on the hypervisor hosts) to allow
using Gluster-based storage also for general purpose use (not only for
oVirt-related uses).


The Gluster volumes meant for general purpose use are the only ones
accessed by means of NFS and/or Samba.


All oVirt-related uses are by means of FUSE-based mountpoints (since
oVirt 3.6.x has not been gfapi-enabled).


Best regards,

Giuseppe





On Mon, Jan 16, 2017, at 19:51, Gambit15 wrote:

> Why are you using NFS for using Gluster with oVirt? oVirt is natively
> able to mount Gluster volumes via FUSE, which is *far* more efficient!
> Doug

> 

> On 12 January 2017 at 18:36, Giuseppe Ragusa
>  wrote:
>> Hi all,

>> 

>>  In light of the future removal of native Gluster-NFS (and also
>>  because of a worrying bug that causes NFS crashes, see
>>  https://bugzilla.redhat.com/show_bug.cgi?id=1381970 then
>>  http://www.gluster.org/pipermail/gluster-users/2016-November/029333.html
>>  and recently
>>  http://www.gluster.org/pipermail/gluster-users/2017-January/029632.html
>>  ) I'm planning to move towards NFS-Ganesha.
>> 

>>  I have a couple of questions for which I could not find answers on
>>  the available docs (sorry if I missed something):
>> 

>>  1) Is it possible (and advisable, in production too) today (3.8.x)
>> to configure a GlusterFS based cluster to use NFS-Ganesha (as NFS
>> v3/v4 solution) and Samba (as CIFS solution) both controlled by
>> CTDB as a highly available *and* load balanced (multiple IPs with
>> DNS round-robin, not active/passive) storage solution? (note: I
>> mean *without* using a full Pacemaker+Corosync stack)
>> 

>>  2) If the answer to the above question is "yes", is the above above
>> mentioned solution capable of coexisting with oVirt in an
>> hyperconverged setup (assuming replica 3 etc. etc.)?
>> 

>>  Many thanks in advance to anyone who can answer the above and/or
>>  point me to any relevant resources/docs.
>> 

>>  Best regards,

>>  Giuseppe

>> 

>>  ___

>>  Gluster-users mailing list

>> Gluster-users@gluster.org

>> http://www.gluster.org/mailman/listinfo/gluster-users


___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Migration path from native Gluster-NFS towards NFS-Ganesha

[Giuseppe Ragusa]

On Mon, Jan 16, 2017, at 14:08, Kaleb S. KEITHLEY wrote:
> On 01/12/2017 04:36 PM, Giuseppe Ragusa wrote:
> > Hi all,
> > 
> > 1) Is it possible (and advisable, in production too) today (3.8.x) to 
> > configure a GlusterFS based cluster to use NFS-Ganesha (as NFS v3/v4 
> > solution) and Samba (as CIFS solution) both controlled by CTDB as a highly 
> > available *and* load balanced (multiple IPs with DNS round-robin, not 
> > active/passive) storage solution? (note: I mean *without* using a full 
> > Pacemaker+Corosync stack)
> 
> It's probably doable.
> 
> The only reason it's not advisable — IMO — is that it's not what we're
> doing, and getting help could be pretty hard.
> 
> The Samba team has all the CTDB experience. I've poked them — hopefully
> they will respond.

I've already integrated native Gluster-NFS with CTDB using the monitoring 
solution tracked in:

https://bugzilla.redhat.com/show_bug.cgi?id=1371178

From the docs it seems that all the actual NFS HA/LB work (in what you're 
doing) is done by Ganesha upcalls, but nonetheless all setup hints/script 
assume that a full Pacemaker+Corosync stack is present, so I should read 
through all scripts to untangle it from them...
I will do this eventually anyway (since Gluster-NFS is deprecated), but I would 
try to avoid doing this in a hurry and without any guidance from the 
developers, only to solve stability issues (that's unfortunately the immediate 
reason why I posted my request for advice on the migration). ;-)

> Is there some reason you don't want to use Pacemaker and Corosync?

All the reasons lie in the fact that I don't think it's advisable to make oVirt 
and Pacemaker+Corosync coexist on the same machines (oVirt has it's own 
PowerManagement which would overlap with Pacemaker fencing, just to name the 
most obvious problem tha comes to my mind...)
If I had a pure storage setup to care for (no hyperconvergence), I would not 
have thought twice and I would already have a nice Pacemaker-enabled setup

> > 2) If the answer to the above question is "yes", is the above above 
> > mentioned solution capable of coexisting with oVirt in an hyperconverged 
> > setup (assuming replica 3 etc. etc.)?
> 
> Off hand I can't think of any reason why not.

Many thanks for your advice!
When I will come to try the integration, I will duly document it and report 
back for any issue.

> > Many thanks in advance to anyone who can answer the above and/or point me 
> > to any relevant resources/docs.
> > 
> 
> https://github.com/linux-ha-storage/storhaug is basis for the Common HA
> solution for NFS-Ganesha and Samba that GlusterFS-3.10 will be using.
> N.B. It's also based on Pacemaker and Corosync.

As I already mentioned to some oVirt developers when we met face to face in 
Italy: oVirt is a nice VMware killer solution, but integrating GlusterFS only 
to kill VSAN it's too humble a project... ;-)
Let's kill also NetApp alltogether while we are at it :-D
And no, don't think I dislike Pacemaker+Corosync at all: it's wonderful (and I 
actually use it alot), but this one seems not its game ;-)

Best regards,
Giuseppe

> -- 
> 
> Kaleb
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] NFS service dying

[Giuseppe Ragusa]

On Mon, Jan 16, 2017, at 11:18, Niels de Vos wrote:
> On Fri, Jan 13, 2017 at 06:43:33PM +0100, Giuseppe Ragusa wrote:
> > On Fri, Jan 13, 2017, at 12:39, Niels de Vos wrote:
> > > On Wed, Jan 11, 2017 at 11:58:29AM -0700, Paul Allen wrote:
> > > > I'm running into an issue where the gluster nfs service keeps dying on a
> > > > new cluster I have setup recently. We've been using Gluster on several
> > > > other clusters now for about a year or so and I have never seen this
> > > > issue before, nor have I been able to find anything remotely similar to
> > > > it while searching on-line. I initially was using the latest version in
> > > > the Gluster Debian repository for Jessie, 3.9.0-1, and then I tried
> > > > using the next one down, 3.8.7-1. Both behave the same for me.
> > > > 
> > > > What I was seeing was after a while the nfs service on the NAS server
> > > > would suddenly die after a number of processes had run on the app server
> > > > I had connected to the new NAS servers for testing (we're upgrading the
> > > > NAS servers for this cluster to newer hardware and expanded storage, the
> > > > current production NAS servers are using nfs-kernel-server with no type
> > > > of clustering of the data). I checked the logs but all it showed me was
> > > > something that looked like a stack trace in the nfs.log and the
> > > > glustershd.log showed the nfs service disconnecting. I turned on
> > > > debugging but it didn't give me a whole lot more, and certainly nothing
> > > > that helps me identify the source of my issue. It is pretty consistent
> > > > in dying shortly after I mount the file system on the servers and start
> > > > testing, usually within 15-30 minutes. But if I have nothing using the
> > > > file system, mounted or no, the service stays running for days. I tried
> > > > mounting it using the gluster client, and it works fine, but I can;t use
> > > > that due to the performance penalty, it slows the websites down by a few
> > > > seconds at a minimum.
> > > 
> > > This seems to be related to the NLM protocol that Gluster/NFS provides.
> > > Earlier this week one of our Red Hat quality engineers also reported
> > > this (or a very similar) problem.
> > > 
> > > https://bugzilla.redhat.com/show_bug.cgi?id=1411344
> > > 
> > > At the moment I suspect that this is related to re-connects of some
> > > kind, but I have not been able to identify the cause sufficiently to be
> > > sure. This definitely is a coding problem in Gluster/NFS, but the more I
> > > look at the NLM implementation, the more potential issues I see with it.
> > 
> > Should we assume that, with the complete removal of Gluster/NFS
> > already on the horizon, debugging and fixing NLM (even if only for the
> > more common, reported crash cases) would be an extremely low-priority
> > task? ;-)
> 
> Crashes are expected to be fixed, if they happen in 'normal'
> circumstances. It is not an extremely low priority, but neither is it
> very high. I'm looking into it, but am also working on other higher
> priority things. Maybe I have a fix by the end of the week, depending on
> whatever distracts me from working on it.

Many thanks for all your efforts!
I will eagerly await any patch and I will upgrade to the version that contains 
it (even if that means upgrading from 3.7.x to 3.8.x) or I could even recompile 
local rpms as soon as a tested patch is available in Gerrit and then upgrade to 
that at the first available maintenance window.

If it can help, I had already added my logs/info to 
https://bugzilla.redhat.com/show_bug.cgi?id=1381970 (but could ultimately be a 
different issue, of course; it is easily reproducible, anyway) and could try to 
help in collecting further details/logs if you need them.

> > Would it be possible for someone to simply check whether the crashes
> > happen also on the 3.7.9-12 codebase used in latest RHGS 3.1.3?
> > My cluster is already at 3.7.12 feature level (and using it), so I
> > suppose I could not easily downgrade.
> > Since Red Hat QA found a similar problem while testing the 3.8.4-10
> > codebase in RHGS 3.2.0, we could trace the problem back to post-3.7.9
> > developments, if RHGS 3.1.3 is immune.
> 
> I expect to see this problem in older versions too. There has been very
> little change to the NLM code in Gluster. It is possible that the Linux
> kernel was updated and the NFS-client behaves a little different,
> exposing this problem just now, or the testing has changed...

I thought that it could not be present in earlier/current RHGS versions *and* 
escape QA/real_use, but your suggestion could be the explanation; maybe even 
the fact that most use could have been on RHEL 6 up to now (so the kernel NFS 
client difference vs RHEL 7 could have been even more marked).

> > > If the workload does not require locking operations, you may be able to
> > > work around the problem by mounting with "-o nolock". Depending on the
> > > application, this can be safe or cause data corruption...

Re: [Gluster-users] Connect to ec2 instance via libgfapi

[Niels de Vos]

On Wed, Jan 11, 2017 at 01:25:16PM +, David Spisla wrote:
> 
> Hello,
> 
> I am from germany and have this issue for discussion: I am running a gluster 
> volume (/gv0) on a CentOS Machine (ec2 instance from Amazon AWS) and want to 
> use libgfapi to connect from my CentOS VM (local computer) to this volume.
> 
> For this purpose I use the GlusterClient class to connect. This is my 
> initialization:
> 
> GlusterClient cl = new 
> GlusterClient(""ec2-54-xxx-xxx-xx.eu-west-1.compute.amazonaws.com"",  24007, 
> "tcp").
> 
> It seems to be that I have a connection and my programming starts to create a 
> file. But after start creating a file my program hanging around and there is 
> no response from the system.
> Does anybody has experiences using libgfapi to an ec2-instance?

You need port tcp/24007 to connect to the GlusterD process to get the
volume layout (.vol file). After that, gfapi will connect to all the
bricks of the volume. That means, you need to open many more ports on
all your Gluster servers in order to get a usable GlusterFS connection.
The gfapi log will most likely show you connection errors.

For questions and discussions related to gfapi, you can also use our
relatively new integrat...@gluster.org list.

HTH,
Niels


> 
> Thank you for your attention!
> 
> 
> David Spisla
> Software Developer
> david.spi...@iternity.com
> www.iTernity.com
> Tel:   +49 761-590 34 841
> 
> [cid:image001.png@01D239C7.FDF7B430]
> 
> iTernity GmbH
> Heinrich-von-Stephan-Str. 21
> 79100 Freiburg - Germany
> ---
> unseren technischen Support erreichen Sie unter +49 761-387 36 66
> ---
> Geschäftsführer: Ralf Steinemann
> Eingetragen beim Amtsgericht Freiburg: HRB-Nr. 701332
> USt.Id de-24266431
> 



> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users



signature.asc
Description: PGP signature
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] An other Gluster 3.8 Long-Term-Maintenance update with the 3.8.8 release

[Niels de Vos]

[from: http://blog.nixpanic.net/2017/01/gluster-388-ltm-update.html
 plain-text generated by 'links -dump ...' and slightly edited by hand]

The Gluster team has been busy over the end-of-year holidays and this
latest update to the 3.8 [9]Long-Term-Maintenance release intends to fix
quite a number of bugs. Packages have been built for [10]many different
distributions and are available from the download server. The
[11]release-notes for 3.8.8 have been included below for the ease of
reference. All users on the 3.8 version are recommended to update to
this current release.

Cheers,
Niels


# Release notes for Gluster 3.8.8

This is a bugfix release. The [12]Release Notes for 3.8.0, [13]3.8.1,
[14]3.8.2, [15]3.8.3, [16]3.8.4, [17]3.8.5, [18]3.8.6 and [19]3.8.7
contain a listing of all the new features that were added and bugs fixed
in the GlusterFS 3.8 stable release.

# Bugs addressed

   A total of 38 patches have been merged, addressing 35 bugs:

 * [20]#1375849: [RFE] enable sharding with virt profile -
   /var/lib/glusterd/groups/virt
 * [21]#1378384: log level set in glfs_set_logging() does not work
 * [22]#1378547: Asynchronous Unsplit-brain still causes Input/Output
   Error on system calls
 * [23]#1389781: build: python on Debian-based dists use
   .../lib/python2.7/dist-packages instead of .../site-packages
 * [24]#1394635: errors appear in brick and nfs logs and getting stale
   files on NFS clients
 * [25]#1395510: Seeing error messages
   [snapview-client.c:283:gf_svc_lookup_cbk] and
   [dht-helper.c:1666ht_inode_ctx_time_update]
   (-->/usr/lib64/glusterfs/3.8.4/xlator/cluster/replicate.so(+0x5d75c)
 * [26]#1399423: GlusterFS client crashes during remove-brick operation
 * [27]#1399432: A hard link is lost during rebalance+lookup
 * [28]#1399468: Wrong value in Last Synced column during Hybrid Crawl
 * [29]#1399915: [SAMBA-CIFS] : IO hungs in cifs mount while graph switch
   on & off
 * [30]#1401029: OOM kill of nfs-ganesha on one node while fs-sanity test
   suite is executed.
 * [31]#1401534: fuse mount point not accessible
 * [32]#1402697: glusterfsd crashed while taking snapshot using scheduler
 * [33]#1402728: Worker restarts on log-rsync-performance config update
 * [34]#1403109: Crash of glusterd when using long username with
   geo-replication
 * [35]#1404105: Incorrect incrementation of volinfo refcnt during volume
   start
 * [36]#1404583: Upcall: Possible use after free when log level set to
   TRACE
 * [37]#1405004: [Perf] : pcs cluster resources went into stopped state
   during Multithreaded perf tests on RHGS layered over RHEL 6
 * [38]#1405130: `gluster volume heal split-brain' does not heal if
   data/metadata/entry self-heal options are turned off
 * [39]#1405450: tests/bugs/snapshot/bug-1316437.t test is causing
   spurious failure
 * [40]#1405577: [GANESHA] failed to create directory of hostname of new
   node in var/lib/nfs/ganesha/ in already existing cluster nodes
 * [41]#1405886: Fix potential leaks in INODELK cbk in protocol/client
 * [42]#1405890: Fix spurious failure in bug-1402841.t-mt-dir-scan-race.t
 * [43]#1405951: NFS-Ganesha:Volume reset for any option causes reset of
   ganesha enable option and bring down the ganesha services
 * [44]#1406740: Fix spurious failure in
   tests/bugs/replicate/bug-1402730.t
 * [45]#1408414: Remove-brick rebalance failed while rm -rf is in
   progress
 * [46]#1408772: [Arbiter] After Killing a brick writes drastically slow
   down
 * [47]#1408786: with granular-entry-self-heal enabled i see that there
   is a gfid mismatch and vm goes to paused state after migrating to
   another host
 * [48]#1410073: Fix failure of split-brain-favorite-child-policy.t in
   CentOS7
 * [49]#1410369: Dict_t leak in dht_migration_complete_check_task and
   dht_rebalance_inprogress_task
 * [50]#1410699: [geo-rep]: Config commands fail when the status is
   'Created'
 * [51]#1410708: glusterd/geo-rep: geo-rep config command leaks fd
 * [52]#1410764: Remove-brick rebalance failed while rm -rf is in
   progress
 * [53]#1411011: atime becomes zero when truncating file via ganesha (or
   gluster-NFS)
 * [54]#1411613: Fix the place where graph switch event is logged

    

References

   1. http://blog.nixpanic.net/2017/01/gluster-388-ltm-update.html
   ... removed links generated by 'links -dump $URL'
   9. https://www.gluster.org/community/release-schedule/
  10. http://gluster.readthedocs.io/en/latest/Install-Guide/Community_Packages/
  11. 
https://github.com/gluster/glusterfs/blob/v3.8.8/doc/release-notes/3.8.8.md
  12. 
https://github.com/gluster/glusterfs/blob/v3.8.8/doc/release-notes/3.8.0.md
  13. 
https://github.com/gluster/glusterfs/blob/v3.8.8/doc/release-notes/3.8.1.md
  14. 

[Gluster-users] rebalance and volume commit hash

[Piotr Misiak]

Hi,

Can you tell me please why every volume rebalance generates a new value
for the volume commit hash?

If I have fully rebalanced cluster (or almost) with millions of
directories then rebalance has to change DHT xattr for every directory
only because there is a new volume commit hash value. It is pointless in
my opinion. Is there any reason behind this? As I observed, the volume
commit hash is set at the rebalance beginning which totally destroys
benefit of lookup optimization algorithm for directories not
scanned/fixed yet by this rebalance run.

I'm also curious what is happening during file lookup. As I know DHT
hash ranges are stored only in DHT xattr on every brick.
If gluster needs to find on which brick the file is located, it has to
read DHT xattr from every distribute brick to build complete hash ring.
Am I right?
I suppose this information is then cached but for how long and how big
is this cache, is it configurable?

Thanks,

-- 
Piotr Misiak
Senior Cloud Engineer
CloudFerro Sp. z o.o.


___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Connect to ec2 instance via libgfapi

[David Spisla]

Hello,

I am from germany and have this issue for discussion: I am running a gluster 
volume (/gv0) on a CentOS Machine (ec2 instance from Amazon AWS) and want to 
use libgfapi to connect from my CentOS VM (local computer) to this volume.

For this purpose I use the GlusterClient class to connect. This is my 
initialization:

GlusterClient cl = new 
GlusterClient(""ec2-54-xxx-xxx-xx.eu-west-1.compute.amazonaws.com"",  24007, 
"tcp").

It seems to be that I have a connection and my programming starts to create a 
file. But after start creating a file my program hanging around and there is no 
response from the system.
Does anybody has experiences using libgfapi to an ec2-instance?

Thank you for your attention!


David Spisla
Software Developer
david.spi...@iternity.com
www.iTernity.com
Tel:   +49 761-590 34 841

[cid:image001.png@01D239C7.FDF7B430]

iTernity GmbH
Heinrich-von-Stephan-Str. 21
79100 Freiburg - Germany
---
unseren technischen Support erreichen Sie unter +49 761-387 36 66
---
Geschäftsführer: Ralf Steinemann
Eingetragen beim Amtsgericht Freiburg: HRB-Nr. 701332
USt.Id de-24266431

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] questions about the implemention of gf_backtrace_fillframes

[C0reFast]

When I trying to listing /tmp in a gluster server. I found that have some bt 
prefix files like:[root@xxx# ll /tmp/
ls: cannot access /tmp/btm1RpMm: No such file or directory
total 116
-rw---  1 root root   129 Jan 13 17:11 bt0tJH3c
-?? ? ???? btm1RpMm
-rw---  1 root root   287 Jan 13 17:11 btyDGNO9
drwx--. 2 root root 16384 Oct 19 09:21 lost+found
 
so i checked the source code. found in libglusterfs/src/common-utils.c


in function gf_backtrace_fillframes:


1. call frames = backtrace (array, GF_BACKTRACE_FRAME_COUNT); to get backtrace.
2. save backtrace to a temp file.
3. read backtrace from temp file to btbuf.


so why need writing a temp file instead just doing this in memory?___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] QoS in GlusterFS

[Yann MAUPU]

Hi,

I am currently working on a project involving 3 nodes using GlusterFS 
3.8.5 in "disperse" mode to act like raid-5, and I need to have QoS.
Is it possible to guarantee a certain bandwidth, given that we know the 
"idle" max bandwidth of the system ?
For example, if I can check that the write bandwidth for one user is 400 
MB/s, can I be sure that, with 4 users, each can get at least 80 MB/s ?


Thanks in advance.
Best regards,
Yann
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Ganesha with Gluster transport RDMA does not work

[Andreas Kurzac]

Hi Jiffin,

i raised bug 1411281.
If you could provide test-rpms i would be very happy to test them in our 
environment.
In the meantime i will switch to tcp,rdma and continue working on our setup, we 
can then switch back to pure rdma any time for testing.

Thanks for your help!

Regards,
Andreas





Von: Jiffin Tony Thottan [mailto:jthot...@redhat.com]
Gesendet: Montag, 9. Januar 2017 06:02
An: Andreas Kurzac ; gluster-users@gluster.org
Betreff: Re: [Gluster-users] Ganesha with Gluster transport RDMA does not work


Hi Andreas,

By checking the code IMO currently this is limitation with in FSAL_GLUSTER. It 
tries to

establish connection with glusterfs servers only using "tcp". It is easy to fix 
as well.

You can raise a bug in  
https://bugzilla.redhat.com/enter_bug.cgi?product=nfs-ganesha

under FSAL_GLUSTER. I don't have any hardware to test the fix. I can either 
help you in

writing up fix for the issue or provide a test rpms with the fix .

Also thanks for trying out nfs-ganesha  with rdma and finding about this issue.

For the time being , if possible you can try with tcp,rdma volume to solve the 
problem.

Regards,

Jiffin


On 06/01/17 22:56, Andreas Kurzac wrote:

Dear All,



i have a glusterfs pool with 3 servers with Centos7.3, Glusterfs 3.8.5, network 
is Infiniband.

Pacemaker/Corosync and Ganesha-NFS is installed and all seems to be OK, no 
error logged.

I created a replica 3 volume with transport rdma (without tcp!).

When i mount this volume via glusterfs and do some IO, no errors are logged and 
everything seems to go pretty well.



When i mount the volume via nfs and do some IO, nfs freezes immediatly and 
following logs are written to

ganesha-gfapi.log:

2017-01-05 23:23:53.536526] W [MSGID: 103004] 
[rdma.c:452:gf_rdma_register_arena] 0-rdma: allocation of mr failed

[2017-01-05 23:23:53.541519] W [MSGID: 103004] 
[rdma.c:1463:__gf_rdma_create_read_chunks_from_vector] 0-rpc-transport/rdma: 
memory registration failed (peer:10.40.1.1:49152) [Keine Berechtigung]

[2017-01-05 23:23:53.541547] W [MSGID: 103029] 
[rdma.c:1558:__gf_rdma_create_read_chunks] 0-rpc-transport/rdma: cannot create 
read chunks from vector entry->prog_payload

[2017-01-05 23:23:53.541553] W [MSGID: 103033] 
[rdma.c:2063:__gf_rdma_ioq_churn_request] 0-rpc-transport/rdma: creation of 
read chunks failed

[2017-01-05 23:23:53.541557] W [MSGID: 103040] 
[rdma.c:2775:__gf_rdma_ioq_churn_entry] 0-rpc-transport/rdma: failed to process 
request ioq entry to peer(10.40.1.1:49152)

[2017-01-05 23:23:53.541562] W [MSGID: 103040] [rdma.c:2859:gf_rdma_writev] 
0-vmstor1-client-0: processing ioq entry destined to (10.40.1.1:49152) failed

[2017-01-05 23:23:53.541569] W [MSGID: 103037] 
[rdma.c:3016:gf_rdma_submit_request] 0-rpc-transport/rdma: sending request to 
peer (10.40.1.1:49152) failed

[...]



Some additional info:

Firewall is disabled, SELinux is disabled.

Different hardware with Centos 7.1 and the Mellanox OFED 3.4 packages instead 
of the Centos Infiniband packages lead to the same results.

Just to mention: I am not trying to do NFS over RDMA, the Ganesha FSAL is just 
configured to "glusterfs".



I hope someone could help me, i am running out of ideas...



Kind regards,

Andreas





___

Gluster-users mailing list

Gluster-users@gluster.org

http://www.gluster.org/mailman/listinfo/gluster-users

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Migration path from native Gluster-NFS towards NFS-Ganesha

[Gambit15]

Why are you using NFS for using Gluster with oVirt? oVirt is natively able
to mount Gluster volumes via FUSE, which is *far* more efficient!

Doug

On 12 January 2017 at 18:36, Giuseppe Ragusa 
wrote:

> Hi all,
>
> In light of the future removal of native Gluster-NFS (and also because of
> a worrying bug that causes NFS crashes, see https://bugzilla.redhat.com/
> show_bug.cgi?id=1381970 then http://www.gluster.org/
> pipermail/gluster-users/2016-November/029333.html and recently
> http://www.gluster.org/pipermail/gluster-users/2017-January/029632.html )
> I'm planning to move towards NFS-Ganesha.
>
> I have a couple of questions for which I could not find answers on the
> available docs (sorry if I missed something):
>
> 1) Is it possible (and advisable, in production too) today (3.8.x) to
> configure a GlusterFS based cluster to use NFS-Ganesha (as NFS v3/v4
> solution) and Samba (as CIFS solution) both controlled by CTDB as a highly
> available *and* load balanced (multiple IPs with DNS round-robin, not
> active/passive) storage solution? (note: I mean *without* using a full
> Pacemaker+Corosync stack)
>
> 2) If the answer to the above question is "yes", is the above above
> mentioned solution capable of coexisting with oVirt in an hyperconverged
> setup (assuming replica 3 etc. etc.)?
>
> Many thanks in advance to anyone who can answer the above and/or point me
> to any relevant resources/docs.
>
> Best regards,
> Giuseppe
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] rdma transport uses scif0 device (iWARP) on on server with XeonPHI

[Fedele Stabile]

Hello all,
at the end I found the cause of my problems:
if in the gluster-server is installed a Xeon-PHI (mic) card and the
server is configured with a scif0 (virtual adapter for bridging withmic)  
device and a qib0 
(you can see output of ibv_devinfo below)
Gluster uses by default scif0 on which there is no RDMA support.
So the question is: can we change device in any configuration file?
Thank you in advance,
Fedele

# ibv_devinfo 
hca_id: scif0
transport:  iWARP (1)
fw_ver: 0.0.1
node_guid:  4c79:baff:fe66:0781
sys_image_guid: 4c79:baff:fe66:0781
vendor_id:  0x8086
vendor_part_id: 0
hw_ver: 0x1
phys_port_cnt:  1
port:   1
state:  PORT_ACTIVE (4)
max_mtu:4096 (5)
active_mtu: 4096 (5)
sm_lid: 1
port_lid:   1000
port_lmc:   0x00
link_layer: Ethernet

hca_id: qib0
transport:  InfiniBand (0)
fw_ver: 0.0.0
node_guid:  0011:7500:006f:7446
sys_image_guid: 0011:7500:006f:7446
vendor_id:  0x1175
vendor_part_id: 29474
hw_ver: 0x2
board_id:   InfiniPath_QLE7340
phys_port_cnt:  1
port:   1
state:  PORT_ACTIVE (4)
max_mtu:4096 (5)
active_mtu: 2048 (4)
sm_lid: 1
port_lid:   34
port_lmc:   0x00
link_layer: InfiniBand

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] arbiter node sharing

[p...@email.cz]

Hello dears,

how can i share the arbiter node between two-three gluster clusters ??

I've got two clusters ( centos 7.2) with gluster (3.8)  filesystem and 
I'd need to share arbiter node between them to spare server nodes.

exam:
gluster volume create SDAP1 replica 3 arbiter 1 
16.0.0.161:/GLUSTER/sdaP1/GFS 16.0.0.162:/GLUSTER/sdaP1/GFS 
16.0.0.159:/GLUSTER/1KVM12-sda1/GFS  force


but gluster peer returns error:
peer probe: failed: 16.0.0.159 is either already part of another cluster 
or having volumes configured  ( YES, it IS , I know)


So, exists any way how to fix this wish ?

regards
Paf1

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] [Gluster-devel] Lot of EIO errors in disperse volume

[Ankireddypalle Reddy]

Xavi,
   Thanks. I will start by tracking the delta if any in return codes 
from the bricks for writes in EC. I have a feeling that  if we could get to the 
bottom of this issue then the EIO errors could be ultimately avoided.  Please 
note that while we are testing all these on a standby setup we continue to 
encounter the EIO errors on our production setup which is running @ 3.7.18.

Thanks and Regards,
Ram

-Original Message-
From: Xavier Hernandez [mailto:xhernan...@datalab.es] 
Sent: Monday, January 16, 2017 6:54 AM
To: Ankireddypalle Reddy
Cc: gluster-users@gluster.org; Gluster Devel (gluster-de...@gluster.org)
Subject: Re: [Gluster-devel] [Gluster-users] Lot of EIO errors in disperse 
volume

Hi Ram,

On 16/01/17 12:33, Ankireddypalle Reddy wrote:
> Xavi,
>   Thanks. Is there any other way to map from GFID to path.

The only way I know is to search all files from bricks and lookup for 
the trusted.gfid xattr.

> I will look for a way to share the TRACE logs. Easier way might be to add 
> some extra logging. I could do that if you could let me know functions in 
> which you are interested..

The problem is that I don't know where the problem is. One possibility 
could be to track all return values from all bricks for all writes and 
then identify which ones belong to an inconsistent file.

But if this doesn't reveal anything interesting we'll need to look at 
some other place. And this can be very tedious and slow.

Anyway, what we are looking now is not the source of an EIO, since there 
are two bricks with consistent state and the file should be perfectly 
readable and writable. It's true that there's some problem here and it 
could derive in EIO if one of the healthy bricks degrades, but at least 
this file shouldn't be giving EIO errors for now.

Xavi

>
> Sent on from my iPhone
>
>> On Jan 16, 2017, at 6:23 AM, Xavier Hernandez  wrote:
>>
>> Hi Ram,
>>
>>> On 13/01/17 18:41, Ankireddypalle Reddy wrote:
>>> Xavi,
>>> I enabled TRACE logging. The log grew up to 120GB and could not 
>>> make much out of it. Then I started logging GFID in the code where we were 
>>> seeing errors.
>>>
>>> [2017-01-13 17:02:01.761349] I [dict.c:3065:dict_dump_to_log] 
>>> 0-glusterfsProd-disperse-0: dict=0x7fa6706bc690 
>>> ((trusted.ec.size:0:0:0:0:30:6b:0:0:)(trusted.ec.version:0:0:0:0:0:0:2a:38:0:0:0:0:0:0:2a:38:))
>>> [2017-01-13 17:02:01.761360] I [dict.c:3065:dict_dump_to_log] 
>>> 0-glusterfsProd-disperse-0: dict=0x7fa6706bed64 
>>> ((trusted.ec.size:0:0:0:0:0:0:0:0:)(trusted.ec.version:0:0:0:0:0:0:0:0:0:0:0:0:0:0:2a:38:))
>>> [2017-01-13 17:02:01.761365] W [MSGID: 122056] 
>>> [ec-combine.c:881:ec_combine_check] 0-glusterfsProd-disperse-0: Mismatching 
>>> xdata in answers of 'LOOKUP'
>>> [2017-01-13 17:02:01.761405] I [dict.c:166:key_value_cmp] 
>>> 0-glusterfsProd-disperse-0: 'trusted.ec.size' is different in two dicts (8, 
>>> 8)
>>> [2017-01-13 17:02:01.761417] I [dict.c:3065:dict_dump_to_log] 
>>> 0-glusterfsProd-disperse-0: dict=0x7fa6706bbb14 
>>> ((trusted.ec.size:0:0:0:0:30:6b:0:0:)(trusted.ec.version:0:0:0:0:0:0:2a:38:0:0:0:0:0:0:2a:38:))
>>> [2017-01-13 17:02:01.761428] I [dict.c:3065:dict_dump_to_log] 
>>> 0-glusterfsProd-disperse-0: dict=0x7fa6706bed64 
>>> ((trusted.ec.size:0:0:0:0:0:0:0:0:)(trusted.ec.version:0:0:0:0:0:0:0:0:0:0:0:0:0:0:2a:38:))
>>> [2017-01-13 17:02:01.761433] W [MSGID: 122056] 
>>> [ec-combine.c:881:ec_combine_check] 0-glusterfsProd-disperse-0: Mismatching 
>>> xdata in answers of 'LOOKUP'
>>> [2017-01-13 17:02:01.761442] W [MSGID: 122006] 
>>> [ec-combine.c:214:ec_iatt_combine] 0-glusterfsProd-disperse-0: Failed to 
>>> combine iatt (inode: 11275691004192850514-11275691004192850514, gfid: 
>>> 60b990ed-d741-4176-9c7b-4d3a25fb8252  -  
>>> 60b990ed-d741-4176-9c7b-4d3a25fb8252,  links: 1-1, uid: 0-0, gid: 0-0, 
>>> rdev: 0-0,size: 406650880-406683648, mode: 100775-100775)
>>>
>>> The file for which we are seeing this error turns out to be having a GFID 
>>> of 60b990ed-d741-4176-9c7b-4d3a25fb8252
>>>
>>> Then I tried looking for find out the file with this GFID. It pointed me to 
>>> following path. I was expecting a real file system path from the following 
>>> turorial:
>>> https://gluster.readthedocs.io/en/latest/Troubleshooting/gfid-to-path/
>>
>> I think this method only works if bricks have the inode cached.
>>
>>>
>>> getfattr -n trusted.glusterfs.pathinfo -e text 
>>> /mnt/gfid/.gfid/60b990ed-d741-4176-9c7b-4d3a25fb8252
>>> getfattr: Removing leading '/' from absolute path names
>>> # file: mnt/gfid/.gfid/60b990ed-d741-4176-9c7b-4d3a25fb8252
>>> trusted.glusterfs.pathinfo="( 
>>> ( 
>>> 
>>>  
>>> ))"
>>>
>>> Then I looked for the xatttrs for these files from all the 3 bricks
>>>
>>> 

Re: [Gluster-users] Migration path from native Gluster-NFS towards NFS-Ganesha

[Kaleb S. KEITHLEY]

On 01/12/2017 04:36 PM, Giuseppe Ragusa wrote:
> Hi all,
> 
> 1) Is it possible (and advisable, in production too) today (3.8.x) to 
> configure a GlusterFS based cluster to use NFS-Ganesha (as NFS v3/v4 
> solution) and Samba (as CIFS solution) both controlled by CTDB as a highly 
> available *and* load balanced (multiple IPs with DNS round-robin, not 
> active/passive) storage solution? (note: I mean *without* using a full 
> Pacemaker+Corosync stack)

It's probably doable.

The only reason it's not advisable — IMO — is that it's not what we're
doing, and getting help could be pretty hard.

The Samba team has all the CTDB experience. I've poked them — hopefully
they will respond.

Is there some reason you don't want to use Pacemaker and Corosync?

> 
> 2) If the answer to the above question is "yes", is the above above mentioned 
> solution capable of coexisting with oVirt in an hyperconverged setup 
> (assuming replica 3 etc. etc.)?

Off hand I can't think of any reason why not.

> 
> Many thanks in advance to anyone who can answer the above and/or point me to 
> any relevant resources/docs.
> 

https://github.com/linux-ha-storage/storhaug is basis for the Common HA
solution for NFS-Ganesha and Samba that GlusterFS-3.10 will be using.
N.B. It's also based on Pacemaker and Corosync.

-- 

Kaleb
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] [Gluster-devel] Lot of EIO errors in disperse volume

[Xavier Hernandez]

Hi Ram,

On 16/01/17 12:33, Ankireddypalle Reddy wrote:

Xavi,
  Thanks. Is there any other way to map from GFID to path.


The only way I know is to search all files from bricks and lookup for 
the trusted.gfid xattr.



I will look for a way to share the TRACE logs. Easier way might be to add some 
extra logging. I could do that if you could let me know functions in which you 
are interested..


The problem is that I don't know where the problem is. One possibility 
could be to track all return values from all bricks for all writes and 
then identify which ones belong to an inconsistent file.


But if this doesn't reveal anything interesting we'll need to look at 
some other place. And this can be very tedious and slow.


Anyway, what we are looking now is not the source of an EIO, since there 
are two bricks with consistent state and the file should be perfectly 
readable and writable. It's true that there's some problem here and it 
could derive in EIO if one of the healthy bricks degrades, but at least 
this file shouldn't be giving EIO errors for now.


Xavi



Sent on from my iPhone


On Jan 16, 2017, at 6:23 AM, Xavier Hernandez  wrote:

Hi Ram,


On 13/01/17 18:41, Ankireddypalle Reddy wrote:
Xavi,
I enabled TRACE logging. The log grew up to 120GB and could not 
make much out of it. Then I started logging GFID in the code where we were 
seeing errors.

[2017-01-13 17:02:01.761349] I [dict.c:3065:dict_dump_to_log] 
0-glusterfsProd-disperse-0: dict=0x7fa6706bc690 
((trusted.ec.size:0:0:0:0:30:6b:0:0:)(trusted.ec.version:0:0:0:0:0:0:2a:38:0:0:0:0:0:0:2a:38:))
[2017-01-13 17:02:01.761360] I [dict.c:3065:dict_dump_to_log] 
0-glusterfsProd-disperse-0: dict=0x7fa6706bed64 
((trusted.ec.size:0:0:0:0:0:0:0:0:)(trusted.ec.version:0:0:0:0:0:0:0:0:0:0:0:0:0:0:2a:38:))
[2017-01-13 17:02:01.761365] W [MSGID: 122056] 
[ec-combine.c:881:ec_combine_check] 0-glusterfsProd-disperse-0: Mismatching 
xdata in answers of 'LOOKUP'
[2017-01-13 17:02:01.761405] I [dict.c:166:key_value_cmp] 
0-glusterfsProd-disperse-0: 'trusted.ec.size' is different in two dicts (8, 8)
[2017-01-13 17:02:01.761417] I [dict.c:3065:dict_dump_to_log] 
0-glusterfsProd-disperse-0: dict=0x7fa6706bbb14 
((trusted.ec.size:0:0:0:0:30:6b:0:0:)(trusted.ec.version:0:0:0:0:0:0:2a:38:0:0:0:0:0:0:2a:38:))
[2017-01-13 17:02:01.761428] I [dict.c:3065:dict_dump_to_log] 
0-glusterfsProd-disperse-0: dict=0x7fa6706bed64 
((trusted.ec.size:0:0:0:0:0:0:0:0:)(trusted.ec.version:0:0:0:0:0:0:0:0:0:0:0:0:0:0:2a:38:))
[2017-01-13 17:02:01.761433] W [MSGID: 122056] 
[ec-combine.c:881:ec_combine_check] 0-glusterfsProd-disperse-0: Mismatching 
xdata in answers of 'LOOKUP'
[2017-01-13 17:02:01.761442] W [MSGID: 122006] 
[ec-combine.c:214:ec_iatt_combine] 0-glusterfsProd-disperse-0: Failed to 
combine iatt (inode: 11275691004192850514-11275691004192850514, gfid: 
60b990ed-d741-4176-9c7b-4d3a25fb8252  -  60b990ed-d741-4176-9c7b-4d3a25fb8252,  
links: 1-1, uid: 0-0, gid: 0-0, rdev: 0-0,size: 406650880-406683648, mode: 
100775-100775)

The file for which we are seeing this error turns out to be having a GFID of 
60b990ed-d741-4176-9c7b-4d3a25fb8252

Then I tried looking for find out the file with this GFID. It pointed me to 
following path. I was expecting a real file system path from the following 
turorial:
https://gluster.readthedocs.io/en/latest/Troubleshooting/gfid-to-path/


I think this method only works if bricks have the inode cached.



getfattr -n trusted.glusterfs.pathinfo -e text 
/mnt/gfid/.gfid/60b990ed-d741-4176-9c7b-4d3a25fb8252
getfattr: Removing leading '/' from absolute path names
# file: mnt/gfid/.gfid/60b990ed-d741-4176-9c7b-4d3a25fb8252
trusted.glusterfs.pathinfo="( ( 

 
))"

Then I looked for the xatttrs for these files from all the 3 bricks

[root@glusterfs4 glusterfs]# getfattr -d -m . -e hex 
/ws/disk1/ws_brick/.glusterfs/60/b9/60b990ed-d741-4176-9c7b-4d3a25fb8252
getfattr: Removing leading '/' from absolute path names
# file: ws/disk1/ws_brick/.glusterfs/60/b9/60b990ed-d741-4176-9c7b-4d3a25fb8252
trusted.bit-rot.version=0x02005877a8dc00041138
trusted.ec.config=0x080301000200
trusted.ec.size=0x
trusted.ec.version=0x2a38
trusted.gfid=0x60b990edd74141769c7b4d3a25fb8252

[root@glusterfs5 bricks]# getfattr -d -m . -e hex 
/ws/disk1/ws_brick/.glusterfs/60/b9/60b990ed-d741-4176-9c7b-4d3a25fb8252
getfattr: Removing leading '/' from absolute path names
# file: ws/disk1/ws_brick/.glusterfs/60/b9/60b990ed-d741-4176-9c7b-4d3a25fb8252
trusted.bit-rot.version=0x02005877a8dc000c92d0
trusted.ec.config=0x080301000200
trusted.ec.dirty=0x0016
trusted.ec.size=0x306b

Re: [Gluster-users] [Gluster-devel] Lot of EIO errors in disperse volume

[Ankireddypalle Reddy]

Xavi,
  Thanks. Is there any other way to map from GFID to path. I will look 
for a way to share the TRACE logs. Easier way might be to add some extra 
logging. I could do that if you could let me know functions in which you are 
interested..

Sent on from my iPhone

> On Jan 16, 2017, at 6:23 AM, Xavier Hernandez  wrote:
> 
> Hi Ram,
> 
>> On 13/01/17 18:41, Ankireddypalle Reddy wrote:
>> Xavi,
>> I enabled TRACE logging. The log grew up to 120GB and could not 
>> make much out of it. Then I started logging GFID in the code where we were 
>> seeing errors.
>> 
>> [2017-01-13 17:02:01.761349] I [dict.c:3065:dict_dump_to_log] 
>> 0-glusterfsProd-disperse-0: dict=0x7fa6706bc690 
>> ((trusted.ec.size:0:0:0:0:30:6b:0:0:)(trusted.ec.version:0:0:0:0:0:0:2a:38:0:0:0:0:0:0:2a:38:))
>> [2017-01-13 17:02:01.761360] I [dict.c:3065:dict_dump_to_log] 
>> 0-glusterfsProd-disperse-0: dict=0x7fa6706bed64 
>> ((trusted.ec.size:0:0:0:0:0:0:0:0:)(trusted.ec.version:0:0:0:0:0:0:0:0:0:0:0:0:0:0:2a:38:))
>> [2017-01-13 17:02:01.761365] W [MSGID: 122056] 
>> [ec-combine.c:881:ec_combine_check] 0-glusterfsProd-disperse-0: Mismatching 
>> xdata in answers of 'LOOKUP'
>> [2017-01-13 17:02:01.761405] I [dict.c:166:key_value_cmp] 
>> 0-glusterfsProd-disperse-0: 'trusted.ec.size' is different in two dicts (8, 
>> 8)
>> [2017-01-13 17:02:01.761417] I [dict.c:3065:dict_dump_to_log] 
>> 0-glusterfsProd-disperse-0: dict=0x7fa6706bbb14 
>> ((trusted.ec.size:0:0:0:0:30:6b:0:0:)(trusted.ec.version:0:0:0:0:0:0:2a:38:0:0:0:0:0:0:2a:38:))
>> [2017-01-13 17:02:01.761428] I [dict.c:3065:dict_dump_to_log] 
>> 0-glusterfsProd-disperse-0: dict=0x7fa6706bed64 
>> ((trusted.ec.size:0:0:0:0:0:0:0:0:)(trusted.ec.version:0:0:0:0:0:0:0:0:0:0:0:0:0:0:2a:38:))
>> [2017-01-13 17:02:01.761433] W [MSGID: 122056] 
>> [ec-combine.c:881:ec_combine_check] 0-glusterfsProd-disperse-0: Mismatching 
>> xdata in answers of 'LOOKUP'
>> [2017-01-13 17:02:01.761442] W [MSGID: 122006] 
>> [ec-combine.c:214:ec_iatt_combine] 0-glusterfsProd-disperse-0: Failed to 
>> combine iatt (inode: 11275691004192850514-11275691004192850514, gfid: 
>> 60b990ed-d741-4176-9c7b-4d3a25fb8252  -  
>> 60b990ed-d741-4176-9c7b-4d3a25fb8252,  links: 1-1, uid: 0-0, gid: 0-0, rdev: 
>> 0-0,size: 406650880-406683648, mode: 100775-100775)
>> 
>> The file for which we are seeing this error turns out to be having a GFID of 
>> 60b990ed-d741-4176-9c7b-4d3a25fb8252
>> 
>> Then I tried looking for find out the file with this GFID. It pointed me to 
>> following path. I was expecting a real file system path from the following 
>> turorial:
>> https://gluster.readthedocs.io/en/latest/Troubleshooting/gfid-to-path/
> 
> I think this method only works if bricks have the inode cached.
> 
>> 
>> getfattr -n trusted.glusterfs.pathinfo -e text 
>> /mnt/gfid/.gfid/60b990ed-d741-4176-9c7b-4d3a25fb8252
>> getfattr: Removing leading '/' from absolute path names
>> # file: mnt/gfid/.gfid/60b990ed-d741-4176-9c7b-4d3a25fb8252
>> trusted.glusterfs.pathinfo="( 
>> ( 
>> 
>>  
>> ))"
>> 
>> Then I looked for the xatttrs for these files from all the 3 bricks
>> 
>> [root@glusterfs4 glusterfs]# getfattr -d -m . -e hex 
>> /ws/disk1/ws_brick/.glusterfs/60/b9/60b990ed-d741-4176-9c7b-4d3a25fb8252
>> getfattr: Removing leading '/' from absolute path names
>> # file: 
>> ws/disk1/ws_brick/.glusterfs/60/b9/60b990ed-d741-4176-9c7b-4d3a25fb8252
>> trusted.bit-rot.version=0x02005877a8dc00041138
>> trusted.ec.config=0x080301000200
>> trusted.ec.size=0x
>> trusted.ec.version=0x2a38
>> trusted.gfid=0x60b990edd74141769c7b4d3a25fb8252
>> 
>> [root@glusterfs5 bricks]# getfattr -d -m . -e hex 
>> /ws/disk1/ws_brick/.glusterfs/60/b9/60b990ed-d741-4176-9c7b-4d3a25fb8252
>> getfattr: Removing leading '/' from absolute path names
>> # file: 
>> ws/disk1/ws_brick/.glusterfs/60/b9/60b990ed-d741-4176-9c7b-4d3a25fb8252
>> trusted.bit-rot.version=0x02005877a8dc000c92d0
>> trusted.ec.config=0x080301000200
>> trusted.ec.dirty=0x0016
>> trusted.ec.size=0x306b
>> trusted.ec.version=0x2a382a38
>> trusted.gfid=0x60b990edd74141769c7b4d3a25fb8252
>> 
>> [root@glusterfs6 ee]# getfattr -d -m . -e hex 
>> /ws/disk1/ws_brick/.glusterfs/60/b9/60b990ed-d741-4176-9c7b-4d3a25fb8252
>> getfattr: Removing leading '/' from absolute path names
>> # file: 
>> ws/disk1/ws_brick/.glusterfs/60/b9/60b990ed-d741-4176-9c7b-4d3a25fb8252
>> trusted.bit-rot.version=0x02005877a8dc000c9436
>> trusted.ec.config=0x080301000200
>> trusted.ec.dirty=0x0016
>> trusted.ec.size=0x306b
>> 

Re: [Gluster-users] [Gluster-devel] Lot of EIO errors in disperse volume

[Xavier Hernandez]

Hi Ram,

On 13/01/17 18:41, Ankireddypalle Reddy wrote:

Xavi,
 I enabled TRACE logging. The log grew up to 120GB and could not 
make much out of it. Then I started logging GFID in the code where we were 
seeing errors.

[2017-01-13 17:02:01.761349] I [dict.c:3065:dict_dump_to_log] 
0-glusterfsProd-disperse-0: dict=0x7fa6706bc690 
((trusted.ec.size:0:0:0:0:30:6b:0:0:)(trusted.ec.version:0:0:0:0:0:0:2a:38:0:0:0:0:0:0:2a:38:))
[2017-01-13 17:02:01.761360] I [dict.c:3065:dict_dump_to_log] 
0-glusterfsProd-disperse-0: dict=0x7fa6706bed64 
((trusted.ec.size:0:0:0:0:0:0:0:0:)(trusted.ec.version:0:0:0:0:0:0:0:0:0:0:0:0:0:0:2a:38:))
[2017-01-13 17:02:01.761365] W [MSGID: 122056] 
[ec-combine.c:881:ec_combine_check] 0-glusterfsProd-disperse-0: Mismatching 
xdata in answers of 'LOOKUP'
[2017-01-13 17:02:01.761405] I [dict.c:166:key_value_cmp] 
0-glusterfsProd-disperse-0: 'trusted.ec.size' is different in two dicts (8, 8)
[2017-01-13 17:02:01.761417] I [dict.c:3065:dict_dump_to_log] 
0-glusterfsProd-disperse-0: dict=0x7fa6706bbb14 
((trusted.ec.size:0:0:0:0:30:6b:0:0:)(trusted.ec.version:0:0:0:0:0:0:2a:38:0:0:0:0:0:0:2a:38:))
[2017-01-13 17:02:01.761428] I [dict.c:3065:dict_dump_to_log] 
0-glusterfsProd-disperse-0: dict=0x7fa6706bed64 
((trusted.ec.size:0:0:0:0:0:0:0:0:)(trusted.ec.version:0:0:0:0:0:0:0:0:0:0:0:0:0:0:2a:38:))
[2017-01-13 17:02:01.761433] W [MSGID: 122056] 
[ec-combine.c:881:ec_combine_check] 0-glusterfsProd-disperse-0: Mismatching 
xdata in answers of 'LOOKUP'
[2017-01-13 17:02:01.761442] W [MSGID: 122006] 
[ec-combine.c:214:ec_iatt_combine] 0-glusterfsProd-disperse-0: Failed to 
combine iatt (inode: 11275691004192850514-11275691004192850514, gfid: 
60b990ed-d741-4176-9c7b-4d3a25fb8252  -  60b990ed-d741-4176-9c7b-4d3a25fb8252,  
links: 1-1, uid: 0-0, gid: 0-0, rdev: 0-0,size: 406650880-406683648, mode: 
100775-100775)

The file for which we are seeing this error turns out to be having a GFID of 
60b990ed-d741-4176-9c7b-4d3a25fb8252

Then I tried looking for find out the file with this GFID. It pointed me to 
following path. I was expecting a real file system path from the following 
turorial:
https://gluster.readthedocs.io/en/latest/Troubleshooting/gfid-to-path/


I think this method only works if bricks have the inode cached.



getfattr -n trusted.glusterfs.pathinfo -e text 
/mnt/gfid/.gfid/60b990ed-d741-4176-9c7b-4d3a25fb8252
getfattr: Removing leading '/' from absolute path names
# file: mnt/gfid/.gfid/60b990ed-d741-4176-9c7b-4d3a25fb8252
trusted.glusterfs.pathinfo="( ( 

 
))"

Then I looked for the xatttrs for these files from all the 3 bricks

[root@glusterfs4 glusterfs]# getfattr -d -m . -e hex 
/ws/disk1/ws_brick/.glusterfs/60/b9/60b990ed-d741-4176-9c7b-4d3a25fb8252
getfattr: Removing leading '/' from absolute path names
# file: ws/disk1/ws_brick/.glusterfs/60/b9/60b990ed-d741-4176-9c7b-4d3a25fb8252
trusted.bit-rot.version=0x02005877a8dc00041138
trusted.ec.config=0x080301000200
trusted.ec.size=0x
trusted.ec.version=0x2a38
trusted.gfid=0x60b990edd74141769c7b4d3a25fb8252

[root@glusterfs5 bricks]# getfattr -d -m . -e hex 
/ws/disk1/ws_brick/.glusterfs/60/b9/60b990ed-d741-4176-9c7b-4d3a25fb8252
getfattr: Removing leading '/' from absolute path names
# file: ws/disk1/ws_brick/.glusterfs/60/b9/60b990ed-d741-4176-9c7b-4d3a25fb8252
trusted.bit-rot.version=0x02005877a8dc000c92d0
trusted.ec.config=0x080301000200
trusted.ec.dirty=0x0016
trusted.ec.size=0x306b
trusted.ec.version=0x2a382a38
trusted.gfid=0x60b990edd74141769c7b4d3a25fb8252

[root@glusterfs6 ee]# getfattr -d -m . -e hex 
/ws/disk1/ws_brick/.glusterfs/60/b9/60b990ed-d741-4176-9c7b-4d3a25fb8252
getfattr: Removing leading '/' from absolute path names
# file: ws/disk1/ws_brick/.glusterfs/60/b9/60b990ed-d741-4176-9c7b-4d3a25fb8252
trusted.bit-rot.version=0x02005877a8dc000c9436
trusted.ec.config=0x080301000200
trusted.ec.dirty=0x0016
trusted.ec.size=0x306b
trusted.ec.version=0x2a382a38
trusted.gfid=0x60b990edd74141769c7b4d3a25fb8252

It turns out that the size and version in fact does not match for one of the 
files.


It seems as if the brick on glusterfs4 didn't receive any write request 
(or they failed for some reason). Do you still have the trace log ? is 
there any way I could download it ?


Xavi



Thanks and Regards,
Ram

-Original Message-
From: gluster-devel-boun...@gluster.org 
[mailto:gluster-devel-boun...@gluster.org] On Behalf Of Ankireddypalle Reddy
Sent: Friday, January 13, 2017 4:17 AM
To: Xavier Hernandez
Cc: gluster-users@gluster.org; Gluster Devel 

Re: [Gluster-users] NFS service dying

[Niels de Vos]

On Fri, Jan 13, 2017 at 06:43:33PM +0100, Giuseppe Ragusa wrote:
> On Fri, Jan 13, 2017, at 12:39, Niels de Vos wrote:
> > On Wed, Jan 11, 2017 at 11:58:29AM -0700, Paul Allen wrote:
> > > I'm running into an issue where the gluster nfs service keeps dying on a
> > > new cluster I have setup recently. We've been using Gluster on several
> > > other clusters now for about a year or so and I have never seen this
> > > issue before, nor have I been able to find anything remotely similar to
> > > it while searching on-line. I initially was using the latest version in
> > > the Gluster Debian repository for Jessie, 3.9.0-1, and then I tried
> > > using the next one down, 3.8.7-1. Both behave the same for me.
> > > 
> > > What I was seeing was after a while the nfs service on the NAS server
> > > would suddenly die after a number of processes had run on the app server
> > > I had connected to the new NAS servers for testing (we're upgrading the
> > > NAS servers for this cluster to newer hardware and expanded storage, the
> > > current production NAS servers are using nfs-kernel-server with no type
> > > of clustering of the data). I checked the logs but all it showed me was
> > > something that looked like a stack trace in the nfs.log and the
> > > glustershd.log showed the nfs service disconnecting. I turned on
> > > debugging but it didn't give me a whole lot more, and certainly nothing
> > > that helps me identify the source of my issue. It is pretty consistent
> > > in dying shortly after I mount the file system on the servers and start
> > > testing, usually within 15-30 minutes. But if I have nothing using the
> > > file system, mounted or no, the service stays running for days. I tried
> > > mounting it using the gluster client, and it works fine, but I can;t use
> > > that due to the performance penalty, it slows the websites down by a few
> > > seconds at a minimum.
> > 
> > This seems to be related to the NLM protocol that Gluster/NFS provides.
> > Earlier this week one of our Red Hat quality engineers also reported
> > this (or a very similar) problem.
> > 
> > https://bugzilla.redhat.com/show_bug.cgi?id=1411344
> > 
> > At the moment I suspect that this is related to re-connects of some
> > kind, but I have not been able to identify the cause sufficiently to be
> > sure. This definitely is a coding problem in Gluster/NFS, but the more I
> > look at the NLM implementation, the more potential issues I see with it.
> 
> Should we assume that, with the complete removal of Gluster/NFS
> already on the horizon, debugging and fixing NLM (even if only for the
> more common, reported crash cases) would be an extremely low-priority
> task? ;-)

Crashes are expected to be fixed, if they happen in 'normal'
circumstances. It is not an extremely low priority, but neither is it
very high. I'm looking into it, but am also working on other higher
priority things. Maybe I have a fix by the end of the week, depending on
whatever distracts me from working on it.


> Would it be possible for someone to simply check whether the crashes
> happen also on the 3.7.9-12 codebase used in latest RHGS 3.1.3?
> My cluster is already at 3.7.12 feature level (and using it), so I
> suppose I could not easily downgrade.
> Since Red Hat QA found a similar problem while testing the 3.8.4-10
> codebase in RHGS 3.2.0, we could trace the problem back to post-3.7.9
> developments, if RHGS 3.1.3 is immune.

I expect to see this problem in older versions too. There has been very
little change to the NLM code in Gluster. It is possible that the Linux
kernel was updated and the NFS-client behaves a little different,
exposing this problem just now, or the testing has changed...

> > If the workload does not require locking operations, you may be able to
> > work around the problem by mounting with "-o nolock". Depending on the
> > application, this can be safe or cause data corruption...
> 
> If I'm not mistaken, typical NFS uses such as YUM repositories and
> home directories would be barred (I think that SQLite needs locking
> and both createrepo and firefox use SQLite, right?).

Only if access to the same content is happening from different
NFS-clients. The case of createrepo is normally safe, it generates new
files and renames them over the older ones.

SQLite needs locking if multiple processes read/write the file. By
default SQLite uses in-memory (SHM) locks and does not try to use file
locks at all. At least that was the behaviour months (or years?) back.
This causes troubles for any SQLite database stored on a network
filesystem and accessed from different clients.


> > An other alternative is to use NFS-Ganesha instead of Gluster/NFS.
> > Ganesha is more mature than Gluster/NFS and is more actively developed.
> > Gluster/NFS is being deprecated in favour of NFS-Ganesha.
> 
> Pure storage infrastructure uses should be migratable, I suppose, but
> extended testing and a suitable maintenance window (a rolling live
> migration from 

[Gluster-users] Configuration validation on 32 nodes-gluster: expecting 64 TB but I have only 35 TB

[Fedele Stabile]

Hello,
I would ask to you if this configuration is valid:
32 nodes cluster with CentOS 6.6 kernel 2.6.32-504.8.1.el6.x86_64
glusterf version is 3.7.8
I have infiniband working at 40 GBs  
and on each node I have configured two bricks: 
one on a 1 TB disk with a xfs filesystem dedicated to gluster
the other with a 1 TB disk with a ext4 filesystem also dedicated to
gluster
I have configured only one big replicated volume made of 64 bricks
This is the volume configuration in summary:
Volume Name: scratch
Type: Distribute
Volume ID: fc6f18b6-a06c-4fdf-ac08-23e9b4f8053e
Status: Started
Number of Bricks: 64
Transport-type: tcp,rdma
Bricks:
Brick1: ib-wn001:/bricks/brick1/gscratch0
Brick2: ib-wn002:/bricks/brick1/gscratch0

Brick31: ib-wn032:/bricks/brick1/gscratch0
Brick32: ib-wn030:/bricks/brick1/gscratch0
Brick33: ib-wn001:/bricks/brick2/gsp2

Brick64: ib-wn032:/bricks/brick2/gsp2
Options Reconfigured:
features.inode-quota: off
features.quota: off
features.scrub-freq: daily
features.scrub: Inactive
features.bitrot: off
config.transport: tcp,rdma
performance.readdir-ahead: on
nfs.disable: true

It seems working well, but the output of df -h |grep scratch
gives this output:
ib-wn001:/scratch.rdma
   35T   13T   22T  36% /scratch

But I expect a volume of 64 TB

Can you help me to understand?

Volume is currently used but the size is different by that expected...

Thank you to all
Fedele Stabile

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users