Re: [Gluster-users] "Another Transaction is in progres..."

[Krist van Besien]

Thanks for the suggestion, this solved it for us, and we probably found the
cause as well. We had performance co-pilot running and it was continously
enabling profiling on volumes...
We found the reference to the node that had the lock, and restarted
glusterd on that node, and all went well from there on.

Krist


On 31 May 2017 at 15:56, Vijay Bellur  wrote:

>
>
> On Wed, May 31, 2017 at 9:32 AM, Krist van Besien 
> wrote:
>
>> Hi all,
>>
>> I am trying to do trivial things, like setting quota, or just querying
>> the status and keep getting
>>
>> "Another transaction is in progres for "
>>
>> These messages pop up, then disappear for a while, then pop up again...
>>
>> What do these messages mean? How do I figure out which "transaction" is
>> meant here, and what do I do about it?
>>
>
>
> This message usually means that a different gluster command is being
> executed in the cluster. Most gluster commands are serialized by a cluster
> wide lock. Upon not being able to acquire the cluster lock, this message is
> displayed.
>
> You can check /var/log/glusterfs/cmd_history.log on all storage nodes to
> observe what other commands are in progress at the time of getting this
> error message. Are you per chance using oVirt to manage Gluster? oVirt
> periodically does a "gluster volume status" to determine the volume health
> and that can conflict with other commands being executed.
>
> Regards,
> Vijay
>
>


-- 
Vriendelijke Groet |  Best Regards | Freundliche Grüße | Cordialement
--
Krist van Besien | Senior Architect | Red Hat EMEA Cloud Practice | RHCE |
RHCSA Open Stack
@: kr...@redhat.com | M: +41-79-5936260
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Disconnected gluster node things it is still connected...

[Krist van Besien]

Hi all,

Trying to do some availability testing.

We have three nodes: node1, node2, node3. Volumes are all replica 2, across
all three nodes.

As a test we disconnected node1, buy removing the vlan tag for that host on
the switch it is connected to. As a result node2 and node3 now show node1
in disconnected status, and show the volumes as degraded.
This is ecpected.

However logging in to node1 (via the ilo, as there is no network) showed
that this node still though it was connected to node2 and node3, even
though it could no longer communicate with it.

Also it did keep its bricks up...

This is not as expected. What I expected is that node1 detects it is no
longer part of a quorum, and takes all its bricks down.

So what did we miss?

Krist




-- 
Vriendelijke Groet |  Best Regards | Freundliche Grüße | Cordialement
--
Krist van Besien | Senior Architect | Red Hat EMEA Cloud Practice | RHCE |
RHCSA Open Stack
@: kr...@redhat.com | M: +41-79-5936260
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] FW: ATTN: nbalacha IRC - Gluster - BlackoutWNCT requested info for 0byte file issue

[Joshua Coyle]

Hey Nithya,

root@PB-WA-AA-00-A:/# glusterfs -V
glusterfs 3.10.1
Repository revision: git://git.gluster.org/glusterfs.git
Copyright (c) 2006-2016 Red Hat, Inc. 
GlusterFS comes with ABSOLUTELY NO WARRANTY.
It is licensed to you under your choice of the GNU Lesser
General Public License, version 3 or any later version (LGPLv3
or later), or the GNU General Public License, version 2 (GPLv2),
in all cases as published by the Free Software Foundation.

We have also set the performance.parallel-readdir option to off, based on the 
information provided in the Bugzilla report you included.
We’re currently pending confirmation as to if this has resolved the issue for 
the short term. I’ll update you with this once I’ve received confirmation.

We’re also planning an upgrade to 3.10.2 shortly.

Regards,
[Probax]

Joshua Coyle


P

1300 885 117

E

joshua.co...@probax.io

W

probax.io


A

QV1 Perth,


Level 33/250 St Georges Tce,


Perth WA 6000



smart safe local cloud




From: Nithya Balachandran [mailto:nbala...@redhat.com]
Sent: Wednesday, May 31, 2017 6:20 PM
To: Joshua Coyle 
Cc: gluster-users@gluster.org; Gurusiddaiah, Poornima ; 
Raghavendra Gowdappa ; Ravishankar Narayanankutty 

Subject: Re: [Gluster-users] FW: ATTN: nbalacha IRC - Gluster - BlackoutWNCT 
requested info for 0byte file issue

CCing Ravi (arbiter) , Poornima and Raghavendra (parallel readdir)

Hi Joshua,

I had a quick look at the files you sent across.  To summarize the issue, you 
see empty linkto files on the mount point.

From the logs I see  that parallel readdir is enabled for this volume:

performance.readdir-ahead: on
performance.parallel-readdir: on

There was a similar issue reported earlier [1] and fixed as part of [2]. Which 
version of gluster are you running? Poornima and Raghavendra, can you take a 
look and see if it is the same issue?



An interesting observation - the value of the  trusted.glusterfs.dht.linkto 
xattr is different on the one of the bricks of the replica set for some of the 
files(Ravi, any idea why ?)

-T 2 root root 0 May 18 16:49 
/arbiterAA01/gvAA01/brick1/vaultAA01/support-couplers/CPL-SBS01 
(CPL-SBS01)/D_VOL-b002-i1873-cd.md5

root@PB-WA-AA-00-A:/# getfattr -e hex -m . -d 
/arbiterAA01/gvAA01/brick1/vaultAA01/support-couplers/CPL-SBS01\ 
\(CPL-SBS01\)/D_VOL-b002-i1873-cd.md5
getfattr: Removing leading '/' from absolute path names
# file: arbiterAA01/gvAA01/brick1/vaultAA01/support-couplers/CPL-SBS01 
(CPL-SBS01)/D_VOL-b002-i1873-cd.md5
trusted.gfid=0xfc31c380697a4c70b6af1af35822e519
trusted.glusterfs.dht.linkto=0x6776414130312d726561646469722d61686561642d3100

This value is "gvAA01-readdir-ahead-1"



-T 2 root root 0 May 18 16:49 
/brick1/gvAA01/brick/vaultAA01/support-couplers/CPL-SBS01 
(CPL-SBS01)/D_VOL-b002-i1873-cd.md5

root@PB-WA-AA-01-B:/# getfattr -e hex -m . -d 
/brick1/gvAA01/brick/vaultAA01/support-couplers/CPL-SBS01\ 
\(CPL-SBS01\)/D_VOL-b002-i1873-cd.md5
getfattr: Removing leading '/' from absolute path names
# file: brick1/gvAA01/brick/vaultAA01/support-couplers/CPL-SBS01 
(CPL-SBS01)/D_VOL-b002-i1873-cd.md5
trusted.gfid=0xfc31c380697a4c70b6af1af35822e519
trusted.glusterfs.dht.linkto=0x6776414130312d726561646469722d61686561642d3100

This value is "gvAA01-readdir-ahead-1"



-T 2 root root 0 May 30 14:25 
/brick1/gvAA01/brick/vaultAA01/support-couplers/CPL-SBS01 
(CPL-SBS01)/D_VOL-b002-i1873-cd.md5

root@PB-WA-AA-02-B:/# getfattr -e hex -m . -d 
/brick1/gvAA01/brick/vaultAA01/support-couplers/CPL-SBS01\ 
\(CPL-SBS01\)/D_VOL-b002-i1873-cd.md5
getfattr: Removing leading '/' from absolute path names
# file: brick1/gvAA01/brick/vaultAA01/support-couplers/CPL-SBS01 
(CPL-SBS01)/D_VOL-b002-i1873-cd.md5
trusted.afr.dirty=0x
trusted.bit-rot.version=0x020058df0ab7000710ee
trusted.gfid=0xfc31c380697a4c70b6af1af35822e519
trusted.glusterfs.dht.linkto=0x6776414130312d7265706c69636174652d3100

This value is "gvAA01-replicate-1"


[1] http://lists.gluster.org/pipermail/gluster-users/2017-March/030254.html
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1435942

On 30 May 2017 at 14:20, Joshua Coyle 
mailto:joshua.co...@probax.io>> wrote:


[Probax]

Joshua Coyle


P

1300 885 117

E

joshua.co...@probax.io

W

probax.io


A

QV1 Perth,


Level 33/250 St Georges Tce,


Perth WA 6000



smart safe local cloud




From: Joshua Coyle
Sent: Tuesday, May 30, 2017 3:27 PM
To: 'gluster-de...@gluster.org' 
mailto:gluster-de...@gluster.org>>
Subject: ATTN: nbalacha IRC - Gluster - BlackoutWNCT requested info for 0byte 
file issue

Hey nbalacha,

Please find the requested information attached for the issue being discussed in 
IRC.

If you need any further info, please let me know.

Thanks,

BlackoutWNCT
[Probax]

Joshua Coyle


P

1300 885 117

E

joshua.co...@probax.io

[Gluster-users] Glusterfs 3.10.3 has been tagged

[Raghavendra Talur]

Glusterfs 3.10.3 has been tagged.

Packages for the various distributions will be available in a few days,
and with that a more formal release announcement will be made.

- Tagged code: https://github.com/gluster/glusterfs/tree/v3.10.3
- Release notes:
https://github.com/gluster/glusterfs/blob/release-3.10/doc/release-notes/3.10.3.md

Thanks,
Raghavendra Talur

NOTE: Tracker bug for 3.10.3 will be closed in a couple of days and
tracker for 3.10.4 will be opened, and an announcement for 3.10.4 will
be sent with the details
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] URGENT: Update issues from 3.6.6 to 3.10.2 Accessing files via samba come up with permission denied

[Diego Remolina]

The vfs gluster plugin for samba is linked against libglusterfs.so.0
which is provided by glusterfs-libs-3.10.2-1.el7.x86_64, please see
below.

# ldd /usr/lib64/samba/vfs/glusterfs.so | grep glus
   libglusterfs.so.0 => /lib64/libglusterfs.so.0 (0x7f61da858000)

]# yum provides /lib64/libglusterfs.so.0
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: centos.vwtonline.net
* extras: mirror.cs.vt.edu
* updates: centosv.centos.org
glusterfs-libs-3.10.2-1.el7.x86_64 : GlusterFS common libraries
Repo: @centos-gluster310
Matched from:
Filename: /lib64/libglusterfs.so.0



On Wed, May 31, 2017 at 12:39 PM, Diego Remolina  wrote:
> Samba is running in the same machine as glusterd. The machines were
> rebooted after the upgrades and samba has been restarted a few times.
>
> # rpm -qa | grep gluster
> glusterfs-client-xlators-3.10.2-1.el7.x86_64
> glusterfs-server-3.10.2-1.el7.x86_64
> glusterfs-api-3.10.2-1.el7.x86_64
> glusterfs-3.10.2-1.el7.x86_64
> glusterfs-cli-3.10.2-1.el7.x86_64
> centos-release-gluster310-1.0-1.el7.centos.noarch
> samba-vfs-glusterfs-4.4.4-14.el7_3.x86_64
> glusterfs-fuse-3.10.2-1.el7.x86_64
> glusterfs-libs-3.10.2-1.el7.x86_64
> glusterfs-rdma-3.10.2-1.el7.x86_64
>
> # rpm -qa | grep samba
> samba-common-libs-4.4.4-14.el7_3.x86_64
> samba-common-tools-4.4.4-14.el7_3.x86_64
> samba-libs-4.4.4-14.el7_3.x86_64
> samba-4.4.4-14.el7_3.x86_64
> samba-client-libs-4.4.4-14.el7_3.x86_64
> samba-vfs-glusterfs-4.4.4-14.el7_3.x86_64
> samba-common-4.4.4-14.el7_3.noarch
>
> # cat /etc/redhat-release
> CentOS Linux release 7.3.1611 (Core)
>
> I also raised the op version.
>
> # gluster volume get all cluster.op-version
> Option  Value
> --  -
> cluster.op-version  31000
>
> # gluster volume get all cluster.max-op-version
> Option  Value
> --  -
> cluster.max-op-version  31000
>
> On Wed, May 31, 2017 at 12:21 PM, Raghavendra Talur  wrote:
>> Also, please attach your smb.conf. You can directly attach in the list
>> and need not have a google drive link.
>>
>>
>> On Wed, May 31, 2017 at 9:37 PM, Raghavendra Talur  wrote:
>>> Diego,
>>>
>>> I see that Samba is still using 3.6.6 Gluster. Is it possible you did
>>> not restart smb after upgrading gluster(if samba is on same machine as
>>> Gluster) or if you forgot to update Gluster client packages on Samba
>>> node?
>>>
>>> On Wed, May 31, 2017 at 9:04 PM, Diego Remolina  wrote:
 Please download the log file from this link:

 https://drive.google.com/open?id=0B8EAPWIe4oyKN0h0X1pZVkRWVEU

 Let me know if you need any other log files.

 Diego

 On Wed, May 31, 2017 at 11:19 AM, Raghavendra Talur  
 wrote:
> If possible please share the glusterfs-* log files from /var/log/samba.
>
> This might be because of cluster.lookup-optimize. Adding Poornima and
> Raghavendra Gowdappa to help with this.
>
>
> On Wed, May 31, 2017 at 1:03 AM, Diego Remolina  
> wrote:
>> This is a bit puzzling, not sure what difference it would make, but:
>>
>> 1. Try to open file that has a problem, ie. MyRevitFile.rvt
>> Revit opens and shows a window that says access denied.
>>
>> 2. Rename file, i.e from windows explorer right click, rename to
>> MyRevitFile2.rvt
>>
>> 3. Without opening the file, rename file back to the original, i.e
>> MyRevitFile.rvt
>>
>> 4. Double click on file and now it will open just fine without the
>> Access Denied error.
>>
>> Any explanation for this? Could the rename operation be forcing or
>> updating some attributes that then allow the file to open?
>>
>> Diego
>>
>> On Tue, May 30, 2017 at 10:57 AM, Diego Remolina  
>> wrote:
>>> This is what I see in the logs set from smb.conf via line ->
>>> glusterfs:logfile = /var/log/samba/glusterfs-projects.log
>>>
>>> [2017-05-30 14:52:31.051524] E [MSGID: 123001]
>>> [io-cache.c:564:ioc_open_cbk] 0-export-io-cache: inode context is NULL
>>> (a97bc9bb-68cf-4a69-aef7-39766b323c14) [Invalid argument]
>>> [2017-05-30 14:52:31.241850] W [MSGID: 114031]
>>> [client-rpc-fops.c:1100:client3_3_getxattr_cbk] 0-export-client-0:
>>> remote operation failed. Path:
>>> /projects/INACTIVE/WESTCOAST/Automotive/Acura/AS-Acura of Richmond/02
>>> DRAWINGS/02 ARCH/CrownAcura-SD02-ArchModel.rvt (a97bc9bb-68cf-4a69-
>>> aef7-39766b323c14). Key: glusterfs.get_real_filename:desktop.ini [Not
>>> a directory]
>>> [2017-05-30 14:52:31.242956] W [MSGID: 114031]
>>> [client-rpc-fops.c:1100:client3_3_getxattr_cbk] 0-export-client-1:
>>> remote operation failed. Path:
>>> /projects/INACTIVE/WESTCOAST/Automotive/Acura/AS-Acura of Richmond/02
>>> DRAWINGS/02 ARCH/CrownAcura-SD02-ArchModel.rvt (

Re: [Gluster-users] URGENT: Update issues from 3.6.6 to 3.10.2 Accessing files via samba come up with permission denied

[Diego Remolina]

Samba is running in the same machine as glusterd. The machines were
rebooted after the upgrades and samba has been restarted a few times.

# rpm -qa | grep gluster
glusterfs-client-xlators-3.10.2-1.el7.x86_64
glusterfs-server-3.10.2-1.el7.x86_64
glusterfs-api-3.10.2-1.el7.x86_64
glusterfs-3.10.2-1.el7.x86_64
glusterfs-cli-3.10.2-1.el7.x86_64
centos-release-gluster310-1.0-1.el7.centos.noarch
samba-vfs-glusterfs-4.4.4-14.el7_3.x86_64
glusterfs-fuse-3.10.2-1.el7.x86_64
glusterfs-libs-3.10.2-1.el7.x86_64
glusterfs-rdma-3.10.2-1.el7.x86_64

# rpm -qa | grep samba
samba-common-libs-4.4.4-14.el7_3.x86_64
samba-common-tools-4.4.4-14.el7_3.x86_64
samba-libs-4.4.4-14.el7_3.x86_64
samba-4.4.4-14.el7_3.x86_64
samba-client-libs-4.4.4-14.el7_3.x86_64
samba-vfs-glusterfs-4.4.4-14.el7_3.x86_64
samba-common-4.4.4-14.el7_3.noarch

# cat /etc/redhat-release
CentOS Linux release 7.3.1611 (Core)

I also raised the op version.

# gluster volume get all cluster.op-version
Option  Value
--  -
cluster.op-version  31000

# gluster volume get all cluster.max-op-version
Option  Value
--  -
cluster.max-op-version  31000

On Wed, May 31, 2017 at 12:21 PM, Raghavendra Talur  wrote:
> Also, please attach your smb.conf. You can directly attach in the list
> and need not have a google drive link.
>
>
> On Wed, May 31, 2017 at 9:37 PM, Raghavendra Talur  wrote:
>> Diego,
>>
>> I see that Samba is still using 3.6.6 Gluster. Is it possible you did
>> not restart smb after upgrading gluster(if samba is on same machine as
>> Gluster) or if you forgot to update Gluster client packages on Samba
>> node?
>>
>> On Wed, May 31, 2017 at 9:04 PM, Diego Remolina  wrote:
>>> Please download the log file from this link:
>>>
>>> https://drive.google.com/open?id=0B8EAPWIe4oyKN0h0X1pZVkRWVEU
>>>
>>> Let me know if you need any other log files.
>>>
>>> Diego
>>>
>>> On Wed, May 31, 2017 at 11:19 AM, Raghavendra Talur  
>>> wrote:
 If possible please share the glusterfs-* log files from /var/log/samba.

 This might be because of cluster.lookup-optimize. Adding Poornima and
 Raghavendra Gowdappa to help with this.


 On Wed, May 31, 2017 at 1:03 AM, Diego Remolina  wrote:
> This is a bit puzzling, not sure what difference it would make, but:
>
> 1. Try to open file that has a problem, ie. MyRevitFile.rvt
> Revit opens and shows a window that says access denied.
>
> 2. Rename file, i.e from windows explorer right click, rename to
> MyRevitFile2.rvt
>
> 3. Without opening the file, rename file back to the original, i.e
> MyRevitFile.rvt
>
> 4. Double click on file and now it will open just fine without the
> Access Denied error.
>
> Any explanation for this? Could the rename operation be forcing or
> updating some attributes that then allow the file to open?
>
> Diego
>
> On Tue, May 30, 2017 at 10:57 AM, Diego Remolina  
> wrote:
>> This is what I see in the logs set from smb.conf via line ->
>> glusterfs:logfile = /var/log/samba/glusterfs-projects.log
>>
>> [2017-05-30 14:52:31.051524] E [MSGID: 123001]
>> [io-cache.c:564:ioc_open_cbk] 0-export-io-cache: inode context is NULL
>> (a97bc9bb-68cf-4a69-aef7-39766b323c14) [Invalid argument]
>> [2017-05-30 14:52:31.241850] W [MSGID: 114031]
>> [client-rpc-fops.c:1100:client3_3_getxattr_cbk] 0-export-client-0:
>> remote operation failed. Path:
>> /projects/INACTIVE/WESTCOAST/Automotive/Acura/AS-Acura of Richmond/02
>> DRAWINGS/02 ARCH/CrownAcura-SD02-ArchModel.rvt (a97bc9bb-68cf-4a69-
>> aef7-39766b323c14). Key: glusterfs.get_real_filename:desktop.ini [Not
>> a directory]
>> [2017-05-30 14:52:31.242956] W [MSGID: 114031]
>> [client-rpc-fops.c:1100:client3_3_getxattr_cbk] 0-export-client-1:
>> remote operation failed. Path:
>> /projects/INACTIVE/WESTCOAST/Automotive/Acura/AS-Acura of Richmond/02
>> DRAWINGS/02 ARCH/CrownAcura-SD02-ArchModel.rvt (a97bc9bb-68cf-4a69-
>> aef7-39766b323c14). Key: glusterfs.get_real_filename:desktop.ini [Not
>> a directory]
>>
>>
>> On Tue, May 30, 2017 at 10:37 AM, Diego Remolina  
>> wrote:
>>> Hi,
>>>
>>> Over the weekend we updated a two server glusterfs 3.6.6 install to
>>> 3.10.2 We also updated samba and samba-vfs to the latest in CentOS. I
>>> enabled several of the newer caching features from gluster 3.9 for
>>> small file performance and samba, and we now seem to have some issues
>>> with accessing files from glusterfs. When users try to access some
>>> files, they get a Permission denied message. This seems to be only via
>>> samba as I am able to su - username and then do strings on the file.
>>>
>>> [root@ysmha02 gluster-backups]# rpm -qa | grep gluster

Re: [Gluster-users] [Gluster-devel] Empty info file preventing glusterd from starting

[Niels de Vos]

On Wed, May 31, 2017 at 04:08:06PM +0530, ABHISHEK PALIWAL wrote:
> We are using 3.7.6 and on link https://review.gluster.org/#/c/16279 status
> is "can't merge"

Note that 3.7.x will not get any updates anymore. We currently maintain
version 3.8.x, 3.10.x and 3.11.x. See the release schedele for more
details:
  https://www.gluster.org/community/release-schedule/

Niels


> 
> On Wed, May 31, 2017 at 4:05 PM, Amar Tumballi  wrote:
> 
> > This is already part of 3.11.0 release?
> >
> > On Wed, May 31, 2017 at 3:47 PM, ABHISHEK PALIWAL  > > wrote:
> >
> >> Hi Atin,
> >>
> >> Could you please let us know any time plan for deliver of this patch.
> >>
> >> Regards,
> >> Abhishek
> >>
> >> On Tue, May 9, 2017 at 6:37 PM, ABHISHEK PALIWAL  >> > wrote:
> >>
> >>> Actually it is very risky if it will reproduce in production thats is
> >>> why I said it is on high priority as want to resolve it before production.
> >>>
> >>> On Tue, May 9, 2017 at 6:20 PM, Atin Mukherjee 
> >>> wrote:
> >>>
> 
> 
>  On Tue, May 9, 2017 at 6:10 PM, ABHISHEK PALIWAL <
>  abhishpali...@gmail.com> wrote:
> 
> > Hi Atin,
> >
> > Thanks for your reply.
> >
> >
> > Its urgent because this error is very rarely reproducible we have seen
> > this 2 3 times in our system till now.
> >
> > We have delivery in near future so that we want it asap. Please try to
> > review it internally.
> >
> 
>  I don't think your statements justified the reason of urgency as (a)
>  you have mentioned it to be *rarely* reproducible and (b) I am still
>  waiting for a real use case where glusterd will go through multiple
>  restarts in a loop?
> 
> 
> > Regards,
> > Abhishek
> >
> > On Tue, May 9, 2017 at 5:58 PM, Atin Mukherjee 
> > wrote:
> >
> >>
> >>
> >> On Tue, May 9, 2017 at 3:37 PM, ABHISHEK PALIWAL <
> >> abhishpali...@gmail.com> wrote:
> >>
> >>> + Muthu-vingeshwaran
> >>>
> >>> On Tue, May 9, 2017 at 11:30 AM, ABHISHEK PALIWAL <
> >>> abhishpali...@gmail.com> wrote:
> >>>
>  Hi Atin/Team,
> 
>  We are using gluster-3.7.6 with setup of two brick and while
>  restart of system I have seen that the glusterd daemon is getting 
>  failed
>  from start.
> 
> 
>  At the time of analyzing the logs from etc-glusterfs...log file
>  I have received the below logs
> 
> 
>  [2017-05-06 03:33:39.798087] I [MSGID: 100030]
>  [glusterfsd.c:2348:main] 0-/usr/sbin/glusterd: Started running
>  /usr/sbin/glusterd version 3.7.6 (args: /usr/sbin/glusterd -p
>  /var/run/glusterd.pid --log-level INFO)
>  [2017-05-06 03:33:39.807859] I [MSGID: 106478]
>  [glusterd.c:1350:init] 0-management: Maximum allowed open file 
>  descriptors
>  set to 65536
>  [2017-05-06 03:33:39.807974] I [MSGID: 106479]
>  [glusterd.c:1399:init] 0-management: Using /system/glusterd as 
>  working
>  directory
>  [2017-05-06 03:33:39.826833] I [MSGID: 106513]
>  [glusterd-store.c:2047:glusterd_restore_op_version] 0-glusterd:
>  retrieved op-version: 30706
>  [2017-05-06 03:33:39.827515] E [MSGID: 106206]
>  [glusterd-store.c:2562:glusterd_store_update_volinfo]
>  0-management: Failed to get next store iter
>  [2017-05-06 03:33:39.827563] E [MSGID: 106207]
>  [glusterd-store.c:2844:glusterd_store_retrieve_volume]
>  0-management: Failed to update volinfo for c_glusterfs volume
>  [2017-05-06 03:33:39.827625] E [MSGID: 106201]
>  [glusterd-store.c:3042:glusterd_store_retrieve_volumes]
>  0-management: Unable to restore volume: c_glusterfs
>  [2017-05-06 03:33:39.827722] E [MSGID: 101019]
>  [xlator.c:428:xlator_init] 0-management: Initialization of volume
>  'management' failed, review your volfile again
>  [2017-05-06 03:33:39.827762] E [graph.c:322:glusterfs_graph_init]
>  0-management: initializing translator failed
>  [2017-05-06 03:33:39.827784] E [graph.c:661:glusterfs_graph_activate]
>  0-graph: init failed
>  [2017-05-06 03:33:39.828396] W [glusterfsd.c:1238:cleanup_and_exit]
>  (-->/usr/sbin/glusterd(glusterfs_volumes_init-0x1b0b8)
>  [0x1000a648] -->/usr/sbin/glusterd(glusterfs_process_volfp-0x1b210)
>  [0x1000a4d8] -->/usr/sbin/glusterd(cleanup_and_exit-0x1beac)
>  [0x100097ac] ) 0-: received signum (0), shutting down
> 
> >>>
> >> Abhishek,
> >>
> >> This patch needs to be thoroughly reviewed to ensure that it doesn't
> >> cause any regression given this touches on the core store management
> >> functionality of glusterd. AFAICT, we get into an empty info file only 
> >> when
> >> volume set operation is executed and i

Re: [Gluster-users] URGENT: Update issues from 3.6.6 to 3.10.2 Accessing files via samba come up with permission denied

[Raghavendra Talur]

Also, please attach your smb.conf. You can directly attach in the list
and need not have a google drive link.


On Wed, May 31, 2017 at 9:37 PM, Raghavendra Talur  wrote:
> Diego,
>
> I see that Samba is still using 3.6.6 Gluster. Is it possible you did
> not restart smb after upgrading gluster(if samba is on same machine as
> Gluster) or if you forgot to update Gluster client packages on Samba
> node?
>
> On Wed, May 31, 2017 at 9:04 PM, Diego Remolina  wrote:
>> Please download the log file from this link:
>>
>> https://drive.google.com/open?id=0B8EAPWIe4oyKN0h0X1pZVkRWVEU
>>
>> Let me know if you need any other log files.
>>
>> Diego
>>
>> On Wed, May 31, 2017 at 11:19 AM, Raghavendra Talur  
>> wrote:
>>> If possible please share the glusterfs-* log files from /var/log/samba.
>>>
>>> This might be because of cluster.lookup-optimize. Adding Poornima and
>>> Raghavendra Gowdappa to help with this.
>>>
>>>
>>> On Wed, May 31, 2017 at 1:03 AM, Diego Remolina  wrote:
 This is a bit puzzling, not sure what difference it would make, but:

 1. Try to open file that has a problem, ie. MyRevitFile.rvt
 Revit opens and shows a window that says access denied.

 2. Rename file, i.e from windows explorer right click, rename to
 MyRevitFile2.rvt

 3. Without opening the file, rename file back to the original, i.e
 MyRevitFile.rvt

 4. Double click on file and now it will open just fine without the
 Access Denied error.

 Any explanation for this? Could the rename operation be forcing or
 updating some attributes that then allow the file to open?

 Diego

 On Tue, May 30, 2017 at 10:57 AM, Diego Remolina  
 wrote:
> This is what I see in the logs set from smb.conf via line ->
> glusterfs:logfile = /var/log/samba/glusterfs-projects.log
>
> [2017-05-30 14:52:31.051524] E [MSGID: 123001]
> [io-cache.c:564:ioc_open_cbk] 0-export-io-cache: inode context is NULL
> (a97bc9bb-68cf-4a69-aef7-39766b323c14) [Invalid argument]
> [2017-05-30 14:52:31.241850] W [MSGID: 114031]
> [client-rpc-fops.c:1100:client3_3_getxattr_cbk] 0-export-client-0:
> remote operation failed. Path:
> /projects/INACTIVE/WESTCOAST/Automotive/Acura/AS-Acura of Richmond/02
> DRAWINGS/02 ARCH/CrownAcura-SD02-ArchModel.rvt (a97bc9bb-68cf-4a69-
> aef7-39766b323c14). Key: glusterfs.get_real_filename:desktop.ini [Not
> a directory]
> [2017-05-30 14:52:31.242956] W [MSGID: 114031]
> [client-rpc-fops.c:1100:client3_3_getxattr_cbk] 0-export-client-1:
> remote operation failed. Path:
> /projects/INACTIVE/WESTCOAST/Automotive/Acura/AS-Acura of Richmond/02
> DRAWINGS/02 ARCH/CrownAcura-SD02-ArchModel.rvt (a97bc9bb-68cf-4a69-
> aef7-39766b323c14). Key: glusterfs.get_real_filename:desktop.ini [Not
> a directory]
>
>
> On Tue, May 30, 2017 at 10:37 AM, Diego Remolina  
> wrote:
>> Hi,
>>
>> Over the weekend we updated a two server glusterfs 3.6.6 install to
>> 3.10.2 We also updated samba and samba-vfs to the latest in CentOS. I
>> enabled several of the newer caching features from gluster 3.9 for
>> small file performance and samba, and we now seem to have some issues
>> with accessing files from glusterfs. When users try to access some
>> files, they get a Permission denied message. This seems to be only via
>> samba as I am able to su - username and then do strings on the file.
>>
>> [root@ysmha02 gluster-backups]# rpm -qa | grep gluster
>> glusterfs-client-xlators-3.10.2-1.el7.x86_64
>> glusterfs-server-3.10.2-1.el7.x86_64
>> glusterfs-api-3.10.2-1.el7.x86_64
>> glusterfs-3.10.2-1.el7.x86_64
>> glusterfs-cli-3.10.2-1.el7.x86_64
>> centos-release-gluster310-1.0-1.el7.centos.noarch
>> samba-vfs-glusterfs-4.4.4-14.el7_3.x86_64
>> glusterfs-fuse-3.10.2-1.el7.x86_64
>> glusterfs-libs-3.10.2-1.el7.x86_64
>> glusterfs-rdma-3.10.2-1.el7.x86_64
>>
>> [root@ysmha02 gluster-backups]# rpm -qa | grep samba
>> samba-common-libs-4.4.4-14.el7_3.x86_64
>> samba-common-tools-4.4.4-14.el7_3.x86_64
>> samba-libs-4.4.4-14.el7_3.x86_64
>> samba-4.4.4-14.el7_3.x86_64
>> samba-client-libs-4.4.4-14.el7_3.x86_64
>> samba-vfs-glusterfs-4.4.4-14.el7_3.x86_64
>> samba-common-4.4.4-14.el7_3.noarch
>>
>> On the samba logs for the machine I notice something weird, samba
>> seems to be trying to stat the file we are trying as a directory to
>> see if it contains desktop.ini:
>>
>> [2017/05/30 10:13:07.297026,  0]
>> ../source3/modules/vfs_glusterfs.c:870(vfs_gluster_stat)
>>  glfs_stat(ACTIVE/Automotive/FORD/AN - Ford East/02
>> DRAWINGS/CURRENT/AN-FORD EAST_04-05-17_CD_R17.rvt/desktop.ini) failed:
>> Not a directory
>> [2017/05/30 10:13:07.298155,  0]
>> ../source3/modules/vfs_glusterfs.c:870(vfs_gluster_stat)
>>  glfs_stat(ACTIVE/Automotive/FORD/AN - F

Re: [Gluster-users] URGENT: Update issues from 3.6.6 to 3.10.2 Accessing files via samba come up with permission denied

[Raghavendra Talur]

Diego,

I see that Samba is still using 3.6.6 Gluster. Is it possible you did
not restart smb after upgrading gluster(if samba is on same machine as
Gluster) or if you forgot to update Gluster client packages on Samba
node?

On Wed, May 31, 2017 at 9:04 PM, Diego Remolina  wrote:
> Please download the log file from this link:
>
> https://drive.google.com/open?id=0B8EAPWIe4oyKN0h0X1pZVkRWVEU
>
> Let me know if you need any other log files.
>
> Diego
>
> On Wed, May 31, 2017 at 11:19 AM, Raghavendra Talur  wrote:
>> If possible please share the glusterfs-* log files from /var/log/samba.
>>
>> This might be because of cluster.lookup-optimize. Adding Poornima and
>> Raghavendra Gowdappa to help with this.
>>
>>
>> On Wed, May 31, 2017 at 1:03 AM, Diego Remolina  wrote:
>>> This is a bit puzzling, not sure what difference it would make, but:
>>>
>>> 1. Try to open file that has a problem, ie. MyRevitFile.rvt
>>> Revit opens and shows a window that says access denied.
>>>
>>> 2. Rename file, i.e from windows explorer right click, rename to
>>> MyRevitFile2.rvt
>>>
>>> 3. Without opening the file, rename file back to the original, i.e
>>> MyRevitFile.rvt
>>>
>>> 4. Double click on file and now it will open just fine without the
>>> Access Denied error.
>>>
>>> Any explanation for this? Could the rename operation be forcing or
>>> updating some attributes that then allow the file to open?
>>>
>>> Diego
>>>
>>> On Tue, May 30, 2017 at 10:57 AM, Diego Remolina  wrote:
 This is what I see in the logs set from smb.conf via line ->
 glusterfs:logfile = /var/log/samba/glusterfs-projects.log

 [2017-05-30 14:52:31.051524] E [MSGID: 123001]
 [io-cache.c:564:ioc_open_cbk] 0-export-io-cache: inode context is NULL
 (a97bc9bb-68cf-4a69-aef7-39766b323c14) [Invalid argument]
 [2017-05-30 14:52:31.241850] W [MSGID: 114031]
 [client-rpc-fops.c:1100:client3_3_getxattr_cbk] 0-export-client-0:
 remote operation failed. Path:
 /projects/INACTIVE/WESTCOAST/Automotive/Acura/AS-Acura of Richmond/02
 DRAWINGS/02 ARCH/CrownAcura-SD02-ArchModel.rvt (a97bc9bb-68cf-4a69-
 aef7-39766b323c14). Key: glusterfs.get_real_filename:desktop.ini [Not
 a directory]
 [2017-05-30 14:52:31.242956] W [MSGID: 114031]
 [client-rpc-fops.c:1100:client3_3_getxattr_cbk] 0-export-client-1:
 remote operation failed. Path:
 /projects/INACTIVE/WESTCOAST/Automotive/Acura/AS-Acura of Richmond/02
 DRAWINGS/02 ARCH/CrownAcura-SD02-ArchModel.rvt (a97bc9bb-68cf-4a69-
 aef7-39766b323c14). Key: glusterfs.get_real_filename:desktop.ini [Not
 a directory]


 On Tue, May 30, 2017 at 10:37 AM, Diego Remolina  
 wrote:
> Hi,
>
> Over the weekend we updated a two server glusterfs 3.6.6 install to
> 3.10.2 We also updated samba and samba-vfs to the latest in CentOS. I
> enabled several of the newer caching features from gluster 3.9 for
> small file performance and samba, and we now seem to have some issues
> with accessing files from glusterfs. When users try to access some
> files, they get a Permission denied message. This seems to be only via
> samba as I am able to su - username and then do strings on the file.
>
> [root@ysmha02 gluster-backups]# rpm -qa | grep gluster
> glusterfs-client-xlators-3.10.2-1.el7.x86_64
> glusterfs-server-3.10.2-1.el7.x86_64
> glusterfs-api-3.10.2-1.el7.x86_64
> glusterfs-3.10.2-1.el7.x86_64
> glusterfs-cli-3.10.2-1.el7.x86_64
> centos-release-gluster310-1.0-1.el7.centos.noarch
> samba-vfs-glusterfs-4.4.4-14.el7_3.x86_64
> glusterfs-fuse-3.10.2-1.el7.x86_64
> glusterfs-libs-3.10.2-1.el7.x86_64
> glusterfs-rdma-3.10.2-1.el7.x86_64
>
> [root@ysmha02 gluster-backups]# rpm -qa | grep samba
> samba-common-libs-4.4.4-14.el7_3.x86_64
> samba-common-tools-4.4.4-14.el7_3.x86_64
> samba-libs-4.4.4-14.el7_3.x86_64
> samba-4.4.4-14.el7_3.x86_64
> samba-client-libs-4.4.4-14.el7_3.x86_64
> samba-vfs-glusterfs-4.4.4-14.el7_3.x86_64
> samba-common-4.4.4-14.el7_3.noarch
>
> On the samba logs for the machine I notice something weird, samba
> seems to be trying to stat the file we are trying as a directory to
> see if it contains desktop.ini:
>
> [2017/05/30 10:13:07.297026,  0]
> ../source3/modules/vfs_glusterfs.c:870(vfs_gluster_stat)
>  glfs_stat(ACTIVE/Automotive/FORD/AN - Ford East/02
> DRAWINGS/CURRENT/AN-FORD EAST_04-05-17_CD_R17.rvt/desktop.ini) failed:
> Not a directory
> [2017/05/30 10:13:07.298155,  0]
> ../source3/modules/vfs_glusterfs.c:870(vfs_gluster_stat)
>  glfs_stat(ACTIVE/Automotive/FORD/AN - Ford East/02
> DRAWINGS/CURRENT/AN-FORD EAST_04-05-17_CD_R17.rvt/desktop.ini) failed:
> Not a directory
>
> This seems to be happening only with files with the .rvt extension.
> Though these files are usually larger in size vs other smaller excel,
> power point, etc files.
>

Re: [Gluster-users] URGENT: Update issues from 3.6.6 to 3.10.2 Accessing files via samba come up with permission denied

[Diego Remolina]

Please download the log file from this link:

https://drive.google.com/open?id=0B8EAPWIe4oyKN0h0X1pZVkRWVEU

Let me know if you need any other log files.

Diego

On Wed, May 31, 2017 at 11:19 AM, Raghavendra Talur  wrote:
> If possible please share the glusterfs-* log files from /var/log/samba.
>
> This might be because of cluster.lookup-optimize. Adding Poornima and
> Raghavendra Gowdappa to help with this.
>
>
> On Wed, May 31, 2017 at 1:03 AM, Diego Remolina  wrote:
>> This is a bit puzzling, not sure what difference it would make, but:
>>
>> 1. Try to open file that has a problem, ie. MyRevitFile.rvt
>> Revit opens and shows a window that says access denied.
>>
>> 2. Rename file, i.e from windows explorer right click, rename to
>> MyRevitFile2.rvt
>>
>> 3. Without opening the file, rename file back to the original, i.e
>> MyRevitFile.rvt
>>
>> 4. Double click on file and now it will open just fine without the
>> Access Denied error.
>>
>> Any explanation for this? Could the rename operation be forcing or
>> updating some attributes that then allow the file to open?
>>
>> Diego
>>
>> On Tue, May 30, 2017 at 10:57 AM, Diego Remolina  wrote:
>>> This is what I see in the logs set from smb.conf via line ->
>>> glusterfs:logfile = /var/log/samba/glusterfs-projects.log
>>>
>>> [2017-05-30 14:52:31.051524] E [MSGID: 123001]
>>> [io-cache.c:564:ioc_open_cbk] 0-export-io-cache: inode context is NULL
>>> (a97bc9bb-68cf-4a69-aef7-39766b323c14) [Invalid argument]
>>> [2017-05-30 14:52:31.241850] W [MSGID: 114031]
>>> [client-rpc-fops.c:1100:client3_3_getxattr_cbk] 0-export-client-0:
>>> remote operation failed. Path:
>>> /projects/INACTIVE/WESTCOAST/Automotive/Acura/AS-Acura of Richmond/02
>>> DRAWINGS/02 ARCH/CrownAcura-SD02-ArchModel.rvt (a97bc9bb-68cf-4a69-
>>> aef7-39766b323c14). Key: glusterfs.get_real_filename:desktop.ini [Not
>>> a directory]
>>> [2017-05-30 14:52:31.242956] W [MSGID: 114031]
>>> [client-rpc-fops.c:1100:client3_3_getxattr_cbk] 0-export-client-1:
>>> remote operation failed. Path:
>>> /projects/INACTIVE/WESTCOAST/Automotive/Acura/AS-Acura of Richmond/02
>>> DRAWINGS/02 ARCH/CrownAcura-SD02-ArchModel.rvt (a97bc9bb-68cf-4a69-
>>> aef7-39766b323c14). Key: glusterfs.get_real_filename:desktop.ini [Not
>>> a directory]
>>>
>>>
>>> On Tue, May 30, 2017 at 10:37 AM, Diego Remolina  wrote:
 Hi,

 Over the weekend we updated a two server glusterfs 3.6.6 install to
 3.10.2 We also updated samba and samba-vfs to the latest in CentOS. I
 enabled several of the newer caching features from gluster 3.9 for
 small file performance and samba, and we now seem to have some issues
 with accessing files from glusterfs. When users try to access some
 files, they get a Permission denied message. This seems to be only via
 samba as I am able to su - username and then do strings on the file.

 [root@ysmha02 gluster-backups]# rpm -qa | grep gluster
 glusterfs-client-xlators-3.10.2-1.el7.x86_64
 glusterfs-server-3.10.2-1.el7.x86_64
 glusterfs-api-3.10.2-1.el7.x86_64
 glusterfs-3.10.2-1.el7.x86_64
 glusterfs-cli-3.10.2-1.el7.x86_64
 centos-release-gluster310-1.0-1.el7.centos.noarch
 samba-vfs-glusterfs-4.4.4-14.el7_3.x86_64
 glusterfs-fuse-3.10.2-1.el7.x86_64
 glusterfs-libs-3.10.2-1.el7.x86_64
 glusterfs-rdma-3.10.2-1.el7.x86_64

 [root@ysmha02 gluster-backups]# rpm -qa | grep samba
 samba-common-libs-4.4.4-14.el7_3.x86_64
 samba-common-tools-4.4.4-14.el7_3.x86_64
 samba-libs-4.4.4-14.el7_3.x86_64
 samba-4.4.4-14.el7_3.x86_64
 samba-client-libs-4.4.4-14.el7_3.x86_64
 samba-vfs-glusterfs-4.4.4-14.el7_3.x86_64
 samba-common-4.4.4-14.el7_3.noarch

 On the samba logs for the machine I notice something weird, samba
 seems to be trying to stat the file we are trying as a directory to
 see if it contains desktop.ini:

 [2017/05/30 10:13:07.297026,  0]
 ../source3/modules/vfs_glusterfs.c:870(vfs_gluster_stat)
  glfs_stat(ACTIVE/Automotive/FORD/AN - Ford East/02
 DRAWINGS/CURRENT/AN-FORD EAST_04-05-17_CD_R17.rvt/desktop.ini) failed:
 Not a directory
 [2017/05/30 10:13:07.298155,  0]
 ../source3/modules/vfs_glusterfs.c:870(vfs_gluster_stat)
  glfs_stat(ACTIVE/Automotive/FORD/AN - Ford East/02
 DRAWINGS/CURRENT/AN-FORD EAST_04-05-17_CD_R17.rvt/desktop.ini) failed:
 Not a directory

 This seems to be happening only with files with the .rvt extension.
 Though these files are usually larger in size vs other smaller excel,
 power point, etc files.

 Here are the complete options for the volume:

 https://pastebin.com/ZH2vMsMN

 I turned off performance.cache-samba-metadata again to see if that
 would help, but seems it does not help.

 I really appreciate any help with this.

 DIego
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http

Re: [Gluster-users] URGENT: Update issues from 3.6.6 to 3.10.2 Accessing files via samba come up with permission denied

[Raghavendra Talur]

If possible please share the glusterfs-* log files from /var/log/samba.

This might be because of cluster.lookup-optimize. Adding Poornima and
Raghavendra Gowdappa to help with this.


On Wed, May 31, 2017 at 1:03 AM, Diego Remolina  wrote:
> This is a bit puzzling, not sure what difference it would make, but:
>
> 1. Try to open file that has a problem, ie. MyRevitFile.rvt
> Revit opens and shows a window that says access denied.
>
> 2. Rename file, i.e from windows explorer right click, rename to
> MyRevitFile2.rvt
>
> 3. Without opening the file, rename file back to the original, i.e
> MyRevitFile.rvt
>
> 4. Double click on file and now it will open just fine without the
> Access Denied error.
>
> Any explanation for this? Could the rename operation be forcing or
> updating some attributes that then allow the file to open?
>
> Diego
>
> On Tue, May 30, 2017 at 10:57 AM, Diego Remolina  wrote:
>> This is what I see in the logs set from smb.conf via line ->
>> glusterfs:logfile = /var/log/samba/glusterfs-projects.log
>>
>> [2017-05-30 14:52:31.051524] E [MSGID: 123001]
>> [io-cache.c:564:ioc_open_cbk] 0-export-io-cache: inode context is NULL
>> (a97bc9bb-68cf-4a69-aef7-39766b323c14) [Invalid argument]
>> [2017-05-30 14:52:31.241850] W [MSGID: 114031]
>> [client-rpc-fops.c:1100:client3_3_getxattr_cbk] 0-export-client-0:
>> remote operation failed. Path:
>> /projects/INACTIVE/WESTCOAST/Automotive/Acura/AS-Acura of Richmond/02
>> DRAWINGS/02 ARCH/CrownAcura-SD02-ArchModel.rvt (a97bc9bb-68cf-4a69-
>> aef7-39766b323c14). Key: glusterfs.get_real_filename:desktop.ini [Not
>> a directory]
>> [2017-05-30 14:52:31.242956] W [MSGID: 114031]
>> [client-rpc-fops.c:1100:client3_3_getxattr_cbk] 0-export-client-1:
>> remote operation failed. Path:
>> /projects/INACTIVE/WESTCOAST/Automotive/Acura/AS-Acura of Richmond/02
>> DRAWINGS/02 ARCH/CrownAcura-SD02-ArchModel.rvt (a97bc9bb-68cf-4a69-
>> aef7-39766b323c14). Key: glusterfs.get_real_filename:desktop.ini [Not
>> a directory]
>>
>>
>> On Tue, May 30, 2017 at 10:37 AM, Diego Remolina  wrote:
>>> Hi,
>>>
>>> Over the weekend we updated a two server glusterfs 3.6.6 install to
>>> 3.10.2 We also updated samba and samba-vfs to the latest in CentOS. I
>>> enabled several of the newer caching features from gluster 3.9 for
>>> small file performance and samba, and we now seem to have some issues
>>> with accessing files from glusterfs. When users try to access some
>>> files, they get a Permission denied message. This seems to be only via
>>> samba as I am able to su - username and then do strings on the file.
>>>
>>> [root@ysmha02 gluster-backups]# rpm -qa | grep gluster
>>> glusterfs-client-xlators-3.10.2-1.el7.x86_64
>>> glusterfs-server-3.10.2-1.el7.x86_64
>>> glusterfs-api-3.10.2-1.el7.x86_64
>>> glusterfs-3.10.2-1.el7.x86_64
>>> glusterfs-cli-3.10.2-1.el7.x86_64
>>> centos-release-gluster310-1.0-1.el7.centos.noarch
>>> samba-vfs-glusterfs-4.4.4-14.el7_3.x86_64
>>> glusterfs-fuse-3.10.2-1.el7.x86_64
>>> glusterfs-libs-3.10.2-1.el7.x86_64
>>> glusterfs-rdma-3.10.2-1.el7.x86_64
>>>
>>> [root@ysmha02 gluster-backups]# rpm -qa | grep samba
>>> samba-common-libs-4.4.4-14.el7_3.x86_64
>>> samba-common-tools-4.4.4-14.el7_3.x86_64
>>> samba-libs-4.4.4-14.el7_3.x86_64
>>> samba-4.4.4-14.el7_3.x86_64
>>> samba-client-libs-4.4.4-14.el7_3.x86_64
>>> samba-vfs-glusterfs-4.4.4-14.el7_3.x86_64
>>> samba-common-4.4.4-14.el7_3.noarch
>>>
>>> On the samba logs for the machine I notice something weird, samba
>>> seems to be trying to stat the file we are trying as a directory to
>>> see if it contains desktop.ini:
>>>
>>> [2017/05/30 10:13:07.297026,  0]
>>> ../source3/modules/vfs_glusterfs.c:870(vfs_gluster_stat)
>>>  glfs_stat(ACTIVE/Automotive/FORD/AN - Ford East/02
>>> DRAWINGS/CURRENT/AN-FORD EAST_04-05-17_CD_R17.rvt/desktop.ini) failed:
>>> Not a directory
>>> [2017/05/30 10:13:07.298155,  0]
>>> ../source3/modules/vfs_glusterfs.c:870(vfs_gluster_stat)
>>>  glfs_stat(ACTIVE/Automotive/FORD/AN - Ford East/02
>>> DRAWINGS/CURRENT/AN-FORD EAST_04-05-17_CD_R17.rvt/desktop.ini) failed:
>>> Not a directory
>>>
>>> This seems to be happening only with files with the .rvt extension.
>>> Though these files are usually larger in size vs other smaller excel,
>>> power point, etc files.
>>>
>>> Here are the complete options for the volume:
>>>
>>> https://pastebin.com/ZH2vMsMN
>>>
>>> I turned off performance.cache-samba-metadata again to see if that
>>> would help, but seems it does not help.
>>>
>>> I really appreciate any help with this.
>>>
>>> DIego
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster client mount fails in mid flight with signum 15

[Sunil Kumar Heggodu Gopala Acharya]

Hi Gabriel,

What is the version of gluster you are running on client? Also, please
share the steps you followed to hit the issue.

Regards,

Sunil kumar Acharya

Senior Software Engineer

Red Hat



T: +91-8067935170 


TRIED. TESTED. TRUSTED. 


On Wed, May 31, 2017 at 12:10 PM, Gabriel Lindeborg <
gabriel.lindeb...@svenskaspel.se> wrote:

> Hello again
>
> The volumes are old, since version 3.6 if I remember correctly…
> These är det 40 last rows of the mnt-loggs go all the mounts
>
> ==> /var/log/glusterfs/mnt-gluster-alfresco.log <==
> /DAEMON/INFO [2017-05-31T07:55:12.787035+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:55:18.153779+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:55:18.160995+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:55:18.167075+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:55:18.171872+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:55:46.427057+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:55:46.432239+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:55:46.439572+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:55:46.444286+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:55:52.035710+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:55:52.042366+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:55:52.049401+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:55:52.054483+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:13.524103+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:13.540314+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:13.543748+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:13.558459+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:18.623109+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:18.643979+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:18.648717+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:18.662941+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:26.268446+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:26.284765+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:26.287503+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:26.302461+02:00] [] []
> /DAEMON/INFO [2017-05-31T08:09:49.408757+02:00] [] []
> /DAEMON/INFO [2017-05-31T08:09:49.415831+02:00] [] []
> /DAEMON/INFO [2017-05-31T08:09:49.421736+02:00] [] []
> /DAEMON/INFO [2017-05-31T08:09:49.421887+02:00] [] []
> /DAEMON/INFO [2017-05-31T08:09:49.431820+02:00] [] []
> [2017-05-31 06:09:54.514841] I [MSGID: 108031]
> [afr-common.c:2340:afr_local_discovery_cbk] 0-alfresco-replicate-0:
> selecting local read_child alfresco-client-2
> /DAEMON/INFO [2017-05-31T08:11:10.179031+02:00] [] []
> /DAEMON/INFO [2017-05-31T08:11:10.186811+02:00] [] []
> /DAEMON/INFO [2017-05-31T08:11:10.194886+02:00] [] []
> /DAEMON/INFO [2017-05-31T08:11:10.195062+02:00] [] []
> /DAEMON/INFO [2017-05-31T08:11:10.205582+02:00] [] []
> [2017-05-31 06:11:14.513620] I [MSGID: 108031]
> [afr-common.c:2340:afr_local_discovery_cbk] 0-alfresco-replicate-0:
> selecting local read_child alfresco-client-2
> [2017-05-31 06:21:21.579748] W [glusterfsd.c:1332:cleanup_and_exit]
> (-->/lib64/libpthread.so.0(+0x7dc5) [0x7fa1d2e4cdc5]
> -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7fa1d44e4fd5]
> -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7fa1d44e4dfb] ) 0-:
> received signum (15), shutting down
> /DAEMON/INFO [2017-05-31T08:21:21.580626+02:00] [] []
> /DAEMON/INFO [2017-05-31T08:21:21.581440+02:00] [] []
>
> ==> /var/log/glusterfs/mnt-gluster-c1_2.log <==
>  82: type meta
>  83: subvolumes c1
>  84: end-volume
>  85:
> +---
> ---+
> /DAEMON/ERR [2017-05-24T10:39:11.130038+02:00] [] []
> [2017-05-24 08:39:11.130638] E [MSGID: 114058] 
> [client-handshake.c:1538:client_query_portmap_cbk]
> 2-c1-client-1: failed to get the port number for remote subvolume. Please
> run 'gluster volume status' on server to see if brick process is running.
> [2017-05-24 08:39:11.130693] I [MSGID: 114018] 
> [client.c:2276:client_rpc_notify]
> 2-c1-client-1: disconnected from c1-client-1. Client process will keep
> trying to connect to glusterd until brick's port is available
> [2017-05-24 08:39:11.130711] E [MSGID: 108006]
> [afr-common.c:4781:afr_notify] 2-c1-replicate-0: All subvolumes are down.
> Going offline until atleast one of them comes back up.
> /DAEMON/INFO [2017-05-24T10:39:11.471603+02:00] [] []
> /DAEMON/ERR [2017-05-24T10:39:11.473783+02:00] [] []
> /DAEMON/INFO [2017-05-24T10:39:14.479153+02:00] [] []
> /DAEMON/ERR [2017-05-24T10:39:14.481430+02:00] [] []
> [2017-05-24 08:39:14.513688] I [MSGID: 108006]
> [afr-common.c:4923:afr_local_init] 2-c1-replicate-0: no subvolumes up
> /DAEMON/INFO [2017-05-24T10:39:14.513712+02:00] [] []
> [2017-05-24 08:39:14.513858] I [MSGID: 114021] [client.c:2361:notify]
> 0-c1-client-2: current graph is no longer active, destroying rpc_client
> [2017-05-24 08:39:14.513879] I [MSGID: 114021] [client.c:2361:notify]
> 0-c1-client-3: current graph is no longer active, destroying rpc_client
> [20

Re: [Gluster-users] Slow write times to gluster disk

[Pat Haley]

Hi Soumya,

What pattern should we be trying to view with the tcpump? Is a one 
minute capture of a copy operation sufficient or are you looking for 
something else?


Pat


On 05/31/2017 06:56 AM, Soumya Koduri wrote:



On 05/31/2017 07:24 AM, Pranith Kumar Karampuri wrote:

Thanks this is good information.

+Soumya

Soumya,
   We are trying to find why kNFS is performing way better than
plain distribute glusterfs+fuse. What information do you think will
benefit us to compare the operations with kNFS vs gluster+fuse? We
already have profile output from fuse.

Could be because all operations done by kNFS are local to the system. 
The operations done by FUSE mount over network could be more in number 
and time-consuming than the ones sent by NFS-client. We could compare 
and examine the pattern from tcpump taken over fuse-mount and 
NFS-mount. Also nfsstat [1] may give some clue.


Sorry I hadn't followed this mail from the beginning. But is this 
comparison between single brick volume and kNFS exporting that brick? 
Otherwise its not a fair comparison if the volume is replicated or 
distributed.


Thanks,
Soumya

[1] https://linux.die.net/man/8/nfsstat



On Wed, May 31, 2017 at 7:10 AM, Pat Haley mailto:pha...@mit.edu>> wrote:


Hi Pranith,

The "dd" command was:

dd if=/dev/zero count=4096 bs=1048576 of=zeros.txt conv=sync

There were 2 instances where dd reported 22 seconds. The output from
the dd tests are in

http://mseas.mit.edu/download/phaley/GlusterUsers/TestVol/dd_testvol_gluster.txt


Pat


On 05/30/2017 09:27 PM, Pranith Kumar Karampuri wrote:

Pat,
   What is the command you used? As per the following output,
it seems like at least one write operation took 16 seconds. Which
is really bad.
 96.391165.10 us  89.00 us *16487014.00 us* 
393212   WRITE



On Tue, May 30, 2017 at 10:36 PM, Pat Haley mailto:pha...@mit.edu>> wrote:


Hi Pranith,

I ran the same 'dd' test both in the gluster test volume and
in the .glusterfs directory of each brick.  The median results
(12 dd trials in each test) are similar to before

  * gluster test volume: 586.5 MB/s
  * bricks (in .glusterfs): 1.4 GB/s

The profile for the gluster test-volume is in

http://mseas.mit.edu/download/phaley/GlusterUsers/TestVol/profile_testvol_gluster.txt


Thanks

Pat




On 05/30/2017 12:10 PM, Pranith Kumar Karampuri wrote:

Let's start with the same 'dd' test we were testing with to
see, what the numbers are. Please provide profile numbers for
the same. From there on we will start tuning the volume to
see what we can do.

On Tue, May 30, 2017 at 9:16 PM, Pat Haley mailto:pha...@mit.edu>> wrote:


Hi Pranith,

Thanks for the tip.  We now have the gluster volume
mounted under /home.  What tests do you recommend we run?

Thanks

Pat



On 05/17/2017 05:01 AM, Pranith Kumar Karampuri wrote:



On Tue, May 16, 2017 at 9:20 PM, Pat Haley
mailto:pha...@mit.edu>> wrote:


Hi Pranith,

Sorry for the delay.  I never saw received your
reply (but I did receive Ben Turner's follow-up to
your reply).  So we tried to create a gluster volume
under /home using different variations of

gluster volume create test-volume
mseas-data2:/home/gbrick_test_1
mseas-data2:/home/gbrick_test_2 transport tcp

However we keep getting errors of the form

Wrong brick type: transport, use
:

Any thoughts on what we're doing wrong?


You should give transport tcp at the beginning I think.
Anyways, transport tcp is the default, so no need to
specify so remove those two words from the CLI.


Also do you have a list of the test we should be
running once we get this volume created? Given the
time-zone difference it might help if we can run a
small battery of tests and post the results rather
than test-post-new test-post... .


This is the first time I am doing performance analysis
on users as far as I remember. In our team there are
separate engineers who do these tests. Ben who replied
earlier is one such engineer.

Ben,
Have any suggestions?



Thanks

Pat



On 05/11/2017 12:06 PM, Pranith Kumar Karampuri 
wrote:



On Thu, May 11, 2017 at 9:32 PM, Pat Haley
mai

Re: [Gluster-users] "Another Transaction is in progres..."

[Atin Mukherjee]

On Wed, May 31, 2017 at 7:31 PM, Erekle Magradze <
erekle.magra...@recogizer.de> wrote:

> Hello,
> I think I have the same issues, I am using gluster as the VM image storage
> under oVirt, would it make sense to restart gluster services on all hosts
> (of course with turned off VMs)
>

If two transactions collide then one of the transaction will fail, but the
theory is you shouldn't be ending up with a stale lock. If further commands
continue to fail then that indicates glusterd has ended up with a stale
lock and then a glusterd service restart is required. We have seen issues
when any op-sm originated transaction (gluster v rebalance, gluster v
profile) collides with any mgmt v3 or syncop based transactions (other
gluster commands) where due to race we can end up with a stale lock but
that will be a rare scenario.




> Cheers
> Erekle
>
>
> On 05/31/2017 03:56 PM, Vijay Bellur wrote:
>
>
>
> On Wed, May 31, 2017 at 9:32 AM, Krist van Besien 
> wrote:
>
>> Hi all,
>>
>> I am trying to do trivial things, like setting quota, or just querying
>> the status and keep getting
>>
>> "Another transaction is in progres for "
>>
>> These messages pop up, then disappear for a while, then pop up again...
>>
>> What do these messages mean? How do I figure out which "transaction" is
>> meant here, and what do I do about it?
>>
>
>
> This message usually means that a different gluster command is being
> executed in the cluster. Most gluster commands are serialized by a cluster
> wide lock. Upon not being able to acquire the cluster lock, this message is
> displayed.
>
> You can check /var/log/glusterfs/cmd_history.log on all storage nodes to
> observe what other commands are in progress at the time of getting this
> error message. Are you per chance using oVirt to manage Gluster? oVirt
> periodically does a "gluster volume status" to determine the volume health
> and that can conflict with other commands being executed.
>
> Regards,
> Vijay
>
>
>
> ___
> Gluster-users mailing 
> listGluster-users@gluster.orghttp://lists.gluster.org/mailman/listinfo/gluster-users
>
>
> --
> Recogizer Group GmbH
>
> Erekle Magradze
> Lead Big Data Engineering & DevOps
> Rheinwerkallee 2, 53227 Bonn
> Tel: +49 228 29974555
>
> E-Mail erekle.magra...@recogizer.de
> Web: www.recogizer.com
>
> Recogizer auf LinkedIn https://www.linkedin.com/company-beta/10039182/
> Folgen Sie uns auf Twitter https://twitter.com/recogizer
>
> -
> Recogizer Group GmbH
> Geschäftsführer: Oliver Habisch, Carsten Kreutze
> Handelsregister: Amtsgericht Bonn HRB 20724
> Sitz der Gesellschaft: Bonn; USt-ID-Nr.: DE294195993
>
> Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen.
> Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich 
> erhalten haben,
> informieren Sie bitte sofort den Absender und löschen Sie diese Mail.
> Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail und der 
> darin enthaltenen Informationen ist nicht gestattet.
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] "Another Transaction is in progres..."

[Erekle Magradze]

Hello,
I think I have the same issues, I am using gluster as the VM image 
storage under oVirt, would it make sense to restart gluster services on 
all hosts (of course with turned off VMs)

Cheers
Erekle

On 05/31/2017 03:56 PM, Vijay Bellur wrote:



On Wed, May 31, 2017 at 9:32 AM, Krist van Besien > wrote:


Hi all,

I am trying to do trivial things, like setting quota, or just
querying the status and keep getting

"Another transaction is in progres for "

These messages pop up, then disappear for a while, then pop up
again...

What do these messages mean? How do I figure out which
"transaction" is meant here, and what do I do about it?



This message usually means that a different gluster command is being 
executed in the cluster. Most gluster commands are serialized by a 
cluster wide lock. Upon not being able to acquire the cluster lock, 
this message is displayed.


You can check /var/log/glusterfs/cmd_history.log on all storage nodes 
to observe what other commands are in progress at the time of getting 
this error message. Are you per chance using oVirt to manage Gluster? 
oVirt periodically does a "gluster volume status" to determine the 
volume health and that can conflict with other commands being executed.


Regards,
Vijay



___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users


--
Recogizer Group GmbH

Erekle Magradze
Lead Big Data Engineering & DevOps
Rheinwerkallee 2, 53227 Bonn
Tel: +49 228 29974555

E-Mail erekle.magra...@recogizer.de
Web: www.recogizer.com
 
Recogizer auf LinkedIn https://www.linkedin.com/company-beta/10039182/

Folgen Sie uns auf Twitter https://twitter.com/recogizer
 
-

Recogizer Group GmbH
Geschäftsführer: Oliver Habisch, Carsten Kreutze
Handelsregister: Amtsgericht Bonn HRB 20724
Sitz der Gesellschaft: Bonn; USt-ID-Nr.: DE294195993
 
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen.

Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten 
haben,
informieren Sie bitte sofort den Absender und löschen Sie diese Mail.
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail und der 
darin enthaltenen Informationen ist nicht gestattet.

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] "Another Transaction is in progres..."

[Vijay Bellur]

On Wed, May 31, 2017 at 9:32 AM, Krist van Besien  wrote:

> Hi all,
>
> I am trying to do trivial things, like setting quota, or just querying the
> status and keep getting
>
> "Another transaction is in progres for "
>
> These messages pop up, then disappear for a while, then pop up again...
>
> What do these messages mean? How do I figure out which "transaction" is
> meant here, and what do I do about it?
>


This message usually means that a different gluster command is being
executed in the cluster. Most gluster commands are serialized by a cluster
wide lock. Upon not being able to acquire the cluster lock, this message is
displayed.

You can check /var/log/glusterfs/cmd_history.log on all storage nodes to
observe what other commands are in progress at the time of getting this
error message. Are you per chance using oVirt to manage Gluster? oVirt
periodically does a "gluster volume status" to determine the volume health
and that can conflict with other commands being executed.

Regards,
Vijay
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] problem with gluster lock

[Erekle Magradze]

Dear Gluster Users,
May be you have experienced similar issue:
I am running glusterfs 3.8.10 as the storage for oVirt images and also I 
have a separate volume as the data storage.

After running this
/gluster volume status /
I am getting this
*Another transaction is in progress for virtdata. Please try again after 
sometime.*
This happens only for the VM image storage volume and data storage 
volumes (of course the most critical ones).

What would be the suggestion?
Where should I start from?
Thanks in advance
Cheers
Erekle
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] "Another Transaction is in progres..."

[Krist van Besien]

Hi all,

I am trying to do trivial things, like setting quota, or just querying the
status and keep getting

"Another transaction is in progres for "

These messages pop up, then disappear for a while, then pop up again...

What do these messages mean? How do I figure out which "transaction" is
meant here, and what do I do about it?

Krist


-- 
Vriendelijke Groet |  Best Regards | Freundliche Grüße | Cordialement
--
Krist van Besien | Senior Architect | Red Hat EMEA Cloud Practice | RHCE |
RHCSA Open Stack
@: kr...@redhat.com | M: +41-79-5936260
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] volume on a segmented network

[lemonnierk]

Hi,

If I understand correctly, it shouldn't work.
You need all your peers and all of your clients to be
able to talk to each other.

The only way I've found around that is to use NFS, that
way the client only needs to see one of the bricks, but
you lose HA so it's not that interesting.


On Wed, May 31, 2017 at 01:46:23PM +0100, lejeczek wrote:
> dear fellas
> 
> I've a pool, it's same one subnet. Now I'd like to add a 
> peer which is on a subnet which will not be 
> available/accessible to all the peers.
> a VolV
> peer X 10.0.0.1 <-> 10.0.0.2 peer Y 192.168.0.2 <-> peer Z 
> 192.168.0.3 # so here 192.168.0.3 and 10.0.0.1 do not see 
> each other.
> 
> Above is not the best practice I understand, or, it does not 
> matter to gluster and it will just work?
> Does anybody have experience with such a setups?
> 
> many thanks.
> L.
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users


signature.asc
Description: Digital signature
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] volume on a segmented network

[lejeczek]

dear fellas

I've a pool, it's same one subnet. Now I'd like to add a 
peer which is on a subnet which will not be 
available/accessible to all the peers.

a VolV
peer X 10.0.0.1 <-> 10.0.0.2 peer Y 192.168.0.2 <-> peer Z 
192.168.0.3 # so here 192.168.0.3 and 10.0.0.1 do not see 
each other.


Above is not the best practice I understand, or, it does not 
matter to gluster and it will just work?

Does anybody have experience with such a setups?

many thanks.
L.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Adding a new replica to the cluster

[Atin Mukherjee]

On Tue, May 30, 2017 at 2:14 PM, Merwan Ouddane  wrote:

> Gluster peer status tells me that the peer is connected
>
>
> Here is the log from the servers in the cluster (they are the same for
> both of them):
>
> https://pastebin.com/4aM94PJ6
>
> The log from the server i'm trying to add to the cluster:
>
> https://pastebin.com/YGTcKRyw
>
I'm sorry but these log files are for brick and client. What I am looking
for is glusterd log ( ls -lrt /var/log/glusterfs/*glusterd* ) from all the
nodes. Also please paste the output of gluster volume status, gluster
volume info, gluster peer status from all the nodes.

>
>
> Merwan
>
> --
> *De :* Atin Mukherjee 
> *Envoyé :* mardi 30 mai 2017 04:51
> *À :* Merwan Ouddane
> *Cc :* gluster-users@gluster.org
> *Objet :* Re: [Gluster-users] Adding a new replica to the cluster
>
>
>
> On Mon, May 29, 2017 at 9:52 PM, Merwan Ouddane  wrote:
>
>> Hello,
>>
>> I wanted to play around with gluster and I made a 2 nodes cluster
>> replicated, then I wanted to add a third replica "on the fly".
>>
>> I manage to probe my third server from the cluster, but when I try to add
>> the new brick to the volume, I get a "Request timed out"
>>
>> My command:
>>   gluster volume add-brick vol0 replica 3 192.168.0.4:/data/br0/vol0
>>
>
> This might happen if the originator glusterd lost connection with other
> peers while this command was in process. Can you check if peer status is
> healthy by gluster peer status command? If yes, you'd need to share the
> glusterd log files from all the nodes with us to further analyse the issue.
>
>
>
>>
>> Does anyone have an idea ?
>>
>> Regards,
>> Merwan
>>
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Slow rsync after upgrade from 3.6.6 to 3.10.2

[Diego Remolina]

Hi,

While we were running 3.6.6 we used to do a nightly rsync to keep a
backup copy of our gluster volume on a separate server. Jobs would
usually start at 9PM and end around 2:00 to 2:30AM consistently every
time.

After upgrading to 3.10.2, the same rsync job (nightly cron) is now
taking an extra 2-3 hours now finishing between 4 and 5AM.

What can I do to identify the cause of the slow down? What information
may I be able to provide for you to help me find why we are taking
much longer to perform the rsync now?

Diego
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Failure while upgrading gluster to 3.10.1

[Atin Mukherjee]

On Wed, May 31, 2017 at 3:53 PM, Pawan Alwandi  wrote:

> Hello Atin,
>
> Sure.  A note though, we are running gluster on Debain Jessie/Wheezy
> hosts, but if you let me know what info you would need I'll work to collect
> that and send across.
>

Basically I need glusterd log file (starting from last restart) along with
the brick logs collected from all the nodes.


> Pawan
>
> On Wed, May 31, 2017 at 2:10 PM, Atin Mukherjee 
> wrote:
>
>> Pawan,
>>
>> I'd need the sosreport from all the nodes to debug and figure out what's
>> going wrong. You'd have to give me some time as I have some critical
>> backlog items to work on.
>>
>> On Wed, 31 May 2017 at 11:30, Pawan Alwandi  wrote:
>>
>>> Hello Atin,
>>>
>>> I've tried restarting gluster one after another, but still see the same
>>> result.
>>>
>>>
>>> On Tue, May 30, 2017 at 10:40 AM, Atin Mukherjee 
>>> wrote:
>>>
 Pawan - I couldn't reach to any conclusive analysis so far. But,
 looking at the client (nfs)  & glusterd log files, it does look like that
 there is an issue w.r.t peer connections. Does restarting all the glusterd
 one by one solve this?

 On Mon, May 29, 2017 at 4:50 PM, Pawan Alwandi 
 wrote:

> Sorry for big attachment in previous mail...last 1000 lines of those
> logs attached now.
>
> On Mon, May 29, 2017 at 4:44 PM, Pawan Alwandi 
> wrote:
>
>>
>>
>> On Thu, May 25, 2017 at 9:54 PM, Atin Mukherjee 
>> wrote:
>>
>>>
>>> On Thu, 25 May 2017 at 19:11, Pawan Alwandi 
>>> wrote:
>>>
 Hello Atin,

 Yes, glusterd on other instances are up and running.  Below is the
 requested output on all the three hosts.

 Host 1

 # gluster peer status
 Number of Peers: 2

 Hostname: 192.168.0.7
 Uuid: 5ec54b4f-f60c-48c6-9e55-95f2bb58f633
 State: Peer in Cluster (Disconnected)

>>>
>>> Glusterd is disconnected here.
>>>


 Hostname: 192.168.0.6
 Uuid: 83e9a0b9-6bd5-483b-8516-d8928805ed95
 State: Peer in Cluster (Disconnected)

>>>
>>> Same as above
>>>
>>> Can you please check what does glusterd log have to say here about
>>> these disconnects?
>>>
>>
>> glusterd keeps logging this every 3s
>>
>> [2017-05-29 11:04:52.182782] W [socket.c:852:__socket_keepalive]
>> 0-socket: failed to set keep idle -1 on socket 5, Invalid argument
>> [2017-05-29 11:04:52.182808] E [socket.c:2966:socket_connect]
>> 0-management: Failed to set keep-alive: Invalid argument
>> [2017-05-29 11:04:52.183032] W [socket.c:852:__socket_keepalive]
>> 0-socket: failed to set keep idle -1 on socket 20, Invalid argument
>> [2017-05-29 11:04:52.183052] E [socket.c:2966:socket_connect]
>> 0-management: Failed to set keep-alive: Invalid argument
>> [2017-05-29 11:04:52.183622] E [rpc-clnt.c:362:saved_frames_unwind]
>> (--> 
>> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7f767c46d483]
>> (--> 
>> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1cf)[0x7f767c2383af]
>> (--> 
>> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f767c2384ce]
>> (--> 
>> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)[0x7f767c239c8e]
>> (--> 
>> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x88)[0x7f767c23a4a8]
>> ) 0-management: forced unwinding frame type(GLUSTERD-DUMP) 
>> op(DUMP(1))
>> called at 2017-05-29 11:04:52.183210 (xid=0x23419)
>> [2017-05-29 11:04:52.183735] W 
>> [glusterd-locks.c:681:glusterd_mgmt_v3_unlock]
>> (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.9/xlator/mgmt/gl
>> usterd.so(glusterd_big_locked_notify+0x4b) [0x7f767734dffb]
>> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.9/xlator/mgmt/glu
>> sterd.so(__glusterd_peer_rpc_notify+0x14a) [0x7f7677357c6a]
>> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.9/xlator/mgmt/glu
>> sterd.so(glusterd_mgmt_v3_unlock+0x4c3) [0x7f76773f0ef3] )
>> 0-management: Lock for vol shared not held
>> [2017-05-29 11:04:52.183928] E [rpc-clnt.c:362:saved_frames_unwind]
>> (--> 
>> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7f767c46d483]
>> (--> 
>> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1cf)[0x7f767c2383af]
>> (--> 
>> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f767c2384ce]
>> (--> 
>> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)[0x7f767c239c8e]
>> (--> 
>> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x88)[0x7f767c23a4a8]
>> ) 0-management: forced unwinding frame type(GLUSTERD-DUMP) 
>> op(DUMP(1))
>> called at 2017-05-29 11:04:52.183422 (xid=0x23419)
>> [2017-05-29 11:04:52.1840

Re: [Gluster-users] [Gluster-devel] Empty info file preventing glusterd from starting

[Atin Mukherjee]

We are going to start working on this patch. Gaurav (Cced) will rebase the
patch and put it for reviews.

On Wed, May 31, 2017 at 4:28 PM, ABHISHEK PALIWAL 
wrote:

> So is there any one working on it to fix this issue either by this patch
> or some other way? if yes then please provide the time plan.
>
> On Wed, May 31, 2017 at 4:25 PM, Amar Tumballi 
> wrote:
>
>>
>>
>> On Wed, May 31, 2017 at 4:08 PM, ABHISHEK PALIWAL <
>> abhishpali...@gmail.com> wrote:
>>
>>> We are using 3.7.6 and on link https://review.gluster.org/#/c/16279
>>> status is "can't merge"
>>>
>>> On Wed, May 31, 2017 at 4:05 PM, Amar Tumballi 
>>> wrote:
>>>
 This is already part of 3.11.0 release?

>>>
>> Sorry about confusion! I was thinking of another patch. This patch is not
>> part of any releases yet.
>>
>> It says can't merge because it is failing regression tests and also looks
>> like it needs a rebase to latest master branch too.
>>
>> -Amar
>>
>>
 On Wed, May 31, 2017 at 3:47 PM, ABHISHEK PALIWAL <
 abhishpali...@gmail.com> wrote:

> Hi Atin,
>
> Could you please let us know any time plan for deliver of this patch.
>
> Regards,
> Abhishek
>
> On Tue, May 9, 2017 at 6:37 PM, ABHISHEK PALIWAL <
> abhishpali...@gmail.com> wrote:
>
>> Actually it is very risky if it will reproduce in production thats is
>> why I said it is on high priority as want to resolve it before 
>> production.
>>
>>
>
>
> --
>
>
>
>
> Regards
> Abhishek Paliwal
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Snapshot auto-delete unmount problem

[Mohammed Rafi K C]

Can you give us more logs on this issue. Also by any chance did somebody
unmount the lv's in any cases ?


Regards

Rafi KC


On 05/31/2017 03:00 PM, Gary Lloyd wrote:
> Hi I am having a problem deleting snapshots, gluster is failing to
> unmount them. I am running centos 7.3 with gluster-3.10.2-1
>
> here is some log output:
>
> [2017-05-31 09:21:39.961371] W [MSGID: 106057]
> [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find]
> 0-management: Snap volume
> 331ec972f90d494d8a86dd4f69d718b7.glust01-li.run-gluster-snaps-331ec972f90d494d8a86dd4f69d718b7-brick1-b
> not found [Invalid argument]
> [2017-05-31 09:21:51.520811] W [MSGID: 106112]
> [glusterd-snapshot.c:8128:glusterd_handle_snap_limit] 0-management:
> Soft-limit (value = 27) of volume shares1 is reached. Deleting
> snapshot Snap_GMT-2017.05.31-09.20.04.
> [2017-05-31 09:21:51.531729] E [MSGID: 106095]
> [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management:
> umounting /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick1
> failed (No such file or directory) [No such file or directory]
> [2017-05-31 09:22:00.540373] E [MSGID: 106038]
> [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove]
> 0-management: umount failed for path
> /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick1 (brick:
> /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick1/b): No such
> file or directory.
> [2017-05-31 09:22:02.442048] W [MSGID: 106033]
> [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management:
> Failed to rmdir: /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/,
> err: Directory not empty. More than one glusterd running on this node.
> [Directory not empty]
> [2017-05-31 09:22:02.443336] W [MSGID: 106039]
> [glusterd-snapshot-utils.c:55:glusterd_snapobject_delete]
> 0-management: Failed destroying lockof snap Snap_GMT-2017.05.31-09.20.04
> [2017-05-31 09:22:02.444038] I [MSGID: 106144]
> [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick
> /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick1/b on port 49157
>
>
>
> Can anyone help ?
>
> Thanks
>
>
> /Gary Lloyd/
> 
> I.T. Systems:Keele University
> Finance & IT Directorate
> Keele:Staffs:IC1 Building:ST5 5NB:UK
> 
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Floating IPv6 in a cluster (as NFS-Ganesha VIP)

[Kaleb S. KEITHLEY]

On 05/31/2017 07:03 AM, Soumya Koduri wrote:
> +Andrew and Ken
> 
> On 05/29/2017 11:48 PM, Jan wrote:
>> Hi all,
>>
>> I love this project, Gluster and Ganesha are amazing. Thank you for this
>> great work!
>>
>> The only thing that I miss is IPv6 support. I know that there are some
>> challenges and that’s OK. For me it’s not important whether Gluster
>> servers use IPv4 or IPv6 to speak each other and replicate data.
>>
>> The only thing that I’d like to have is a floating IPv6 for clients when
>> I use Ganesha (just IPv6, dual stack isn’t needed).
>>
>> I tested it and I put IPv6 into ganesha-ha.conf instead of IPv4 and it
>> didn’t work. But I think that it might work since Ganesha supports IPv6:
>>
>> netstat -plnt
>>
>> tcp6   00 :::2049:::*  LISTEN  1856/ganesha.nfsd
>>
>> Is there a way how to do that? Maybe build a cluster with IPv4 and then
>> change “something” in Pacemaker / Corosync and replace IPv4 by IPv6?
>>
> 
> At-least from [1] looks like it is supported. Do you see any
> errors/warnings in the log files? (/var/log/messages,
> /var/log/pacemaker.log, /var/log/corosync.log)
> 
> 
> [1] https://www.systutorials.com/docs/linux/man/7-ocf_heartbeat_IPaddr2/
> 

/usr/lib/ocf/resource.d/heartbeat/IPaddr2 does support IPv6:

  ...
  Manages virtual IPv4 and IPv6 addresses (Linux
specific version)

  
  
  
  The IPv4 (dotted quad notation) or IPv6 address (colon hexadecimal
notation)
  example IPv4 "192.168.1.1".
  example IPv6 "2001:db8:DC28:0:0:FC57:D4C8:1FFF".
  
  ...


If it's not working I suspect the ganesha-ha.sh script may not handle
IPv6 addrs from the ganesha-ha.conf correctly.

Please file a bug at
https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS component:
common-ha, version: 3.10.

Patches are nice too. ;-)

Thanks

-- 

Kaleb
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Floating IPv6 in a cluster (as NFS-Ganesha VIP)

[Soumya Koduri]

+Andrew and Ken

On 05/29/2017 11:48 PM, Jan wrote:

Hi all,

I love this project, Gluster and Ganesha are amazing. Thank you for this
great work!

The only thing that I miss is IPv6 support. I know that there are some
challenges and that’s OK. For me it’s not important whether Gluster
servers use IPv4 or IPv6 to speak each other and replicate data.

The only thing that I’d like to have is a floating IPv6 for clients when
I use Ganesha (just IPv6, dual stack isn’t needed).

I tested it and I put IPv6 into ganesha-ha.conf instead of IPv4 and it
didn’t work. But I think that it might work since Ganesha supports IPv6:

netstat -plnt

tcp6   00 :::2049:::*  LISTEN  1856/ganesha.nfsd

Is there a way how to do that? Maybe build a cluster with IPv4 and then
change “something” in Pacemaker / Corosync and replace IPv4 by IPv6?



At-least from [1] looks like it is supported. Do you see any 
errors/warnings in the log files? (/var/log/messages, 
/var/log/pacemaker.log, /var/log/corosync.log)



[1] https://www.systutorials.com/docs/linux/man/7-ocf_heartbeat_IPaddr2/


Thank you.

Best regards,

Jan



___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users


___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] [Gluster-devel] Empty info file preventing glusterd from starting

[ABHISHEK PALIWAL]

So is there any one working on it to fix this issue either by this patch or
some other way? if yes then please provide the time plan.

On Wed, May 31, 2017 at 4:25 PM, Amar Tumballi  wrote:

>
>
> On Wed, May 31, 2017 at 4:08 PM, ABHISHEK PALIWAL  > wrote:
>
>> We are using 3.7.6 and on link https://review.gluster.org/#/c/16279
>> status is "can't merge"
>>
>> On Wed, May 31, 2017 at 4:05 PM, Amar Tumballi 
>> wrote:
>>
>>> This is already part of 3.11.0 release?
>>>
>>
> Sorry about confusion! I was thinking of another patch. This patch is not
> part of any releases yet.
>
> It says can't merge because it is failing regression tests and also looks
> like it needs a rebase to latest master branch too.
>
> -Amar
>
>
>>> On Wed, May 31, 2017 at 3:47 PM, ABHISHEK PALIWAL <
>>> abhishpali...@gmail.com> wrote:
>>>
 Hi Atin,

 Could you please let us know any time plan for deliver of this patch.

 Regards,
 Abhishek

 On Tue, May 9, 2017 at 6:37 PM, ABHISHEK PALIWAL <
 abhishpali...@gmail.com> wrote:

> Actually it is very risky if it will reproduce in production thats is
> why I said it is on high priority as want to resolve it before production.
>
>


-- 




Regards
Abhishek Paliwal
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Slow write times to gluster disk

[Soumya Koduri]

On 05/31/2017 07:24 AM, Pranith Kumar Karampuri wrote:

Thanks this is good information.

+Soumya

Soumya,
   We are trying to find why kNFS is performing way better than
plain distribute glusterfs+fuse. What information do you think will
benefit us to compare the operations with kNFS vs gluster+fuse? We
already have profile output from fuse.

Could be because all operations done by kNFS are local to the system. 
The operations done by FUSE mount over network could be more in number 
and time-consuming than the ones sent by NFS-client. We could compare 
and examine the pattern from tcpump taken over fuse-mount and NFS-mount. 
Also nfsstat [1] may give some clue.


Sorry I hadn't followed this mail from the beginning. But is this 
comparison between single brick volume and kNFS exporting that brick? 
Otherwise its not a fair comparison if the volume is replicated or 
distributed.


Thanks,
Soumya

[1] https://linux.die.net/man/8/nfsstat



On Wed, May 31, 2017 at 7:10 AM, Pat Haley mailto:pha...@mit.edu>> wrote:


Hi Pranith,

The "dd" command was:

dd if=/dev/zero count=4096 bs=1048576 of=zeros.txt conv=sync

There were 2 instances where dd reported 22 seconds. The output from
the dd tests are in


http://mseas.mit.edu/download/phaley/GlusterUsers/TestVol/dd_testvol_gluster.txt



Pat


On 05/30/2017 09:27 PM, Pranith Kumar Karampuri wrote:

Pat,
   What is the command you used? As per the following output,
it seems like at least one write operation took 16 seconds. Which
is really bad.
 96.391165.10 us  89.00 us *16487014.00 us* 393212  
 WRITE


On Tue, May 30, 2017 at 10:36 PM, Pat Haley mailto:pha...@mit.edu>> wrote:


Hi Pranith,

I ran the same 'dd' test both in the gluster test volume and
in the .glusterfs directory of each brick.  The median results
(12 dd trials in each test) are similar to before

  * gluster test volume: 586.5 MB/s
  * bricks (in .glusterfs): 1.4 GB/s

The profile for the gluster test-volume is in


http://mseas.mit.edu/download/phaley/GlusterUsers/TestVol/profile_testvol_gluster.txt



Thanks

Pat




On 05/30/2017 12:10 PM, Pranith Kumar Karampuri wrote:

Let's start with the same 'dd' test we were testing with to
see, what the numbers are. Please provide profile numbers for
the same. From there on we will start tuning the volume to
see what we can do.

On Tue, May 30, 2017 at 9:16 PM, Pat Haley mailto:pha...@mit.edu>> wrote:


Hi Pranith,

Thanks for the tip.  We now have the gluster volume
mounted under /home.  What tests do you recommend we run?

Thanks

Pat



On 05/17/2017 05:01 AM, Pranith Kumar Karampuri wrote:



On Tue, May 16, 2017 at 9:20 PM, Pat Haley
mailto:pha...@mit.edu>> wrote:


Hi Pranith,

Sorry for the delay.  I never saw received your
reply (but I did receive Ben Turner's follow-up to
your reply).  So we tried to create a gluster volume
under /home using different variations of

gluster volume create test-volume
mseas-data2:/home/gbrick_test_1
mseas-data2:/home/gbrick_test_2 transport tcp

However we keep getting errors of the form

Wrong brick type: transport, use
:

Any thoughts on what we're doing wrong?


You should give transport tcp at the beginning I think.
Anyways, transport tcp is the default, so no need to
specify so remove those two words from the CLI.


Also do you have a list of the test we should be
running once we get this volume created?  Given the
time-zone difference it might help if we can run a
small battery of tests and post the results rather
than test-post-new test-post... .


This is the first time I am doing performance analysis
on users as far as I remember. In our team there are
separate engineers who do these tests. Ben who replied
earlier is one such engineer.

Ben,
Have any suggestions?



Thanks

Pat



On 05/11/2017 12:06 PM, Pranith Kumar Karampuri wrote:



On Thu, May 11, 2017 at 9:32 PM, Pat Haley
mailto:pha...@mit.edu>> wrote:


Hi Pranith,

The /home partition is mounted as ext4
/home  ext4

Re: [Gluster-users] [Gluster-devel] Empty info file preventing glusterd from starting

[Amar Tumballi]

On Wed, May 31, 2017 at 4:08 PM, ABHISHEK PALIWAL 
wrote:

> We are using 3.7.6 and on link https://review.gluster.org/#/c/16279
> status is "can't merge"
>
> On Wed, May 31, 2017 at 4:05 PM, Amar Tumballi 
> wrote:
>
>> This is already part of 3.11.0 release?
>>
>
Sorry about confusion! I was thinking of another patch. This patch is not
part of any releases yet.

It says can't merge because it is failing regression tests and also looks
like it needs a rebase to latest master branch too.

-Amar


>> On Wed, May 31, 2017 at 3:47 PM, ABHISHEK PALIWAL <
>> abhishpali...@gmail.com> wrote:
>>
>>> Hi Atin,
>>>
>>> Could you please let us know any time plan for deliver of this patch.
>>>
>>> Regards,
>>> Abhishek
>>>
>>> On Tue, May 9, 2017 at 6:37 PM, ABHISHEK PALIWAL <
>>> abhishpali...@gmail.com> wrote:
>>>
 Actually it is very risky if it will reproduce in production thats is
 why I said it is on high priority as want to resolve it before production.


___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] [Gluster-devel] Empty info file preventing glusterd from starting

[ABHISHEK PALIWAL]

We are using 3.7.6 and on link https://review.gluster.org/#/c/16279 status
is "can't merge"

On Wed, May 31, 2017 at 4:05 PM, Amar Tumballi  wrote:

> This is already part of 3.11.0 release?
>
> On Wed, May 31, 2017 at 3:47 PM, ABHISHEK PALIWAL  > wrote:
>
>> Hi Atin,
>>
>> Could you please let us know any time plan for deliver of this patch.
>>
>> Regards,
>> Abhishek
>>
>> On Tue, May 9, 2017 at 6:37 PM, ABHISHEK PALIWAL > > wrote:
>>
>>> Actually it is very risky if it will reproduce in production thats is
>>> why I said it is on high priority as want to resolve it before production.
>>>
>>> On Tue, May 9, 2017 at 6:20 PM, Atin Mukherjee 
>>> wrote:
>>>


 On Tue, May 9, 2017 at 6:10 PM, ABHISHEK PALIWAL <
 abhishpali...@gmail.com> wrote:

> Hi Atin,
>
> Thanks for your reply.
>
>
> Its urgent because this error is very rarely reproducible we have seen
> this 2 3 times in our system till now.
>
> We have delivery in near future so that we want it asap. Please try to
> review it internally.
>

 I don't think your statements justified the reason of urgency as (a)
 you have mentioned it to be *rarely* reproducible and (b) I am still
 waiting for a real use case where glusterd will go through multiple
 restarts in a loop?


> Regards,
> Abhishek
>
> On Tue, May 9, 2017 at 5:58 PM, Atin Mukherjee 
> wrote:
>
>>
>>
>> On Tue, May 9, 2017 at 3:37 PM, ABHISHEK PALIWAL <
>> abhishpali...@gmail.com> wrote:
>>
>>> + Muthu-vingeshwaran
>>>
>>> On Tue, May 9, 2017 at 11:30 AM, ABHISHEK PALIWAL <
>>> abhishpali...@gmail.com> wrote:
>>>
 Hi Atin/Team,

 We are using gluster-3.7.6 with setup of two brick and while
 restart of system I have seen that the glusterd daemon is getting 
 failed
 from start.


 At the time of analyzing the logs from etc-glusterfs...log file
 I have received the below logs


 [2017-05-06 03:33:39.798087] I [MSGID: 100030]
 [glusterfsd.c:2348:main] 0-/usr/sbin/glusterd: Started running
 /usr/sbin/glusterd version 3.7.6 (args: /usr/sbin/glusterd -p
 /var/run/glusterd.pid --log-level INFO)
 [2017-05-06 03:33:39.807859] I [MSGID: 106478]
 [glusterd.c:1350:init] 0-management: Maximum allowed open file 
 descriptors
 set to 65536
 [2017-05-06 03:33:39.807974] I [MSGID: 106479]
 [glusterd.c:1399:init] 0-management: Using /system/glusterd as working
 directory
 [2017-05-06 03:33:39.826833] I [MSGID: 106513]
 [glusterd-store.c:2047:glusterd_restore_op_version] 0-glusterd:
 retrieved op-version: 30706
 [2017-05-06 03:33:39.827515] E [MSGID: 106206]
 [glusterd-store.c:2562:glusterd_store_update_volinfo]
 0-management: Failed to get next store iter
 [2017-05-06 03:33:39.827563] E [MSGID: 106207]
 [glusterd-store.c:2844:glusterd_store_retrieve_volume]
 0-management: Failed to update volinfo for c_glusterfs volume
 [2017-05-06 03:33:39.827625] E [MSGID: 106201]
 [glusterd-store.c:3042:glusterd_store_retrieve_volumes]
 0-management: Unable to restore volume: c_glusterfs
 [2017-05-06 03:33:39.827722] E [MSGID: 101019]
 [xlator.c:428:xlator_init] 0-management: Initialization of volume
 'management' failed, review your volfile again
 [2017-05-06 03:33:39.827762] E [graph.c:322:glusterfs_graph_init]
 0-management: initializing translator failed
 [2017-05-06 03:33:39.827784] E [graph.c:661:glusterfs_graph_activate]
 0-graph: init failed
 [2017-05-06 03:33:39.828396] W [glusterfsd.c:1238:cleanup_and_exit]
 (-->/usr/sbin/glusterd(glusterfs_volumes_init-0x1b0b8)
 [0x1000a648] -->/usr/sbin/glusterd(glusterfs_process_volfp-0x1b210)
 [0x1000a4d8] -->/usr/sbin/glusterd(cleanup_and_exit-0x1beac)
 [0x100097ac] ) 0-: received signum (0), shutting down

>>>
>> Abhishek,
>>
>> This patch needs to be thoroughly reviewed to ensure that it doesn't
>> cause any regression given this touches on the core store management
>> functionality of glusterd. AFAICT, we get into an empty info file only 
>> when
>> volume set operation is executed and in parallel one of the glusterd
>> instance in other nodes have been brought down and whole sequence of
>> operation happens in a loop. The test case through which you can get into
>> this situation is not something you'd hit in production. Please help me 
>> to
>> understand the urgency here.
>>
>> Also in one of the earlier thread, I did mention the workaround of
>> this issue back to Xin through http://lists.gluster.org/piper
>> mail/gluster-users/2017-January/029600.html
>>
>> "If 

Re: [Gluster-users] [Gluster-devel] Empty info file preventing glusterd from starting

[Amar Tumballi]

This is already part of 3.11.0 release?

On Wed, May 31, 2017 at 3:47 PM, ABHISHEK PALIWAL 
wrote:

> Hi Atin,
>
> Could you please let us know any time plan for deliver of this patch.
>
> Regards,
> Abhishek
>
> On Tue, May 9, 2017 at 6:37 PM, ABHISHEK PALIWAL 
> wrote:
>
>> Actually it is very risky if it will reproduce in production thats is why
>> I said it is on high priority as want to resolve it before production.
>>
>> On Tue, May 9, 2017 at 6:20 PM, Atin Mukherjee 
>> wrote:
>>
>>>
>>>
>>> On Tue, May 9, 2017 at 6:10 PM, ABHISHEK PALIWAL <
>>> abhishpali...@gmail.com> wrote:
>>>
 Hi Atin,

 Thanks for your reply.


 Its urgent because this error is very rarely reproducible we have seen
 this 2 3 times in our system till now.

 We have delivery in near future so that we want it asap. Please try to
 review it internally.

>>>
>>> I don't think your statements justified the reason of urgency as (a) you
>>> have mentioned it to be *rarely* reproducible and (b) I am still waiting
>>> for a real use case where glusterd will go through multiple restarts in a
>>> loop?
>>>
>>>
 Regards,
 Abhishek

 On Tue, May 9, 2017 at 5:58 PM, Atin Mukherjee 
 wrote:

>
>
> On Tue, May 9, 2017 at 3:37 PM, ABHISHEK PALIWAL <
> abhishpali...@gmail.com> wrote:
>
>> + Muthu-vingeshwaran
>>
>> On Tue, May 9, 2017 at 11:30 AM, ABHISHEK PALIWAL <
>> abhishpali...@gmail.com> wrote:
>>
>>> Hi Atin/Team,
>>>
>>> We are using gluster-3.7.6 with setup of two brick and while restart
>>> of system I have seen that the glusterd daemon is getting failed from 
>>> start.
>>>
>>>
>>> At the time of analyzing the logs from etc-glusterfs...log file
>>> I have received the below logs
>>>
>>>
>>> [2017-05-06 03:33:39.798087] I [MSGID: 100030]
>>> [glusterfsd.c:2348:main] 0-/usr/sbin/glusterd: Started running
>>> /usr/sbin/glusterd version 3.7.6 (args: /usr/sbin/glusterd -p
>>> /var/run/glusterd.pid --log-level INFO)
>>> [2017-05-06 03:33:39.807859] I [MSGID: 106478]
>>> [glusterd.c:1350:init] 0-management: Maximum allowed open file 
>>> descriptors
>>> set to 65536
>>> [2017-05-06 03:33:39.807974] I [MSGID: 106479]
>>> [glusterd.c:1399:init] 0-management: Using /system/glusterd as working
>>> directory
>>> [2017-05-06 03:33:39.826833] I [MSGID: 106513]
>>> [glusterd-store.c:2047:glusterd_restore_op_version] 0-glusterd:
>>> retrieved op-version: 30706
>>> [2017-05-06 03:33:39.827515] E [MSGID: 106206]
>>> [glusterd-store.c:2562:glusterd_store_update_volinfo] 0-management:
>>> Failed to get next store iter
>>> [2017-05-06 03:33:39.827563] E [MSGID: 106207]
>>> [glusterd-store.c:2844:glusterd_store_retrieve_volume]
>>> 0-management: Failed to update volinfo for c_glusterfs volume
>>> [2017-05-06 03:33:39.827625] E [MSGID: 106201]
>>> [glusterd-store.c:3042:glusterd_store_retrieve_volumes]
>>> 0-management: Unable to restore volume: c_glusterfs
>>> [2017-05-06 03:33:39.827722] E [MSGID: 101019]
>>> [xlator.c:428:xlator_init] 0-management: Initialization of volume
>>> 'management' failed, review your volfile again
>>> [2017-05-06 03:33:39.827762] E [graph.c:322:glusterfs_graph_init]
>>> 0-management: initializing translator failed
>>> [2017-05-06 03:33:39.827784] E [graph.c:661:glusterfs_graph_activate]
>>> 0-graph: init failed
>>> [2017-05-06 03:33:39.828396] W [glusterfsd.c:1238:cleanup_and_exit]
>>> (-->/usr/sbin/glusterd(glusterfs_volumes_init-0x1b0b8) [0x1000a648]
>>> -->/usr/sbin/glusterd(glusterfs_process_volfp-0x1b210) [0x1000a4d8]
>>> -->/usr/sbin/glusterd(cleanup_and_exit-0x1beac) [0x100097ac] ) 0-:
>>> received signum (0), shutting down
>>>
>>
> Abhishek,
>
> This patch needs to be thoroughly reviewed to ensure that it doesn't
> cause any regression given this touches on the core store management
> functionality of glusterd. AFAICT, we get into an empty info file only 
> when
> volume set operation is executed and in parallel one of the glusterd
> instance in other nodes have been brought down and whole sequence of
> operation happens in a loop. The test case through which you can get into
> this situation is not something you'd hit in production. Please help me to
> understand the urgency here.
>
> Also in one of the earlier thread, I did mention the workaround of
> this issue back to Xin through http://lists.gluster.org/piper
> mail/gluster-users/2017-January/029600.html
>
> "If you end up in having a 0 byte info file you'd need to copy the same 
> info file from other node and put it there and restart glusterd"
>
>
>>>
>>> I have found one of the existing case is there and also solution
>>> patch is available but the status of that patch in "c

Re: [Gluster-users] Failure while upgrading gluster to 3.10.1

[Pawan Alwandi]

Hello Atin,

Sure.  A note though, we are running gluster on Debain Jessie/Wheezy hosts,
but if you let me know what info you would need I'll work to collect that
and send across.

Pawan

On Wed, May 31, 2017 at 2:10 PM, Atin Mukherjee  wrote:

> Pawan,
>
> I'd need the sosreport from all the nodes to debug and figure out what's
> going wrong. You'd have to give me some time as I have some critical
> backlog items to work on.
>
> On Wed, 31 May 2017 at 11:30, Pawan Alwandi  wrote:
>
>> Hello Atin,
>>
>> I've tried restarting gluster one after another, but still see the same
>> result.
>>
>>
>> On Tue, May 30, 2017 at 10:40 AM, Atin Mukherjee 
>> wrote:
>>
>>> Pawan - I couldn't reach to any conclusive analysis so far. But, looking
>>> at the client (nfs)  & glusterd log files, it does look like that there is
>>> an issue w.r.t peer connections. Does restarting all the glusterd one by
>>> one solve this?
>>>
>>> On Mon, May 29, 2017 at 4:50 PM, Pawan Alwandi 
>>> wrote:
>>>
 Sorry for big attachment in previous mail...last 1000 lines of those
 logs attached now.

 On Mon, May 29, 2017 at 4:44 PM, Pawan Alwandi 
 wrote:

>
>
> On Thu, May 25, 2017 at 9:54 PM, Atin Mukherjee 
> wrote:
>
>>
>> On Thu, 25 May 2017 at 19:11, Pawan Alwandi 
>> wrote:
>>
>>> Hello Atin,
>>>
>>> Yes, glusterd on other instances are up and running.  Below is the
>>> requested output on all the three hosts.
>>>
>>> Host 1
>>>
>>> # gluster peer status
>>> Number of Peers: 2
>>>
>>> Hostname: 192.168.0.7
>>> Uuid: 5ec54b4f-f60c-48c6-9e55-95f2bb58f633
>>> State: Peer in Cluster (Disconnected)
>>>
>>
>> Glusterd is disconnected here.
>>
>>>
>>>
>>> Hostname: 192.168.0.6
>>> Uuid: 83e9a0b9-6bd5-483b-8516-d8928805ed95
>>> State: Peer in Cluster (Disconnected)
>>>
>>
>> Same as above
>>
>> Can you please check what does glusterd log have to say here about
>> these disconnects?
>>
>
> glusterd keeps logging this every 3s
>
> [2017-05-29 11:04:52.182782] W [socket.c:852:__socket_keepalive]
> 0-socket: failed to set keep idle -1 on socket 5, Invalid argument
> [2017-05-29 11:04:52.182808] E [socket.c:2966:socket_connect]
> 0-management: Failed to set keep-alive: Invalid argument
> [2017-05-29 11:04:52.183032] W [socket.c:852:__socket_keepalive]
> 0-socket: failed to set keep idle -1 on socket 20, Invalid argument
> [2017-05-29 11:04:52.183052] E [socket.c:2966:socket_connect]
> 0-management: Failed to set keep-alive: Invalid argument
> [2017-05-29 11:04:52.183622] E [rpc-clnt.c:362:saved_frames_unwind]
> (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_
> callingfn+0x1a3)[0x7f767c46d483] (--> /usr/lib/x86_64-linux-gnu/
> libgfrpc.so.0(saved_frames_unwind+0x1cf)[0x7f767c2383af] (-->
> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f767c2384ce]
> (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_
> connection_cleanup+0x7e)[0x7f767c239c8e] (-->
> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x88)[0x7f767c23a4a8]
> ) 0-management: forced unwinding frame type(GLUSTERD-DUMP) op(DUMP(1))
> called at 2017-05-29 11:04:52.183210 (xid=0x23419)
> [2017-05-29 11:04:52.183735] W 
> [glusterd-locks.c:681:glusterd_mgmt_v3_unlock]
> (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.9/xlator/mgmt/
> glusterd.so(glusterd_big_locked_notify+0x4b) [0x7f767734dffb]
> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.9/xlator/mgmt/
> glusterd.so(__glusterd_peer_rpc_notify+0x14a) [0x7f7677357c6a]
> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.9/xlator/mgmt/
> glusterd.so(glusterd_mgmt_v3_unlock+0x4c3) [0x7f76773f0ef3] )
> 0-management: Lock for vol shared not held
> [2017-05-29 11:04:52.183928] E [rpc-clnt.c:362:saved_frames_unwind]
> (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_
> callingfn+0x1a3)[0x7f767c46d483] (--> /usr/lib/x86_64-linux-gnu/
> libgfrpc.so.0(saved_frames_unwind+0x1cf)[0x7f767c2383af] (-->
> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f767c2384ce]
> (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_
> connection_cleanup+0x7e)[0x7f767c239c8e] (-->
> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x88)[0x7f767c23a4a8]
> ) 0-management: forced unwinding frame type(GLUSTERD-DUMP) op(DUMP(1))
> called at 2017-05-29 11:04:52.183422 (xid=0x23419)
> [2017-05-29 11:04:52.184027] W 
> [glusterd-locks.c:681:glusterd_mgmt_v3_unlock]
> (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.9/xlator/mgmt/
> glusterd.so(glusterd_big_locked_notify+0x4b) [0x7f767734dffb]
> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.9/xlator/mgmt/
> glusterd.so(__glusterd_peer_rpc_notify+0x14a) [0x7f7677357c6a]
> -->/usr/lib/x86_64-linux-gnu/glusterf

Re: [Gluster-users] FW: ATTN: nbalacha IRC - Gluster - BlackoutWNCT requested info for 0byte file issue

[Nithya Balachandran]

CCing Ravi (arbiter) , Poornima and Raghavendra (parallel readdir)

Hi Joshua,

I had a quick look at the files you sent across.  To summarize the issue,
you see empty linkto files on the mount point.

>From the logs I see  that parallel readdir is enabled for this volume:

performance.readdir-ahead: on
performance.parallel-readdir: on

There was a similar issue reported earlier [1] and fixed as part of [2].
Which version of gluster are you running? Poornima and Raghavendra, can you
take a look and see if it is the same issue?



An interesting observation - the value of the  trusted.glusterfs.dht.linkto
xattr is different on the one of the bricks of the replica set for some of
the files(Ravi, any idea why ?)

-T 2 root root 0 May 18 16:49
/arbiterAA01/gvAA01/brick1/vaultAA01/support-couplers/CPL-SBS01
(CPL-SBS01)/D_VOL-b002-i1873-cd.md5

root@PB-WA-AA-00-A:/# getfattr -e hex -m . -d
/arbiterAA01/gvAA01/brick1/vaultAA01/support-couplers/CPL-SBS01\
\(CPL-SBS01\)/D_VOL-b002-i1873-cd.md5
getfattr: Removing leading '/' from absolute path names
# file: arbiterAA01/gvAA01/brick1/vaultAA01/support-couplers/CPL-SBS01
(CPL-SBS01)/D_VOL-b002-i1873-cd.md5
trusted.gfid=0xfc31c380697a4c70b6af1af35822e519
trusted.glusterfs.dht.linkto=0x6776414130312d726561646469722d61686561642d3100

This value is "gvAA01-readdir-ahead-1"



-T 2 root root 0 May 18 16:49
/brick1/gvAA01/brick/vaultAA01/support-couplers/CPL-SBS01
(CPL-SBS01)/D_VOL-b002-i1873-cd.md5

root@PB-WA-AA-01-B:/# getfattr -e hex -m . -d
/brick1/gvAA01/brick/vaultAA01/support-couplers/CPL-SBS01\
\(CPL-SBS01\)/D_VOL-b002-i1873-cd.md5
getfattr: Removing leading '/' from absolute path names
# file: brick1/gvAA01/brick/vaultAA01/support-couplers/CPL-SBS01
(CPL-SBS01)/D_VOL-b002-i1873-cd.md5
trusted.gfid=0xfc31c380697a4c70b6af1af35822e519
trusted.glusterfs.dht.linkto=0x6776414130312d726561646469722d61686561642d3100

This value is "gvAA01-readdir-ahead-1"



-T 2 root root 0 May 30 14:25
/brick1/gvAA01/brick/vaultAA01/support-couplers/CPL-SBS01
(CPL-SBS01)/D_VOL-b002-i1873-cd.md5

root@PB-WA-AA-02-B:/# getfattr -e hex -m . -d
/brick1/gvAA01/brick/vaultAA01/support-couplers/CPL-SBS01\
\(CPL-SBS01\)/D_VOL-b002-i1873-cd.md5
getfattr: Removing leading '/' from absolute path names
# file: brick1/gvAA01/brick/vaultAA01/support-couplers/CPL-SBS01
(CPL-SBS01)/D_VOL-b002-i1873-cd.md5
trusted.afr.dirty=0x
trusted.bit-rot.version=0x020058df0ab7000710ee
trusted.gfid=0xfc31c380697a4c70b6af1af35822e519
trusted.glusterfs.dht.linkto=0x6776414130312d7265706c69636174652d3100

This value is "gvAA01-replicate-1"


[1] http://lists.gluster.org/pipermail/gluster-users/2017-March/030254.html
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1435942

On 30 May 2017 at 14:20, Joshua Coyle  wrote:

>
>
>
>
> [image: Probax]
>
> *Joshua Coyle*
>
> *P*
>
> 1300 885 117
>
> *E*
>
> joshua.co...@probax.io
>
> *W*
>
> probax.io 
>
> *A*
>
> QV1 Perth,
>
> Level 33/250 St Georges Tce,
>
> Perth WA 6000
>
> *smart safe local cloud*
>
>
>
>
>
> *From:* Joshua Coyle
> *Sent:* Tuesday, May 30, 2017 3:27 PM
> *To:* 'gluster-de...@gluster.org' 
> *Subject:* ATTN: nbalacha IRC - Gluster - BlackoutWNCT requested info for
> 0byte file issue
>
>
>
> Hey nbalacha,
>
>
>
> Please find the requested information attached for the issue being
> discussed in IRC.
>
>
>
> If you need any further info, please let me know.
>
>
>
> Thanks,
>
>
>
> BlackoutWNCT
>
> [image: Probax]
>
> *Joshua Coyle*
>
> *P*
>
> 1300 885 117
>
> *E*
>
> joshua.co...@probax.io
>
> *W*
>
> probax.io 
>
> *A*
>
> QV1 Perth,
>
> Level 33/250 St Georges Tce,
>
> Perth WA 6000
>
> *smart safe local cloud*
>
>
>
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Empty info file preventing glusterd from starting

[ABHISHEK PALIWAL]

Hi Atin,

Could you please let us know any time plan for deliver of this patch.

Regards,
Abhishek

On Tue, May 9, 2017 at 6:37 PM, ABHISHEK PALIWAL 
wrote:

> Actually it is very risky if it will reproduce in production thats is why
> I said it is on high priority as want to resolve it before production.
>
> On Tue, May 9, 2017 at 6:20 PM, Atin Mukherjee 
> wrote:
>
>>
>>
>> On Tue, May 9, 2017 at 6:10 PM, ABHISHEK PALIWAL > > wrote:
>>
>>> Hi Atin,
>>>
>>> Thanks for your reply.
>>>
>>>
>>> Its urgent because this error is very rarely reproducible we have seen
>>> this 2 3 times in our system till now.
>>>
>>> We have delivery in near future so that we want it asap. Please try to
>>> review it internally.
>>>
>>
>> I don't think your statements justified the reason of urgency as (a) you
>> have mentioned it to be *rarely* reproducible and (b) I am still waiting
>> for a real use case where glusterd will go through multiple restarts in a
>> loop?
>>
>>
>>> Regards,
>>> Abhishek
>>>
>>> On Tue, May 9, 2017 at 5:58 PM, Atin Mukherjee 
>>> wrote:
>>>


 On Tue, May 9, 2017 at 3:37 PM, ABHISHEK PALIWAL <
 abhishpali...@gmail.com> wrote:

> + Muthu-vingeshwaran
>
> On Tue, May 9, 2017 at 11:30 AM, ABHISHEK PALIWAL <
> abhishpali...@gmail.com> wrote:
>
>> Hi Atin/Team,
>>
>> We are using gluster-3.7.6 with setup of two brick and while restart
>> of system I have seen that the glusterd daemon is getting failed from 
>> start.
>>
>>
>> At the time of analyzing the logs from etc-glusterfs...log file I
>> have received the below logs
>>
>>
>> [2017-05-06 03:33:39.798087] I [MSGID: 100030]
>> [glusterfsd.c:2348:main] 0-/usr/sbin/glusterd: Started running
>> /usr/sbin/glusterd version 3.7.6 (args: /usr/sbin/glusterd -p
>> /var/run/glusterd.pid --log-level INFO)
>> [2017-05-06 03:33:39.807859] I [MSGID: 106478] [glusterd.c:1350:init]
>> 0-management: Maximum allowed open file descriptors set to 65536
>> [2017-05-06 03:33:39.807974] I [MSGID: 106479] [glusterd.c:1399:init]
>> 0-management: Using /system/glusterd as working directory
>> [2017-05-06 03:33:39.826833] I [MSGID: 106513]
>> [glusterd-store.c:2047:glusterd_restore_op_version] 0-glusterd:
>> retrieved op-version: 30706
>> [2017-05-06 03:33:39.827515] E [MSGID: 106206]
>> [glusterd-store.c:2562:glusterd_store_update_volinfo] 0-management:
>> Failed to get next store iter
>> [2017-05-06 03:33:39.827563] E [MSGID: 106207]
>> [glusterd-store.c:2844:glusterd_store_retrieve_volume] 0-management:
>> Failed to update volinfo for c_glusterfs volume
>> [2017-05-06 03:33:39.827625] E [MSGID: 106201]
>> [glusterd-store.c:3042:glusterd_store_retrieve_volumes]
>> 0-management: Unable to restore volume: c_glusterfs
>> [2017-05-06 03:33:39.827722] E [MSGID: 101019]
>> [xlator.c:428:xlator_init] 0-management: Initialization of volume
>> 'management' failed, review your volfile again
>> [2017-05-06 03:33:39.827762] E [graph.c:322:glusterfs_graph_init]
>> 0-management: initializing translator failed
>> [2017-05-06 03:33:39.827784] E [graph.c:661:glusterfs_graph_activate]
>> 0-graph: init failed
>> [2017-05-06 03:33:39.828396] W [glusterfsd.c:1238:cleanup_and_exit]
>> (-->/usr/sbin/glusterd(glusterfs_volumes_init-0x1b0b8) [0x1000a648]
>> -->/usr/sbin/glusterd(glusterfs_process_volfp-0x1b210) [0x1000a4d8]
>> -->/usr/sbin/glusterd(cleanup_and_exit-0x1beac) [0x100097ac] ) 0-:
>> received signum (0), shutting down
>>
>
 Abhishek,

 This patch needs to be thoroughly reviewed to ensure that it doesn't
 cause any regression given this touches on the core store management
 functionality of glusterd. AFAICT, we get into an empty info file only when
 volume set operation is executed and in parallel one of the glusterd
 instance in other nodes have been brought down and whole sequence of
 operation happens in a loop. The test case through which you can get into
 this situation is not something you'd hit in production. Please help me to
 understand the urgency here.

 Also in one of the earlier thread, I did mention the workaround of this
 issue back to Xin through http://lists.gluster.org/piper
 mail/gluster-users/2017-January/029600.html

 "If you end up in having a 0 byte info file you'd need to copy the same 
 info file from other node and put it there and restart glusterd"


>>
>> I have found one of the existing case is there and also solution
>> patch is available but the status of that patch in "cannot merge". Also 
>> the
>> "info" file is empty and "info.tmp" file present in "lib/glusterd/vol"
>> directory.
>>
>> Below is the link of the existing case.
>>
>> https://review.gluster.org/#/c/16279/5
>>
>> please let me know what is the plan of

[Gluster-users] Snapshot auto-delete unmount problem

[Gary Lloyd]

Hi I am having a problem deleting snapshots, gluster is failing to unmount
them. I am running centos 7.3 with gluster-3.10.2-1

here is some log output:

[2017-05-31 09:21:39.961371] W [MSGID: 106057]
[glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management:
Snap volume
331ec972f90d494d8a86dd4f69d718b7.glust01-li.run-gluster-snaps-331ec972f90d494d8a86dd4f69d718b7-brick1-b
not found [Invalid argument]
[2017-05-31 09:21:51.520811] W [MSGID: 106112]
[glusterd-snapshot.c:8128:glusterd_handle_snap_limit] 0-management:
Soft-limit (value = 27) of volume shares1 is reached. Deleting snapshot
Snap_GMT-2017.05.31-09.20.04.
[2017-05-31 09:21:51.531729] E [MSGID: 106095]
[glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting
/run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick1 failed (No such
file or directory) [No such file or directory]
[2017-05-31 09:22:00.540373] E [MSGID: 106038]
[glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management:
umount failed for path
/run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick1 (brick:
/run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick1/b): No such file
or directory.
[2017-05-31 09:22:02.442048] W [MSGID: 106033]
[glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management:
Failed to rmdir: /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/, err:
Directory not empty. More than one glusterd running on this node.
[Directory not empty]
[2017-05-31 09:22:02.443336] W [MSGID: 106039]
[glusterd-snapshot-utils.c:55:glusterd_snapobject_delete] 0-management:
Failed destroying lockof snap Snap_GMT-2017.05.31-09.20.04
[2017-05-31 09:22:02.444038] I [MSGID: 106144]
[glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick
/run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick1/b on port 49157



Can anyone help ?

Thanks


*Gary Lloyd*

I.T. Systems:Keele University
Finance & IT Directorate
Keele:Staffs:IC1 Building:ST5 5NB:UK

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] How fill my volume for test?

[Nag Pavan Chilakam]

IMO, I kind of agree that there isn't a standard way to fill a file 
system(every vendor tries their own variant like chunk size,etc)?
Also, Don't take the IO methods for validation of perf tests based on the 
methods I mentioned, as I don't have comprehensive idea of perf testing.
If you are behind a certain usecase for your perf test, Kindly ellaborate the 
intent in details. Hopefully someone on this list with that knowledge can help 
you.



- Original Message -
From: "Tahereh Fattahi" 
To: "Nag Pavan Chilakam" 
Cc: gluster-users@gluster.org
Sent: Wednesday, 31 May, 2017 10:06:11 AM
Subject: Re: [Gluster-users] How fill my volume for test?

Thank you
Is it correct that there is not a standard way to fill a file system?
If I fill my gluster volume with each of these methods, the result of
performance test (speed of read, write, ...) are valid?

On Tue, May 30, 2017 at 4:22 PM, Nag Pavan Chilakam 
wrote:

> Hi,
> Do you mean to fill your gluster volume?
> If so, without getting into the details, I can try giving some pointers:
> 1)As I see you want to fill data randomly with many directories and files,
> you can try to download linux kernel(kernel.org) and untar it
> 2)there are different tools available in github like FIO, smallfile(
> https://github.com/bengland2/smallfile/blob/master/smallfile_cli.py),
> crefi
> 3)One last hack-->If the intention is to fill your volume as fast of
> possible and assuming your volume is inside a directory on a thin lv(not
> directly using the mount path as the brick ), then you can directly copy
> data into the LV mount with some random data or huge files(any log files or
> use above tools)
>   This helps fill faster, as you are avoiding network as in a mount.
>   IMPORTANT NOTE: I would like to reinstate, that you must be filling into
> the LV mount instead of the brick mount. Eg: if you LV mount is /rhs/lv1, I
> expect(as recommended), your brick path is something like
> /rhs/lv1/ . Hence you can fill in data under /rhs/lv1/
>
>
> Regards,
> nag
>
> - Original Message -
> From: "Tahereh Fattahi" 
> To: gluster-users@gluster.org
> Sent: Tuesday, 30 May, 2017 3:29:40 PM
> Subject: [Gluster-users] How fill my volume for test?
>
> Hi
> I need to fill my volume and after that start testing.
> Is there a test case, test bench (I dont know the name) a program that
> create a lot of directory and files with different type and size (the data
> in files is not important, maybe random)?
> I can do this by my self but I dont know is this work correct?
> Is there any standard for fill a file system?
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Failure while upgrading gluster to 3.10.1

[Atin Mukherjee]

Pawan,

I'd need the sosreport from all the nodes to debug and figure out what's
going wrong. You'd have to give me some time as I have some critical
backlog items to work on.

On Wed, 31 May 2017 at 11:30, Pawan Alwandi  wrote:

> Hello Atin,
>
> I've tried restarting gluster one after another, but still see the same
> result.
>
>
> On Tue, May 30, 2017 at 10:40 AM, Atin Mukherjee 
> wrote:
>
>> Pawan - I couldn't reach to any conclusive analysis so far. But, looking
>> at the client (nfs)  & glusterd log files, it does look like that there is
>> an issue w.r.t peer connections. Does restarting all the glusterd one by
>> one solve this?
>>
>> On Mon, May 29, 2017 at 4:50 PM, Pawan Alwandi  wrote:
>>
>>> Sorry for big attachment in previous mail...last 1000 lines of those
>>> logs attached now.
>>>
>>> On Mon, May 29, 2017 at 4:44 PM, Pawan Alwandi 
>>> wrote:
>>>


 On Thu, May 25, 2017 at 9:54 PM, Atin Mukherjee 
 wrote:

>
> On Thu, 25 May 2017 at 19:11, Pawan Alwandi  wrote:
>
>> Hello Atin,
>>
>> Yes, glusterd on other instances are up and running.  Below is the
>> requested output on all the three hosts.
>>
>> Host 1
>>
>> # gluster peer status
>> Number of Peers: 2
>>
>> Hostname: 192.168.0.7
>> Uuid: 5ec54b4f-f60c-48c6-9e55-95f2bb58f633
>> State: Peer in Cluster (Disconnected)
>>
>
> Glusterd is disconnected here.
>
>>
>>
>> Hostname: 192.168.0.6
>> Uuid: 83e9a0b9-6bd5-483b-8516-d8928805ed95
>> State: Peer in Cluster (Disconnected)
>>
>
> Same as above
>
> Can you please check what does glusterd log have to say here about
> these disconnects?
>

 glusterd keeps logging this every 3s

 [2017-05-29 11:04:52.182782] W [socket.c:852:__socket_keepalive]
 0-socket: failed to set keep idle -1 on socket 5, Invalid argument
 [2017-05-29 11:04:52.182808] E [socket.c:2966:socket_connect]
 0-management: Failed to set keep-alive: Invalid argument
 [2017-05-29 11:04:52.183032] W [socket.c:852:__socket_keepalive]
 0-socket: failed to set keep idle -1 on socket 20, Invalid argument
 [2017-05-29 11:04:52.183052] E [socket.c:2966:socket_connect]
 0-management: Failed to set keep-alive: Invalid argument
 [2017-05-29 11:04:52.183622] E [rpc-clnt.c:362:saved_frames_unwind]
 (-->
 /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7f767c46d483]
 (-->
 /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1cf)[0x7f767c2383af]
 (-->
 /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f767c2384ce]
 (-->
 /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)[0x7f767c239c8e]
 (-->
 /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x88)[0x7f767c23a4a8]
 ) 0-management: forced unwinding frame type(GLUSTERD-DUMP) op(DUMP(1))
 called at 2017-05-29 11:04:52.183210 (xid=0x23419)
 [2017-05-29 11:04:52.183735] W
 [glusterd-locks.c:681:glusterd_mgmt_v3_unlock]
 (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x4b)
 [0x7f767734dffb]
 -->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.9/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x14a)
 [0x7f7677357c6a]
 -->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x4c3)
 [0x7f76773f0ef3] ) 0-management: Lock for vol shared not held
 [2017-05-29 11:04:52.183928] E [rpc-clnt.c:362:saved_frames_unwind]
 (-->
 /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7f767c46d483]
 (-->
 /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1cf)[0x7f767c2383af]
 (-->
 /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f767c2384ce]
 (-->
 /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)[0x7f767c239c8e]
 (-->
 /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x88)[0x7f767c23a4a8]
 ) 0-management: forced unwinding frame type(GLUSTERD-DUMP) op(DUMP(1))
 called at 2017-05-29 11:04:52.183422 (xid=0x23419)
 [2017-05-29 11:04:52.184027] W
 [glusterd-locks.c:681:glusterd_mgmt_v3_unlock]
 (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x4b)
 [0x7f767734dffb]
 -->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.9/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x14a)
 [0x7f7677357c6a]
 -->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x4c3)
 [0x7f76773f0ef3] ) 0-management: Lock for vol shared not held



>
>
>>
>> # gluster volume status
>> Status of volume: shared
>> Gluster process TCP Port  RDMA Port
>> Online  Pid
>>
>>