HI, Anyone had tested Gluster + Ganesha NFS for IBM MQ? In our organization we trying to onboard Gluster + Ganesha NFS and use it for IBM MQ and encountered some troubles with Ganesha, I yesterday mentioned that on Ganesha Dev call.
IBM has standard test to ensure that NAS is working with their MQ for which they provide amqmfsck utility. https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_7.5.0/com.ibm.mq.pla.doc/q005820_.htm http://www-01.ibm.com/support/docview.wss?uid=swg21433474 For us, Gluser + fuse mount is working fine while Gluster with Ganesha NFS completely fails IBM MQ concurrency test. It goal to ensure that 2 instances of amqmfsck (IBM NAS check utility) running on different servers are trying to update same file on NAS. It just use general unlock(fcntl ), lock (fcntl), lseek, and write system calls, only fast enough. I did not measure update speed, but it is times per second. I am running Gluster 3(2+1) dual replication + arbiter node. On top of it Ganesha. (I also tried usual 2 repli and 3 repli, increase gluster. network.ping-timeout timeout to 42, Lease_Lifetime=60/ Grace_Period=120 in ganesha result the same). Also I tried to mount with/without sync,noatime,mand options. Is it possible to fix this issue? It looks like Ganesha is having issues with locking. 2 Start a second copy of this program with the same parameters on another server. Writing to test file. This will normally complete within about 60 seconds. .. AMQ5667: Error executing system call 'fcntl(F_SETLKW)[c]' on file '/cftha/testas/amqmfsck.wrt', errno 2 ('No such file or directory'). [mqm@lsp7090d cftha]$ 1 Start a second copy of this program with the same parameters on another server. Writing to test file. This will normally complete within about 60 seconds. ........................................................... The tests on the directory completed successfully. 1. iteration done 2 Start a second copy of this program with the same parameters on another server. Writing to test file. This will normally complete within about 60 seconds. ............... AMQ5667: Error executing system call 'fcntl(F_SETLKW)[c]' on file '/cftha/test2/amqmfsck.wrt', errno 2 ('No such file or directory'). [mqm@lsp7090d test2]$ System call: fcntl(fd,F_SETLK,F_UNLCK) System call: fcntl(fd,F_SETLKW,F_WRLCK) System call: lseek(fd,0,SEEK_END) System call: write(fd,data) System call: fcntl(fd,F_SETLK,F_UNLCK) System call: fcntl(fd,F_SETLKW,F_WRLCK) System call: lseek(fd,0,SEEK_END) System call: write(fd,data) System call: fcntl(fd,F_SETLK,F_UNLCK) System call: fcntl(fd,F_SETLKW,F_WRLCK) AMQ5667: Error executing system call 'fcntl(F_SETLKW)[c]' on file '/cftha/test3/amqmfsck.wrt', errno 2 ('No such file or directory'). [mqm@lsp7091d ~]$ System call: fcntl(fd,F_SETLK,F_UNLCK) System call: fcntl(fd,F_SETLKW,F_WRLCK) System call: lseek(fd,0,SEEK_END) System call: write(fd,data) System call: fcntl(fd,F_SETLK,F_UNLCK) System call: fcntl(fd,F_SETLKW,F_WRLCK) System call: lseek(fd,0,SEEK_END) System call: write(fd,data) System call: fcntl(fd,F_SETLK,F_UNLCK) System call: fcntl(fd,F_SETLKW,F_WRLCK) System call: lseek(fd,0,SEEK_END) System call: write(fd,data) System call: fcntl(fd,F_SETLK,F_UNLCK) System call: fcntl(fd,F_SETLKW,F_WRLCK) AMQ5667: Error executing system call 'fcntl(F_SETLKW)[c]' on file '/cftha/test3/amqmfsck.wrt', errno 2 ('No such file or directory'). [mqm@lsp7090d ~]$ Mount: vip1:/cft on /cftha type nfs4 (rw,relatime,sync,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.36.16.53,local_lock=none,addr=10.36.16.77) [mqm@lsp7090d ~]$ [root@lsp7152c ~]# cat /etc/ganesha/ganesha.conf NFSv4 { # Lifetime for NFSv4 Leases Lease_Lifetime = 40; Grace_Period = 80; } %include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.testvol.conf" %include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.mqm.conf" %include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.cft.conf" [root@lsp7152c ~]# [root@lsp7152c exports]# cat export.cft.conf # EXPORT{ Export_Id = 4; Path = "/cft"; FSAL { name = GLUSTER; hostname="localhost"; volume="cft"; } Delegations = none; Access_type = RW; Disable_ACL = true; Squash="No_root_squash"; Pseudo="/cft"; Protocols = "3", "4" ; Transports = "UDP","TCP"; SecType = "sys"; } [root@lsp7152c exports]# [root@lsp7152c exports]# g v inf cft Volume Name: cft Type: Replicate Volume ID: aac3aba2-66d9-4e64-b5b1-a9dbd96889af Status: Started Snapshot Count: 0 Number of Bricks: 1 x (2 + 1) = 3 Transport-type: tcp Bricks: Brick1: lsp7153c:/rhgs/brick2/2 Brick2: lsp7154c:/rhgs/brick2/2 Brick3: lsp7152c:/rhgs/brick_a2/2 (arbiter) Options Reconfigured: features.scrub-throttle: lazy features.scrub-freq: daily features.scrub: Active features.bitrot: on performance.write-behind-window-size: 524288 server.event-threads: 4 client.event-threads: 4 network.inode-lru-limit: 50000 performance.md-cache-timeout: off performance.cache-invalidation: off performance.stat-prefetch: off features.cache-invalidation-timeout: 600 cluster.lookup-optimize: on ganesha.enable: on features.cache-invalidation: off network.ping-timeout: 3 performance.cache-size: 4194304 transport.address-family: inet performance.readdir-ahead: off nfs.disable: on performance.nfs.strict-write-ordering: on performance.strict-write-ordering: on locks.mandatory-locking: off nfs-ganesha: enable cluster.enable-shared-storage: enable [root@lsp7152c exports]# [root@lsp7152c exports]# rpm -qa | egrep 'gluster|ganesha' glusterfs-libs-3.8.4-18.el7rhgs.x86_64 glusterfs-geo-replication-3.8.4-18.el7rhgs.x86_64 python-gluster-3.8.4-18.el7rhgs.noarch glusterfs-client-xlators-3.8.4-18.el7rhgs.x86_64 vdsm-gluster-4.17.33-1.1.el7rhgs.noarch glusterfs-fuse-3.8.4-18.el7rhgs.x86_64 glusterfs-cli-3.8.4-18.el7rhgs.x86_64 nfs-ganesha-gluster-2.4.1-9.el7rhgs.x86_64 nfs-ganesha-2.4.1-9.el7rhgs.x86_64 glusterfs-ganesha-3.8.4-18.el7rhgs.x86_64 glusterfs-api-3.8.4-18.el7rhgs.x86_64 glusterfs-server-3.8.4-18.el7rhgs.x86_64 gluster-nagios-common-0.2.4-1.el7rhgs.noarch glusterfs-3.8.4-18.el7rhgs.x86_64 glusterfs-rdma-3.8.4-18.el7rhgs.x86_64 gluster-nagios-addons-0.2.8-1.el7rhgs.x86_64 [root@lsp7152c exports]# Code example which used same fnctl.* LOCK calls: #!/usr/bin/python import time import fcntl, struct period = 1 ra_deg = '192.0' dec_deg = '29.0' tel_pos_out = open('tel_position','w') lockdata = struct.pack('hhllhh', fcntl.F_WRLCK, 0, 0, 0, 0, 0 ) print lockdata print "X" while period == 1: try: fcntl.fcntl(tel_pos_out.fileno(),fcntl.F_SETLKW, lockdata) tel_pos_out.write(ra_deg+' '+dec_deg +"\n") print("File /tmp/locktest locked") time.sleep(15) tel_pos_out.close() period = period + 1 except IOError: print("can't immediately write-lock the file ($!), blocking ...") else: print("end of procedure") Regards, Domantas ------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________ Nfs-ganesha-support mailing list nfs-ganesha-supp...@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-support ------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
