Top posting because there are multiple questions.... 1. Atin, it is expected to fail if you don't have RDMA device or if it is not configured.
2. Rafi and Dennis, I was not able to determine from logs if it really is a RDMA bug. The brick logs suggest that brick started and even accepted clients. We should look at the brick log more deeply to see if there is a config issue somewhere. As to why RDMA listener did not initiate on fs2 and fs3, we need to get brick logs for them too. Thanks, Raghavendra Talur On Fri, Sep 30, 2016 at 3:43 PM, Mohammed Rafi K C <rkavu...@redhat.com> wrote: > It seems like an actual bug, if you can file a bug in bugzilla, that > would be great. > > > At least I don't see workaround for this issue, may be till the next > update is available with fix, you can use either rdma alone or tcp alone > volume. > > Let me know whether this is acceptable, if so I can give you the steps to > change the transport of an existing volume. > > > Regards > > Rafi KC > > On 09/30/2016 10:35 AM, Mohammed Rafi K C wrote: > > > > On 09/30/2016 02:35 AM, Dennis Michael wrote: > > > Are there any workarounds to this? RDMA is configured on my servers. > > > > By this, I assume your rdma setup/configuration over IPoIB is working fine. > > Can you tell us what machine you are using and whether SELinux is > configured on the machine or not. > > Also I couldn't see any logs attached here. > > Rafi KC > > > > Dennis > > On Thu, Sep 29, 2016 at 7:19 AM, Atin Mukherjee <amukh...@redhat.com> > wrote: > >> Dennis, >> >> Thanks for sharing the logs. >> >> It seems like a volume configured created with tcp,rdma transport fails >> to start (atleast in my local set up). The issue here is although the brick >> process comes up, but glusterd receives a non zero ret code from the runner >> interface which spawns the brick process(es). >> >> Raghavendra Talur/Rafi, >> >> Is this an intended behaviour if rdma device is not configured? Please >> chime in with your thoughts >> >> >> On Wed, Sep 28, 2016 at 10:22 AM, Atin Mukherjee < <amukh...@redhat.com> >> amukh...@redhat.com> wrote: >> >>> Dennis, >>> >>> It seems like that add-brick has definitely failed and the entry is not >>> committed into glusterd store. volume status and volume info commands are >>> referring the in-memory data for fs4 (which exist) but post a restart they >>> are no longer available. Could you run glusterd with debug log enabled >>> (systemctl stop glusterd; glusterd -LDEBUG) and provide us cmd_history.log, >>> glusterd log along with fs4 brick log files to further analyze the issue? >>> Regarding the missing RDMA ports for fs2, fs3 brick can you cross check if >>> glusterfs-rdma package is installed on both the nodes? >>> >>> On Wed, Sep 28, 2016 at 7:14 AM, Ravishankar N < >>> <ravishan...@redhat.com>ravishan...@redhat.com> wrote: >>> >>>> On 09/27/2016 10:29 PM, Dennis Michael wrote: >>>> >>>> >>>> >>>> [root@fs4 bricks]# gluster volume info >>>> >>>> Volume Name: cees-data >>>> Type: Distribute >>>> Volume ID: 27d2a59c-bdac-4f66-bcd8-e6124e53a4a2 >>>> Status: Started >>>> Number of Bricks: 4 >>>> Transport-type: tcp,rdma >>>> Bricks: >>>> Brick1: fs1:/data/brick >>>> Brick2: fs2:/data/brick >>>> Brick3: fs3:/data/brick >>>> Brick4: fs4:/data/brick >>>> Options Reconfigured: >>>> features.quota-deem-statfs: on >>>> features.inode-quota: on >>>> features.quota: on >>>> performance.readdir-ahead: on >>>> [root@fs4 bricks]# gluster volume status >>>> Status of volume: cees-data >>>> Gluster process TCP Port RDMA Port Online >>>> Pid >>>> ------------------------------------------------------------ >>>> ------------------ >>>> Brick fs1:/data/brick 49152 49153 Y >>>> 1878 >>>> Brick fs2:/data/brick 49152 0 Y >>>> 1707 >>>> Brick fs3:/data/brick 49152 0 Y >>>> 4696 >>>> Brick fs4:/data/brick N/A N/A N >>>> N/A >>>> NFS Server on localhost 2049 0 Y >>>> 13808 >>>> Quota Daemon on localhost N/A N/A Y >>>> 13813 >>>> NFS Server on fs1 2049 0 Y >>>> 6722 >>>> Quota Daemon on fs1 N/A N/A Y >>>> 6730 >>>> NFS Server on fs3 2049 0 Y >>>> 12553 >>>> Quota Daemon on fs3 N/A N/A Y >>>> 12561 >>>> NFS Server on fs2 2049 0 Y >>>> 11702 >>>> Quota Daemon on fs2 N/A N/A Y >>>> 11710 >>>> >>>> Task Status of Volume cees-data >>>> ------------------------------------------------------------ >>>> ------------------ >>>> There are no active volume tasks >>>> >>>> [root@fs4 bricks]# ps auxww | grep gluster >>>> root 13791 0.0 0.0 701472 19768 ? Ssl 09:06 0:00 >>>> /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO >>>> root 13808 0.0 0.0 560236 41420 ? Ssl 09:07 0:00 >>>> /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p >>>> /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S >>>> /var/run/gluster/01c61523374369658a62b75c582b5ac2.socket >>>> root 13813 0.0 0.0 443164 17908 ? Ssl 09:07 0:00 >>>> /usr/sbin/glusterfs -s localhost --volfile-id gluster/quotad -p >>>> /var/lib/glusterd/quotad/run/quotad.pid -l >>>> /var/log/glusterfs/quotad.log -S >>>> /var/run/gluster/3753def90f5c34f656513dba6a544f7d.socket >>>> --xlator-option *replicate*.data-self-heal=off --xlator-option >>>> *replicate*.metadata-self-heal=off --xlator-option >>>> *replicate*.entry-self-heal=off >>>> root 13874 0.0 0.0 1200472 31700 ? Ssl 09:16 0:00 >>>> /usr/sbin/glusterfsd -s fs4 --volfile-id cees-data.fs4.data-brick -p >>>> /var/lib/glusterd/vols/cees-data/run/fs4-data-brick.pid -S >>>> /var/run/gluster/5203ab38be21e1d37c04f6bdfee77d4a.socket --brick-name >>>> /data/brick -l /var/log/glusterfs/bricks/data-brick.log >>>> --xlator-option *-posix.glusterd-uuid=f04b231e-63f8-4374-91ae-17c0c623f165 >>>> --brick-port 49152 49153 --xlator-option >>>> cees-data-server.transport.rdma.listen-port=49153 >>>> --xlator-option cees-data-server.listen-port=49152 >>>> --volfile-server-transport=socket,rdma >>>> root 13941 0.0 0.0 112648 976 pts/0 S+ 09:50 0:00 grep >>>> --color=auto gluster >>>> >>>> [root@fs4 bricks]# systemctl restart glusterfsd glusterd >>>> >>>> [root@fs4 bricks]# ps auxww | grep gluster >>>> root 13808 0.0 0.0 560236 41420 ? Ssl 09:07 0:00 >>>> /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p >>>> /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S >>>> /var/run/gluster/01c61523374369658a62b75c582b5ac2.socket >>>> root 13813 0.0 0.0 443164 17908 ? Ssl 09:07 0:00 >>>> /usr/sbin/glusterfs -s localhost --volfile-id gluster/quotad -p >>>> /var/lib/glusterd/quotad/run/quotad.pid -l >>>> /var/log/glusterfs/quotad.log -S >>>> /var/run/gluster/3753def90f5c34f656513dba6a544f7d.socket >>>> --xlator-option *replicate*.data-self-heal=off --xlator-option >>>> *replicate*.metadata-self-heal=off --xlator-option >>>> *replicate*.entry-self-heal=off >>>> root 13953 0.1 0.0 570740 14988 ? Ssl 09:51 0:00 >>>> /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO >>>> root 13965 0.0 0.0 112648 976 pts/0 S+ 09:51 0:00 grep >>>> --color=auto gluster >>>> >>>> [root@fs4 bricks]# gluster volume info >>>> >>>> Volume Name: cees-data >>>> Type: Distribute >>>> Volume ID: 27d2a59c-bdac-4f66-bcd8-e6124e53a4a2 >>>> Status: Started >>>> Number of Bricks: 3 >>>> Transport-type: tcp,rdma >>>> Bricks: >>>> Brick1: fs1:/data/brick >>>> Brick2: fs2:/data/brick >>>> Brick3: fs3:/data/brick >>>> Options Reconfigured: >>>> performance.readdir-ahead: on >>>> features.quota: on >>>> features.inode-quota: on >>>> features.quota-deem-statfs: on >>>> >>>> >>>> >>>> I'm not sure what's going on here. Restarting glusterd seems to change >>>> the output of gluster volume info? I also see you are using RDMA. Not sure >>>> why the RDMA ports for fs2 and fs3 are not shown in the volume status >>>> output. CC'ing some glusterd/rdma devs for pointers. >>>> >>>> -Ravi >>>> >>>> >>>> >>>> [root@fs4 bricks]# gluster volume status >>>> Status of volume: cees-data >>>> Gluster process TCP Port RDMA Port Online >>>> Pid >>>> ------------------------------------------------------------ >>>> ------------------ >>>> Brick fs1:/data/brick 49152 49153 Y >>>> 1878 >>>> Brick fs2:/data/brick 49152 0 Y >>>> 1707 >>>> Brick fs3:/data/brick 49152 0 Y >>>> 4696 >>>> NFS Server on localhost 2049 0 Y >>>> 13968 >>>> Quota Daemon on localhost N/A N/A Y >>>> 13976 >>>> NFS Server on fs2 2049 0 Y >>>> 11702 >>>> Quota Daemon on fs2 N/A N/A Y >>>> 11710 >>>> NFS Server on fs3 2049 0 Y >>>> 12553 >>>> Quota Daemon on fs3 N/A N/A Y >>>> 12561 >>>> NFS Server on fs1 2049 0 Y >>>> 6722 >>>> >>>> Task Status of Volume cees-data >>>> ------------------------------------------------------------ >>>> ------------------ >>>> There are no active volume tasks >>>> >>>> [root@fs4 bricks]# gluster peer status >>>> Number of Peers: 3 >>>> >>>> Hostname: fs1 >>>> Uuid: ddc0a23e-05e5-48f7-993e-a37e43b21605 >>>> State: Peer in Cluster (Connected) >>>> >>>> Hostname: fs2 >>>> Uuid: e37108f8-d2f1-4f28-adc8-0b3d3401df29 >>>> State: Peer in Cluster (Connected) >>>> >>>> Hostname: fs3 >>>> Uuid: 19a42201-c932-44db-b1a7-8b5b1af32a36 >>>> State: Peer in Cluster (Connected) >>>> >>>> Dennis >>>> >>>> >>>> On Tue, Sep 27, 2016 at 9:40 AM, Ravishankar N < >>>> <ravishan...@redhat.com>ravishan...@redhat.com> wrote: >>>> >>>>> On 09/27/2016 09:53 PM, Dennis Michael wrote: >>>>> >>>>> Yes, you are right. I mixed up the logs. I just ran the add-brick >>>>> command again after cleaning up fs4 and re-installing gluster. This is >>>>> the >>>>> complete fs4 data-brick.log. >>>>> >>>>> [root@fs1 ~]# gluster volume add-brick cees-data fs4:/data/brick >>>>> volume add-brick: failed: Commit failed on fs4. Please check log file >>>>> for details. >>>>> >>>>> [root@fs4 bricks]# pwd >>>>> /var/log/glusterfs/bricks >>>>> [root@fs4 bricks]# cat data-brick.log >>>>> [2016-09-27 16:16:28.095661] I [MSGID: 100030] >>>>> [glusterfsd.c:2338:main] 0-/usr/sbin/glusterfsd: Started running >>>>> /usr/sbin/glusterfsd version 3.7.14 (args: /usr/sbin/glusterfsd -s fs4 >>>>> --volfile-id cees-data.fs4.data-brick -p >>>>> /var/lib/glusterd/vols/cees-data/run/fs4-data-brick.pid >>>>> -S /var/run/gluster/5203ab38be21e1d37c04f6bdfee77d4a.socket >>>>> --brick-name /data/brick -l /var/log/glusterfs/bricks/data-brick.log >>>>> --xlator-option *-posix.glusterd-uuid=f04b231e-63f8-4374-91ae-17c0c623f165 >>>>> --brick-port 49152 --xlator-option >>>>> cees-data-server.transport.rdma.listen-port=49153 >>>>> --xlator-option cees-data-server.listen-port=49152 >>>>> --volfile-server-transport=socket,rdma) >>>>> [2016-09-27 16:16:28.101547] I [MSGID: 101190] >>>>> [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started >>>>> thread with index 1 >>>>> [2016-09-27 16:16:28.104637] I [graph.c:269:gf_add_cmdline_options] >>>>> 0-cees-data-server: adding option 'listen-port' for volume >>>>> 'cees-data-server' with value '49152' >>>>> [2016-09-27 16:16:28.104646] I [graph.c:269:gf_add_cmdline_options] >>>>> 0-cees-data-server: adding option 'transport.rdma.listen-port' for volume >>>>> 'cees-data-server' with value '49153' >>>>> [2016-09-27 16:16:28.104662] I [graph.c:269:gf_add_cmdline_options] >>>>> 0-cees-data-posix: adding option 'glusterd-uuid' for volume >>>>> 'cees-data-posix' with value 'f04b231e-63f8-4374-91ae-17c0c623f165' >>>>> [2016-09-27 16:16:28.104808] I [MSGID: 115034] >>>>> [server.c:403:_check_for_auth_option] 0-/data/brick: skip format >>>>> check for non-addr auth option auth.login./data/brick.allow >>>>> [2016-09-27 16:16:28.104814] I [MSGID: 115034] >>>>> [server.c:403:_check_for_auth_option] 0-/data/brick: skip format >>>>> check for non-addr auth option auth.login.18ddaf4c-ad98-4155- >>>>> 9372-717eae718b4c.password >>>>> [2016-09-27 16:16:28.104883] I [MSGID: 101190] >>>>> [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started >>>>> thread with index 2 >>>>> [2016-09-27 16:16:28.105479] I >>>>> [rpcsvc.c:2196:rpcsvc_set_outstanding_rpc_limit] >>>>> 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64 >>>>> [2016-09-27 16:16:28.105532] W [MSGID: 101002] >>>>> [options.c:957:xl_opt_validate] 0-cees-data-server: option >>>>> 'listen-port' is deprecated, preferred is 'transport.socket.listen-port', >>>>> continuing with correction >>>>> [2016-09-27 16:16:28.109456] W [socket.c:3665:reconfigure] >>>>> 0-cees-data-quota: NBIO on -1 failed (Bad file descriptor) >>>>> [2016-09-27 16:16:28.489255] I [MSGID: 121050] >>>>> [ctr-helper.c:259:extract_ctr_options] 0-gfdbdatastore: CTR Xlator is >>>>> disabled. >>>>> [2016-09-27 16:16:28.489272] W [MSGID: 101105] >>>>> [gfdb_sqlite3.h:239:gfdb_set_sql_params] >>>>> 0-cees-data-changetimerecorder: Failed to retrieve sql-db-pagesize >>>>> from params.Assigning default value: 4096 >>>>> [2016-09-27 16:16:28.489278] W [MSGID: 101105] >>>>> [gfdb_sqlite3.h:239:gfdb_set_sql_params] >>>>> 0-cees-data-changetimerecorder: Failed to retrieve sql-db-journalmode >>>>> from params.Assigning default value: wal >>>>> [2016-09-27 16:16:28.489284] W [MSGID: 101105] >>>>> [gfdb_sqlite3.h:239:gfdb_set_sql_params] >>>>> 0-cees-data-changetimerecorder: Failed to retrieve sql-db-sync from >>>>> params.Assigning default value: off >>>>> [2016-09-27 16:16:28.489288] W [MSGID: 101105] >>>>> [gfdb_sqlite3.h:239:gfdb_set_sql_params] >>>>> 0-cees-data-changetimerecorder: Failed to retrieve sql-db-autovacuum >>>>> from params.Assigning default value: none >>>>> [2016-09-27 16:16:28.490431] I [trash.c:2412:init] 0-cees-data-trash: >>>>> no option specified for 'eliminate', using NULL >>>>> [2016-09-27 16:16:28.672814] W [graph.c:357:_log_if_unknown_option] >>>>> 0-cees-data-server: option 'rpc-auth.auth-glusterfs' is not recognized >>>>> [2016-09-27 16:16:28.672854] W [graph.c:357:_log_if_unknown_option] >>>>> 0-cees-data-server: option 'rpc-auth.auth-unix' is not recognized >>>>> [2016-09-27 16:16:28.672872] W [graph.c:357:_log_if_unknown_option] >>>>> 0-cees-data-server: option 'rpc-auth.auth-null' is not recognized >>>>> [2016-09-27 16:16:28.672924] W [graph.c:357:_log_if_unknown_option] >>>>> 0-cees-data-quota: option 'timeout' is not recognized >>>>> [2016-09-27 16:16:28.672955] W [graph.c:357:_log_if_unknown_option] >>>>> 0-cees-data-trash: option 'brick-path' is not recognized >>>>> Final graph: >>>>> +----------------------------------------------------------- >>>>> -------------------+ >>>>> 1: volume cees-data-posix >>>>> 2: type storage/posix >>>>> 3: option glusterd-uuid f04b231e-63f8-4374-91ae-17c0c623f165 >>>>> 4: option directory /data/brick >>>>> 5: option volume-id 27d2a59c-bdac-4f66-bcd8-e6124e53a4a2 >>>>> 6: option update-link-count-parent on >>>>> 7: end-volume >>>>> 8: >>>>> 9: volume cees-data-trash >>>>> 10: type features/trash >>>>> 11: option trash-dir .trashcan >>>>> 12: option brick-path /data/brick >>>>> 13: option trash-internal-op off >>>>> 14: subvolumes cees-data-posix >>>>> 15: end-volume >>>>> 16: >>>>> 17: volume cees-data-changetimerecorder >>>>> 18: type features/changetimerecorder >>>>> 19: option db-type sqlite3 >>>>> 20: option hot-brick off >>>>> 21: option db-name brick.db >>>>> 22: option db-path /data/brick/.glusterfs/ >>>>> 23: option record-exit off >>>>> 24: option ctr_link_consistency off >>>>> 25: option ctr_lookupheal_link_timeout 300 >>>>> 26: option ctr_lookupheal_inode_timeout 300 >>>>> 27: option record-entry on >>>>> 28: option ctr-enabled off >>>>> 29: option record-counters off >>>>> 30: option ctr-record-metadata-heat off >>>>> 31: option sql-db-cachesize 1000 >>>>> 32: option sql-db-wal-autocheckpoint 1000 >>>>> 33: subvolumes cees-data-trash >>>>> 34: end-volume >>>>> 35: >>>>> 36: volume cees-data-changelog >>>>> 37: type features/changelog >>>>> 38: option changelog-brick /data/brick >>>>> 39: option changelog-dir /data/brick/.glusterfs/changelogs >>>>> 40: option changelog-barrier-timeout 120 >>>>> 41: subvolumes cees-data-changetimerecorder >>>>> 42: end-volume >>>>> 43: >>>>> 44: volume cees-data-bitrot-stub >>>>> 45: type features/bitrot-stub >>>>> 46: option export /data/brick >>>>> 47: subvolumes cees-data-changelog >>>>> 48: end-volume >>>>> 49: >>>>> 50: volume cees-data-access-control >>>>> 51: type features/access-control >>>>> 52: subvolumes cees-data-bitrot-stub >>>>> 53: end-volume >>>>> 54: >>>>> 55: volume cees-data-locks >>>>> 56: type features/locks >>>>> 57: subvolumes cees-data-access-control >>>>> 58: end-volume >>>>> 59: >>>>> 60: volume cees-data-upcall >>>>> 61: type features/upcall >>>>> 62: option cache-invalidation off >>>>> 63: subvolumes cees-data-locks >>>>> 64: end-volume >>>>> 65: >>>>> 66: volume cees-data-io-threads >>>>> 67: type performance/io-threads >>>>> 68: subvolumes cees-data-upcall >>>>> 69: end-volume >>>>> 70: >>>>> 71: volume cees-data-marker >>>>> 72: type features/marker >>>>> 73: option volume-uuid 27d2a59c-bdac-4f66-bcd8-e6124e53a4a2 >>>>> 74: option timestamp-file /var/lib/glusterd/vols/cees-da >>>>> ta/marker.tstamp >>>>> 75: option quota-version 1 >>>>> 76: option xtime off >>>>> 77: option gsync-force-xtime off >>>>> 78: option quota on >>>>> 79: option inode-quota on >>>>> 80: subvolumes cees-data-io-threads >>>>> 81: end-volume >>>>> 82: >>>>> 83: volume cees-data-barrier >>>>> 84: type features/barrier >>>>> 85: option barrier disable >>>>> 86: option barrier-timeout 120 >>>>> 87: subvolumes cees-data-marker >>>>> 88: end-volume >>>>> 89: >>>>> 90: volume cees-data-index >>>>> 91: type features/index >>>>> 92: option index-base /data/brick/.glusterfs/indices >>>>> 93: subvolumes cees-data-barrier >>>>> 94: end-volume >>>>> 95: >>>>> 96: volume cees-data-quota >>>>> 97: type features/quota >>>>> 98: option transport.socket.connect-path >>>>> /var/run/gluster/quotad.socket >>>>> 99: option transport-type socket >>>>> 100: option transport.address-family unix >>>>> 101: option volume-uuid cees-data >>>>> 102: option server-quota on >>>>> 103: option timeout 0 >>>>> 104: option deem-statfs on >>>>> 105: subvolumes cees-data-index >>>>> 106: end-volume >>>>> 107: >>>>> 108: volume cees-data-worm >>>>> 109: type features/worm >>>>> 110: option worm off >>>>> 111: subvolumes cees-data-quota >>>>> 112: end-volume >>>>> 113: >>>>> 114: volume cees-data-read-only >>>>> 115: type features/read-only >>>>> 116: option read-only off >>>>> 117: subvolumes cees-data-worm >>>>> 118: end-volume >>>>> 119: >>>>> 120: volume /data/brick >>>>> 121: type debug/io-stats >>>>> 122: option log-level INFO >>>>> 123: option latency-measurement off >>>>> 124: option count-fop-hits off >>>>> 125: subvolumes cees-data-read-only >>>>> 126: end-volume >>>>> 127: >>>>> 128: volume cees-data-server >>>>> 129: type protocol/server >>>>> 130: option transport.socket.listen-port 49152 >>>>> 131: option rpc-auth.auth-glusterfs on >>>>> 132: option rpc-auth.auth-unix on >>>>> 133: option rpc-auth.auth-null on >>>>> 134: option rpc-auth-allow-insecure on >>>>> 135: option transport.rdma.listen-port 49153 >>>>> 136: option transport-type tcp,rdma >>>>> 137: option auth.login./data/brick.allow >>>>> 18ddaf4c-ad98-4155-9372-717eae718b4c >>>>> 138: option auth.login.18ddaf4c-ad98-4155-9372-717eae718b4c.password >>>>> 9e913e92-7de0-47f9-94ed-d08cbb130d23 >>>>> 139: option auth.addr./data/brick.allow * >>>>> 140: subvolumes /data/brick >>>>> 141: end-volume >>>>> 142: >>>>> +----------------------------------------------------------- >>>>> -------------------+ >>>>> [2016-09-27 16:16:30.079541] I [login.c:81:gf_auth] 0-auth/login: >>>>> allowed user names: 18ddaf4c-ad98-4155-9372-717eae718b4c >>>>> [2016-09-27 16:16:30.079567] I [MSGID: 115029] >>>>> [server-handshake.c:690:server_setvolume] 0-cees-data-server: >>>>> accepted client from >>>>> fs3-12560-2016/09/27-16:16:30:47674-cees-data-client-3-0-0 >>>>> (version: 3.7.14) >>>>> [2016-09-27 16:16:30.081487] I [login.c:81:gf_auth] 0-auth/login: >>>>> allowed user names: 18ddaf4c-ad98-4155-9372-717eae718b4c >>>>> [2016-09-27 16:16:30.081505] I [MSGID: 115029] >>>>> [server-handshake.c:690:server_setvolume] 0-cees-data-server: >>>>> accepted client from >>>>> fs2-11709-2016/09/27-16:16:30:50047-cees-data-client-3-0-0 >>>>> (version: 3.7.14) >>>>> [2016-09-27 16:16:30.111091] I [login.c:81:gf_auth] 0-auth/login: >>>>> allowed user names: 18ddaf4c-ad98-4155-9372-717eae718b4c >>>>> [2016-09-27 16:16:30.111113] I [MSGID: 115029] >>>>> [server-handshake.c:690:server_setvolume] 0-cees-data-server: >>>>> accepted client from >>>>> fs2-11701-2016/09/27-16:16:29:24060-cees-data-client-3-0-0 >>>>> (version: 3.7.14) >>>>> [2016-09-27 16:16:30.112822] I [login.c:81:gf_auth] 0-auth/login: >>>>> allowed user names: 18ddaf4c-ad98-4155-9372-717eae718b4c >>>>> [2016-09-27 16:16:30.112836] I [MSGID: 115029] >>>>> [server-handshake.c:690:server_setvolume] 0-cees-data-server: >>>>> accepted client from >>>>> fs3-12552-2016/09/27-16:16:29:23041-cees-data-client-3-0-0 >>>>> (version: 3.7.14) >>>>> [2016-09-27 16:16:31.950978] I [login.c:81:gf_auth] 0-auth/login: >>>>> allowed user names: 18ddaf4c-ad98-4155-9372-717eae718b4c >>>>> [2016-09-27 16:16:31.950998] I [MSGID: 115029] >>>>> [server-handshake.c:690:server_setvolume] 0-cees-data-server: >>>>> accepted client from >>>>> fs1-6721-2016/09/27-16:16:26:939991-cees-data-client-3-0-0 >>>>> (version: 3.7.14) >>>>> [2016-09-27 16:16:31.981977] I [login.c:81:gf_auth] 0-auth/login: >>>>> allowed user names: 18ddaf4c-ad98-4155-9372-717eae718b4c >>>>> [2016-09-27 16:16:31.981994] I [MSGID: 115029] >>>>> [server-handshake.c:690:server_setvolume] 0-cees-data-server: >>>>> accepted client from >>>>> fs1-6729-2016/09/27-16:16:27:971228-cees-data-client-3-0-0 >>>>> (version: 3.7.14) >>>>> >>>>> >>>>> Hmm, this shows the brick has started. >>>>> Does gluster volume info on fs4 shows all 4 bricks? (I guess it does >>>>> based on your first email). >>>>> Does gluster volume status on fs4 (or ps aux|grep glusterfsd) show >>>>> the brick as running? >>>>> Does gluster peer status on all nodes list the other 3 nodes as >>>>> connected? >>>>> >>>>> If yes, you could try `service glusterd restart` on fs4 and see if if >>>>> brings up the brick? I'm just shooting in the dark here for possible >>>>> clues. >>>>> -Ravi >>>>> >>>>> On Tue, Sep 27, 2016 at 8:46 AM, Ravishankar N < >>>>> <ravishan...@redhat.com>ravishan...@redhat.com> wrote: >>>>> >>>>>> On 09/27/2016 09:06 PM, Dennis Michael wrote: >>>>>> >>>>>> Yes, the brick log /var/log/glusterfs/bricks/data-brick.log is >>>>>> created on fs4, and the snippets showing the errors were from that log. >>>>>> >>>>>> Unless I'm missing something, the snippet below is from glusterd's >>>>>> log and not the brick's as is evident from the function names. >>>>>> -Ravi >>>>>> >>>>>> Dennis >>>>>> >>>>>> On Mon, Sep 26, 2016 at 5:58 PM, Ravishankar N < >>>>>> <ravishan...@redhat.com>ravishan...@redhat.com> wrote: >>>>>> >>>>>>> On 09/27/2016 05:25 AM, Dennis Michael wrote: >>>>>>> >>>>>>>> [2016-09-26 22:44:39.254921] E [MSGID: 106005] >>>>>>>> [glusterd-utils.c:4771:glusterd_brick_start] 0-management: Unable >>>>>>>> to start brick fs4:/data/brick >>>>>>>> [2016-09-26 22:44:39.254949] E [MSGID: 106074] >>>>>>>> [glusterd-brick-ops.c:2372:glusterd_op_add_brick] 0-glusterd: >>>>>>>> Unable to add bricks >>>>>>>> >>>>>>> >>>>>>> Is the brick log created on fs4? Does it contain warnings/errors? >>>>>>> >>>>>>> -Ravi >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>> >>>> >>> >>> >>> -- >>> >>> --Atin >>> >> >> >> >> -- >> >> --Atin >> > > > >
_______________________________________________ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
