Re: [Gluster-users] Geo Replication stops replicating

[deepu srinivasan]

Hi Sunny
Sorry, that was a typo.
I used the following command.

> gluster-mountbroker add code-misc sas
>

On Thu, Jun 6, 2019 at 6:23 PM Sunny Kumar  wrote:

> You should not have used this one:
> >
> > gluster-mountbroker remove --volume code-misc --user sas
>
> -- This one is to remove volume/user from mount broker.
>
> Please try setting up mount broker once again.
>
> -Sunny
>
> On Thu, Jun 6, 2019 at 5:28 PM deepu srinivasan 
> wrote:
> >
> > Hi Sunny
> > Please find the logs attached
> >>
> >> The message "E [MSGID: 106061]
> [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option
> mountbroker-root' missing in glusterd vol file" repeated 13 times between
> [2019-06-06 11:51:43.986788] and [2019-06-06 11:52:32.764546]
> >>
> >> The message "W [MSGID: 106176]
> [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: unsuccessful
> mount request [No such file or directory]" repeated 13 times between
> [2019-06-06 11:51:43.986798] and [2019-06-06 11:52:32.764548]
> >>
> >> The message "I [MSGID: 106488]
> [glusterd-handler.c:1559:__glusterd_handle_cli_get_volume] 0-management:
> Received get vol req" repeated 2 times between [2019-06-06 11:53:07.064332]
> and [2019-06-06 11:53:07.303978]
> >>
> >> [2019-06-06 11:55:35.624320] I [MSGID: 106495]
> [glusterd-handler.c:3137:__glusterd_handle_getwd] 0-glusterd: Received
> getwd req
> >>
> >> [2019-06-06 11:55:35.884345] I [MSGID: 106131]
> [glusterd-proc-mgmt.c:86:glusterd_proc_stop] 0-management: quotad already
> stopped
> >>
> >> [2019-06-06 11:55:35.884373] I [MSGID: 106568]
> [glusterd-svc-mgmt.c:253:glusterd_svc_stop] 0-management: quotad service is
> stopped
> >>
> >> [2019-06-06 11:55:35.884459] I [MSGID: 106131]
> [glusterd-proc-mgmt.c:86:glusterd_proc_stop] 0-management: bitd already
> stopped
> >>
> >> [2019-06-06 11:55:35.884473] I [MSGID: 106568]
> [glusterd-svc-mgmt.c:253:glusterd_svc_stop] 0-management: bitd service is
> stopped
> >>
> >> [2019-06-06 11:55:35.884554] I [MSGID: 106131]
> [glusterd-proc-mgmt.c:86:glusterd_proc_stop] 0-management: scrub already
> stopped
> >>
> >> [2019-06-06 11:55:35.884567] I [MSGID: 106568]
> [glusterd-svc-mgmt.c:253:glusterd_svc_stop] 0-management: scrub service is
> stopped
> >>
> >> [2019-06-06 11:55:35.893823] I [run.c:242:runner_log]
> (-->/usr/lib64/glusterfs/6.2/xlator/mgmt/glusterd.so(+0xe8e1a)
> [0x7f7380d60e1a]
> -->/usr/lib64/glusterfs/6.2/xlator/mgmt/glusterd.so(+0xe88e5)
> [0x7f7380d608e5] -->/lib64/libglusterfs.so.0(runner_log+0x115)
> [0x7f738cbc5df5] ) 0-management: Ran script:
> /var/lib/glusterd/hooks/1/set/post/S30samba-set.sh --volname=code-misc -o
> features.read-only=on --gd-workdir=/var/lib/glusterd
> >>
> >> [2019-06-06 11:55:35.900465] I [run.c:242:runner_log]
> (-->/usr/lib64/glusterfs/6.2/xlator/mgmt/glusterd.so(+0xe8e1a)
> [0x7f7380d60e1a]
> -->/usr/lib64/glusterfs/6.2/xlator/mgmt/glusterd.so(+0xe88e5)
> [0x7f7380d608e5] -->/lib64/libglusterfs.so.0(runner_log+0x115)
> [0x7f738cbc5df5] ) 0-management: Ran script:
> /var/lib/glusterd/hooks/1/set/post/S32gluster_enable_shared_storage.sh
> --volname=code-misc -o features.read-only=on --gd-workdir=/var/lib/glusterd
> >>
> >> [2019-06-06 11:55:43.485284] I [MSGID: 106488]
> [glusterd-handler.c:1559:__glusterd_handle_cli_get_volume] 0-management:
> Received get vol req
> >>
> >> The message "I [MSGID: 106488]
> [glusterd-handler.c:1559:__glusterd_handle_cli_get_volume] 0-management:
> Received get vol req" repeated 2 times between [2019-06-06 11:55:43.485284]
> and [2019-06-06 11:55:43.512321]
> >>
> >> [2019-06-06 11:55:44.055419] I [MSGID: 106496]
> [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: Received
> mount req
> >>
> >> [2019-06-06 11:55:44.055473] E [MSGID: 106061]
> [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option
> mountbroker-root' missing in glusterd vol file
> >>
> >> [2019-06-06 11:55:44.055483] W [MSGID: 106176]
> [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: unsuccessful
> mount request [No such file or directory]
> >>
> >> [2019-06-06 11:55:44.056695] I [MSGID: 106496]
> [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: Received
> mount req
> >>
> >> [2019-06-06 11:55:44.056725] E [MSGID: 106061]
> [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option
> mountbroker-root' missing in glusterd vol file
> >>
> >> [2019-06-06 11:55:44.056734] W [MSGID: 106176]
> [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: unsuccessful
> mount request [No such file or directory]
> >>
> >> [2019-06-06 11:55:44.057522] I [MSGID: 106496]
> [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: Received
> mount req
> >>
> >> [2019-06-06 11:55:44.057552] E [MSGID: 106061]
> [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option
> mountbroker-root' missing in glusterd vol file
> >>
> >> [2019-06-06 11:55:44.057562] W [MSGID: 106176]
> [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: unsuccessful
> mount 

Re: [Gluster-users] Geo Replication stops replicating

[deepu srinivasan]

Hi Sunny
Please find the logs attached

> The message "E [MSGID: 106061]
> [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option
> mountbroker-root' missing in glusterd vol file" repeated 13 times between
> [2019-06-06 11:51:43.986788] and [2019-06-06 11:52:32.764546]
>
> The message "W [MSGID: 106176]
> [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: unsuccessful
> mount request [No such file or directory]" repeated 13 times between
> [2019-06-06 11:51:43.986798] and [2019-06-06 11:52:32.764548]
>
> The message "I [MSGID: 106488]
> [glusterd-handler.c:1559:__glusterd_handle_cli_get_volume] 0-management:
> Received get vol req" repeated 2 times between [2019-06-06 11:53:07.064332]
> and [2019-06-06 11:53:07.303978]
>
> [2019-06-06 11:55:35.624320] I [MSGID: 106495]
> [glusterd-handler.c:3137:__glusterd_handle_getwd] 0-glusterd: Received
> getwd req
>
> [2019-06-06 11:55:35.884345] I [MSGID: 106131]
> [glusterd-proc-mgmt.c:86:glusterd_proc_stop] 0-management: quotad already
> stopped
>
> [2019-06-06 11:55:35.884373] I [MSGID: 106568]
> [glusterd-svc-mgmt.c:253:glusterd_svc_stop] 0-management: quotad service is
> stopped
>
> [2019-06-06 11:55:35.884459] I [MSGID: 106131]
> [glusterd-proc-mgmt.c:86:glusterd_proc_stop] 0-management: bitd already
> stopped
>
> [2019-06-06 11:55:35.884473] I [MSGID: 106568]
> [glusterd-svc-mgmt.c:253:glusterd_svc_stop] 0-management: bitd service is
> stopped
>
> [2019-06-06 11:55:35.884554] I [MSGID: 106131]
> [glusterd-proc-mgmt.c:86:glusterd_proc_stop] 0-management: scrub already
> stopped
>
> [2019-06-06 11:55:35.884567] I [MSGID: 106568]
> [glusterd-svc-mgmt.c:253:glusterd_svc_stop] 0-management: scrub service is
> stopped
>
> [2019-06-06 11:55:35.893823] I [run.c:242:runner_log]
> (-->/usr/lib64/glusterfs/6.2/xlator/mgmt/glusterd.so(+0xe8e1a)
> [0x7f7380d60e1a]
> -->/usr/lib64/glusterfs/6.2/xlator/mgmt/glusterd.so(+0xe88e5)
> [0x7f7380d608e5] -->/lib64/libglusterfs.so.0(runner_log+0x115)
> [0x7f738cbc5df5] ) 0-management: Ran script:
> /var/lib/glusterd/hooks/1/set/post/S30samba-set.sh --volname=code-misc -o
> features.read-only=on --gd-workdir=/var/lib/glusterd
>
> [2019-06-06 11:55:35.900465] I [run.c:242:runner_log]
> (-->/usr/lib64/glusterfs/6.2/xlator/mgmt/glusterd.so(+0xe8e1a)
> [0x7f7380d60e1a]
> -->/usr/lib64/glusterfs/6.2/xlator/mgmt/glusterd.so(+0xe88e5)
> [0x7f7380d608e5] -->/lib64/libglusterfs.so.0(runner_log+0x115)
> [0x7f738cbc5df5] ) 0-management: Ran script:
> /var/lib/glusterd/hooks/1/set/post/S32gluster_enable_shared_storage.sh
> --volname=code-misc -o features.read-only=on --gd-workdir=/var/lib/glusterd
>
> [2019-06-06 11:55:43.485284] I [MSGID: 106488]
> [glusterd-handler.c:1559:__glusterd_handle_cli_get_volume] 0-management:
> Received get vol req
>
> The message "I [MSGID: 106488]
> [glusterd-handler.c:1559:__glusterd_handle_cli_get_volume] 0-management:
> Received get vol req" repeated 2 times between [2019-06-06 11:55:43.485284]
> and [2019-06-06 11:55:43.512321]
>
> [2019-06-06 11:55:44.055419] I [MSGID: 106496]
> [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: Received
> mount req
>
> [2019-06-06 11:55:44.055473] E [MSGID: 106061]
> [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option
> mountbroker-root' missing in glusterd vol file
>
> [2019-06-06 11:55:44.055483] W [MSGID: 106176]
> [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: unsuccessful
> mount request [No such file or directory]
>
> [2019-06-06 11:55:44.056695] I [MSGID: 106496]
> [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: Received
> mount req
>
> [2019-06-06 11:55:44.056725] E [MSGID: 106061]
> [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option
> mountbroker-root' missing in glusterd vol file
>
> [2019-06-06 11:55:44.056734] W [MSGID: 106176]
> [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: unsuccessful
> mount request [No such file or directory]
>
> [2019-06-06 11:55:44.057522] I [MSGID: 106496]
> [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: Received
> mount req
>
> [2019-06-06 11:55:44.057552] E [MSGID: 106061]
> [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option
> mountbroker-root' missing in glusterd vol file
>
> [2019-06-06 11:55:44.057562] W [MSGID: 106176]
> [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: unsuccessful
> mount request [No such file or directory]
>
> [2019-06-06 11:55:54.655681] I [MSGID: 106496]
> [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: Received
> mount req
>
> [2019-06-06 11:55:54.655741] E [MSGID: 106061]
> [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option
> mountbroker-root' missing in glusterd vol file
>
> [2019-06-06 11:55:54.655752] W [MSGID: 106176]
> [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: unsuccessful
> mount request [No such file or directory]
>

On Thu, Jun 6, 2019 at 5:09 PM Sunny Kumar  wrote:

> Whats current trackback 

Re: [Gluster-users] Geo Replication stops replicating

[deepu srinivasan]

Hi Kotresh, Sunny
Found this log in the slave machine.

> [2019-06-05 08:49:10.632583] I [MSGID: 106488]
> [glusterd-handler.c:1559:__glusterd_handle_cli_get_volume] 0-management:
> Received get vol req
>
> The message "I [MSGID: 106488]
> [glusterd-handler.c:1559:__glusterd_handle_cli_get_volume] 0-management:
> Received get vol req" repeated 2 times between [2019-06-05 08:49:10.632583]
> and [2019-06-05 08:49:10.670863]
>
> The message "I [MSGID: 106496]
> [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: Received
> mount req" repeated 34 times between [2019-06-05 08:48:41.005398] and
> [2019-06-05 08:50:37.254063]
>
> The message "E [MSGID: 106061]
> [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option
> mountbroker-root' missing in glusterd vol file" repeated 34 times between
> [2019-06-05 08:48:41.005434] and [2019-06-05 08:50:37.254079]
>
> The message "W [MSGID: 106176]
> [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: unsuccessful
> mount request [No such file or directory]" repeated 34 times between
> [2019-06-05 08:48:41.005444] and [2019-06-05 08:50:37.254080]
>
> [2019-06-05 08:50:46.361347] I [MSGID: 106496]
> [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: Received
> mount req
>
> [2019-06-05 08:50:46.361384] E [MSGID: 106061]
> [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option
> mountbroker-root' missing in glusterd vol file
>
> [2019-06-05 08:50:46.361419] W [MSGID: 106176]
> [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: unsuccessful
> mount request [No such file or directory]
>
> The message "I [MSGID: 106496]
> [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: Received
> mount req" repeated 33 times between [2019-06-05 08:50:46.361347] and
> [2019-06-05 08:52:34.019741]
>
> The message "E [MSGID: 106061]
> [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option
> mountbroker-root' missing in glusterd vol file" repeated 33 times between
> [2019-06-05 08:50:46.361384] and [2019-06-05 08:52:34.019757]
>
> The message "W [MSGID: 106176]
> [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: unsuccessful
> mount request [No such file or directory]" repeated 33 times between
> [2019-06-05 08:50:46.361419] and [2019-06-05 08:52:34.019758]
>
> [2019-06-05 08:52:44.426839] I [MSGID: 106496]
> [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: Received
> mount req
>
> [2019-06-05 08:52:44.426886] E [MSGID: 106061]
> [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option
> mountbroker-root' missing in glusterd vol file
>
> [2019-06-05 08:52:44.426896] W [MSGID: 106176]
> [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: unsuccessful
> mount request [No such file or directory]
>

On Wed, Jun 5, 2019 at 1:06 AM deepu srinivasan  wrote:

> Thankyou Kotresh
>
> On Tue, Jun 4, 2019, 11:20 PM Kotresh Hiremath Ravishankar <
> khire...@redhat.com> wrote:
>
>> Ccing Sunny, who was investing similar issue.
>>
>> On Tue, Jun 4, 2019 at 5:46 PM deepu srinivasan 
>> wrote:
>>
>>> Have already added the path in bashrc . Still in faulty state
>>>
>>> On Tue, Jun 4, 2019, 5:27 PM Kotresh Hiremath Ravishankar <
>>> khire...@redhat.com> wrote:
>>>
 could you please try adding /usr/sbin to $PATH for user 'sas'? If it's
 bash, add 'export PATH=/usr/sbin:$PATH' in
 /home/sas/.bashrc

 On Tue, Jun 4, 2019 at 5:24 PM deepu srinivasan 
 wrote:

> Hi Kortesh
> Please find the logs of the above error
> *Master log snippet*
>
>> [2019-06-04 11:52:09.254731] I [resource(worker
>> /home/sas/gluster/data/code-misc):1379:connect_remote] SSH: Initializing
>> SSH connection between master and slave...
>>  [2019-06-04 11:52:09.308923] D [repce(worker
>> /home/sas/gluster/data/code-misc):196:push] RepceClient: call
>> 89724:139652759443264:1559649129.31 __repce_version__() ...
>>  [2019-06-04 11:52:09.602792] E [syncdutils(worker
>> /home/sas/gluster/data/code-misc):311:log_raise_exception] :
>> connection to peer is broken
>>  [2019-06-04 11:52:09.603312] E [syncdutils(worker
>> /home/sas/gluster/data/code-misc):805:errlog] Popen: command returned 
>> error
>>   cmd=ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i
>> /var/lib/ glusterd/geo-replication/secret.pem -p 22 -oControlMaster=auto 
>> -S
>> /tmp/gsyncd-aux-ssh-4aL2tc/d893f66e0addc32f7d0080bb503f5185.sock
>> sas@192.168.185.107 /usr/libexec/glusterfs/gsyncd slave code-misc
>> sas@   192.168.185.107::code-misc --master-node 192.168.185.106
>> --master-node-id 851b64d0-d885-4ae9-9b38-ab5b15db0fec --master-brick
>> /home/sas/gluster/data/code-misc --local-node 192.168.185.122 
>> --local-node-
>>   id bcaa7af6-c3a1-4411-8e99-4ebecb32eb6a --slave-timeout 120
>> --slave-log-level DEBUG --slave-gluster-log-level INFO
>> --slave-gluster-command-dir /usr/sbin   error=1
>>  

Re: [Gluster-users] Geo Replication stops replicating

[deepu srinivasan]

Hi Kotresh, Sunny
I Have mailed the logs I found in one of the slave machines. Is there
anything to do with permission? Please help.

On Wed, Jun 5, 2019 at 2:28 PM deepu srinivasan  wrote:

> Hi Kotresh, Sunny
> Found this log in the slave machine.
>
>> [2019-06-05 08:49:10.632583] I [MSGID: 106488]
>> [glusterd-handler.c:1559:__glusterd_handle_cli_get_volume] 0-management:
>> Received get vol req
>>
>> The message "I [MSGID: 106488]
>> [glusterd-handler.c:1559:__glusterd_handle_cli_get_volume] 0-management:
>> Received get vol req" repeated 2 times between [2019-06-05 08:49:10.632583]
>> and [2019-06-05 08:49:10.670863]
>>
>> The message "I [MSGID: 106496]
>> [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: Received
>> mount req" repeated 34 times between [2019-06-05 08:48:41.005398] and
>> [2019-06-05 08:50:37.254063]
>>
>> The message "E [MSGID: 106061]
>> [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option
>> mountbroker-root' missing in glusterd vol file" repeated 34 times between
>> [2019-06-05 08:48:41.005434] and [2019-06-05 08:50:37.254079]
>>
>> The message "W [MSGID: 106176]
>> [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: unsuccessful
>> mount request [No such file or directory]" repeated 34 times between
>> [2019-06-05 08:48:41.005444] and [2019-06-05 08:50:37.254080]
>>
>> [2019-06-05 08:50:46.361347] I [MSGID: 106496]
>> [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: Received
>> mount req
>>
>> [2019-06-05 08:50:46.361384] E [MSGID: 106061]
>> [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option
>> mountbroker-root' missing in glusterd vol file
>>
>> [2019-06-05 08:50:46.361419] W [MSGID: 106176]
>> [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: unsuccessful
>> mount request [No such file or directory]
>>
>> The message "I [MSGID: 106496]
>> [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: Received
>> mount req" repeated 33 times between [2019-06-05 08:50:46.361347] and
>> [2019-06-05 08:52:34.019741]
>>
>> The message "E [MSGID: 106061]
>> [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option
>> mountbroker-root' missing in glusterd vol file" repeated 33 times between
>> [2019-06-05 08:50:46.361384] and [2019-06-05 08:52:34.019757]
>>
>> The message "W [MSGID: 106176]
>> [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: unsuccessful
>> mount request [No such file or directory]" repeated 33 times between
>> [2019-06-05 08:50:46.361419] and [2019-06-05 08:52:34.019758]
>>
>> [2019-06-05 08:52:44.426839] I [MSGID: 106496]
>> [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: Received
>> mount req
>>
>> [2019-06-05 08:52:44.426886] E [MSGID: 106061]
>> [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option
>> mountbroker-root' missing in glusterd vol file
>>
>> [2019-06-05 08:52:44.426896] W [MSGID: 106176]
>> [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: unsuccessful
>> mount request [No such file or directory]
>>
>
> On Wed, Jun 5, 2019 at 1:06 AM deepu srinivasan 
> wrote:
>
>> Thankyou Kotresh
>>
>> On Tue, Jun 4, 2019, 11:20 PM Kotresh Hiremath Ravishankar <
>> khire...@redhat.com> wrote:
>>
>>> Ccing Sunny, who was investing similar issue.
>>>
>>> On Tue, Jun 4, 2019 at 5:46 PM deepu srinivasan 
>>> wrote:
>>>
 Have already added the path in bashrc . Still in faulty state

 On Tue, Jun 4, 2019, 5:27 PM Kotresh Hiremath Ravishankar <
 khire...@redhat.com> wrote:

> could you please try adding /usr/sbin to $PATH for user 'sas'? If it's
> bash, add 'export PATH=/usr/sbin:$PATH' in
> /home/sas/.bashrc
>
> On Tue, Jun 4, 2019 at 5:24 PM deepu srinivasan 
> wrote:
>
>> Hi Kortesh
>> Please find the logs of the above error
>> *Master log snippet*
>>
>>> [2019-06-04 11:52:09.254731] I [resource(worker
>>> /home/sas/gluster/data/code-misc):1379:connect_remote] SSH: Initializing
>>> SSH connection between master and slave...
>>>  [2019-06-04 11:52:09.308923] D [repce(worker
>>> /home/sas/gluster/data/code-misc):196:push] RepceClient: call
>>> 89724:139652759443264:1559649129.31 __repce_version__() ...
>>>  [2019-06-04 11:52:09.602792] E [syncdutils(worker
>>> /home/sas/gluster/data/code-misc):311:log_raise_exception] :
>>> connection to peer is broken
>>>  [2019-06-04 11:52:09.603312] E [syncdutils(worker
>>> /home/sas/gluster/data/code-misc):805:errlog] Popen: command returned 
>>> error
>>>   cmd=ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i
>>> /var/lib/ glusterd/geo-replication/secret.pem -p 22 
>>> -oControlMaster=auto -S
>>> /tmp/gsyncd-aux-ssh-4aL2tc/d893f66e0addc32f7d0080bb503f5185.sock
>>> sas@192.168.185.107 /usr/libexec/glusterfs/gsyncd slave code-misc
>>> sas@   192.168.185.107::code-misc --master-node 192.168.185.106
>>> --master-node-id 

Re: [Gluster-users] Geo Replication stops replicating

[deepu srinivasan]

Thankyou Kotresh

On Tue, Jun 4, 2019, 11:20 PM Kotresh Hiremath Ravishankar <
khire...@redhat.com> wrote:

> Ccing Sunny, who was investing similar issue.
>
> On Tue, Jun 4, 2019 at 5:46 PM deepu srinivasan 
> wrote:
>
>> Have already added the path in bashrc . Still in faulty state
>>
>> On Tue, Jun 4, 2019, 5:27 PM Kotresh Hiremath Ravishankar <
>> khire...@redhat.com> wrote:
>>
>>> could you please try adding /usr/sbin to $PATH for user 'sas'? If it's
>>> bash, add 'export PATH=/usr/sbin:$PATH' in
>>> /home/sas/.bashrc
>>>
>>> On Tue, Jun 4, 2019 at 5:24 PM deepu srinivasan 
>>> wrote:
>>>
 Hi Kortesh
 Please find the logs of the above error
 *Master log snippet*

> [2019-06-04 11:52:09.254731] I [resource(worker
> /home/sas/gluster/data/code-misc):1379:connect_remote] SSH: Initializing
> SSH connection between master and slave...
>  [2019-06-04 11:52:09.308923] D [repce(worker
> /home/sas/gluster/data/code-misc):196:push] RepceClient: call
> 89724:139652759443264:1559649129.31 __repce_version__() ...
>  [2019-06-04 11:52:09.602792] E [syncdutils(worker
> /home/sas/gluster/data/code-misc):311:log_raise_exception] :
> connection to peer is broken
>  [2019-06-04 11:52:09.603312] E [syncdutils(worker
> /home/sas/gluster/data/code-misc):805:errlog] Popen: command returned 
> error
>   cmd=ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i
> /var/lib/ glusterd/geo-replication/secret.pem -p 22 -oControlMaster=auto 
> -S
> /tmp/gsyncd-aux-ssh-4aL2tc/d893f66e0addc32f7d0080bb503f5185.sock
> sas@192.168.185.107 /usr/libexec/glusterfs/gsyncd slave code-misc sas@
>   192.168.185.107::code-misc --master-node 192.168.185.106
> --master-node-id 851b64d0-d885-4ae9-9b38-ab5b15db0fec --master-brick
> /home/sas/gluster/data/code-misc --local-node 192.168.185.122 
> --local-node-
>   id bcaa7af6-c3a1-4411-8e99-4ebecb32eb6a --slave-timeout 120
> --slave-log-level DEBUG --slave-gluster-log-level INFO
> --slave-gluster-command-dir /usr/sbin   error=1
>  [2019-06-04 11:52:09.614996] I [repce(agent
> /home/sas/gluster/data/code-misc):97:service_loop] RepceServer: 
> terminating
> on reaching EOF.
>  [2019-06-04 11:52:09.615545] D [monitor(monitor):271:monitor]
> Monitor: worker(/home/sas/gluster/data/code-misc) connected
>  [2019-06-04 11:52:09.616528] I [monitor(monitor):278:monitor]
> Monitor: worker died in startup phase 
> brick=/home/sas/gluster/data/code-misc
>  [2019-06-04 11:52:09.619391] I
> [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status
> Change status=Faulty
>

 *Slave log snippet*

> [2019-06-04 11:50:09.782668] E [syncdutils(slave
> 192.168.185.106/home/sas/gluster/data/code-misc):809:logerr] Popen:
> /usr/sbin/gluster> 2 : failed with this errno (No such file or directory)
> [2019-06-04 11:50:11.188167] W [gsyncd(slave
> 192.168.185.125/home/sas/gluster/data/code-misc):305:main] :
> Session config file not exists, using the default config
> path=/var/lib/glusterd/geo-replication/code-misc_192.168.185.107_code-misc/gsyncd.conf
> [2019-06-04 11:50:11.201070] I [resource(slave
> 192.168.185.125/home/sas/gluster/data/code-misc):1098:connect]
> GLUSTER: Mounting gluster volume locally...
> [2019-06-04 11:50:11.271231] E [resource(slave
> 192.168.185.125/home/sas/gluster/data/code-misc):1006:handle_mounter]
> MountbrokerMounter: glusterd answered mnt=
> [2019-06-04 11:50:11.271998] E [syncdutils(slave
> 192.168.185.125/home/sas/gluster/data/code-misc):805:errlog] Popen:
> command returned error cmd=/usr/sbin/gluster --remote-host=localhost
> system:: mount sas user-map-root=sas aux-gfid-mount acl log-level=INFO
> log-file=/var/log/glusterfs/geo-replication-slaves/code-misc_192.168.185.107_code-misc/mnt-192.168.185.125-home-sas-gluster-data-code-misc.log
> volfile-server=localhost volfile-id=code-misc client-pid=-1 error=1
> [2019-06-04 11:50:11.272113] E [syncdutils(slave
> 192.168.185.125/home/sas/gluster/data/code-misc):809:logerr] Popen:
> /usr/sbin/gluster> 2 : failed with this errno (No such file or directory)


 On Tue, Jun 4, 2019 at 5:10 PM deepu srinivasan 
 wrote:

> Hi
> As discussed I have upgraded gluster from 4.1 to 6.2 version. But the
> Geo replication failed to start.
> Stays in faulty state
>
> On Fri, May 31, 2019, 5:32 PM deepu srinivasan 
> wrote:
>
>> Checked the data. It remains in 2708. No progress.
>>
>> On Fri, May 31, 2019 at 4:36 PM Kotresh Hiremath Ravishankar <
>> khire...@redhat.com> wrote:
>>
>>> That means it could be working and the defunct process might be some
>>> old zombie one. Could you check, that data progress ?
>>>
>>> On Fri, May 31, 2019 at 4:29 PM deepu srinivasan 
>>> wrote:
>>>

Re: [Gluster-users] Geo Replication stops replicating

[deepu srinivasan]

Hi
As discussed I have upgraded gluster from 4.1 to 6.2 version. But the Geo
replication failed to start.
Stays in faulty state

On Fri, May 31, 2019, 5:32 PM deepu srinivasan  wrote:

> Checked the data. It remains in 2708. No progress.
>
> On Fri, May 31, 2019 at 4:36 PM Kotresh Hiremath Ravishankar <
> khire...@redhat.com> wrote:
>
>> That means it could be working and the defunct process might be some old
>> zombie one. Could you check, that data progress ?
>>
>> On Fri, May 31, 2019 at 4:29 PM deepu srinivasan 
>> wrote:
>>
>>> Hi
>>> When i change the rsync option the rsync process doesnt seem to start .
>>> Only a defunt process is listed in ps aux. Only when i set rsync option to
>>> " " and restart all the process the rsync process is listed in ps aux.
>>>
>>>
>>> On Fri, May 31, 2019 at 4:23 PM Kotresh Hiremath Ravishankar <
>>> khire...@redhat.com> wrote:
>>>
 Yes, rsync config option should have fixed this issue.

 Could you share the output of the following?

 1. gluster volume geo-replication  ::
 config rsync-options
 2. ps -ef | grep rsync

 On Fri, May 31, 2019 at 4:11 PM deepu srinivasan 
 wrote:

> Done.
> We got the following result .
>
>> 1559298781.338234 write(2, "rsync: link_stat
>> \"/tmp/gsyncd-aux-mount-EEJ_sY/.gfid/3fa6aed8-802e-4efe-9903-8bc171176d88\"
>> failed: No such file or directory (2)", 128
>
> seems like a file is missing ?
>
> On Fri, May 31, 2019 at 3:25 PM Kotresh Hiremath Ravishankar <
> khire...@redhat.com> wrote:
>
>> Hi,
>>
>> Could you take the strace with with more string size? The argument
>> strings are truncated.
>>
>> strace -s 500 -ttt -T -p 
>>
>> On Fri, May 31, 2019 at 3:17 PM deepu srinivasan 
>> wrote:
>>
>>> Hi Kotresh
>>> The above-mentioned work around did not work properly.
>>>
>>> On Fri, May 31, 2019 at 3:16 PM deepu srinivasan 
>>> wrote:
>>>
 Hi Kotresh
 We have tried the above-mentioned rsync option and we are planning
 to have the version upgrade to 6.0.

 On Fri, May 31, 2019 at 11:04 AM Kotresh Hiremath Ravishankar <
 khire...@redhat.com> wrote:

> Hi,
>
> This looks like the hang because stderr buffer filled up with
> errors messages and no one reading it.
> I think this issue is fixed in latest releases. As a workaround,
> you can do following and check if it works.
>
> Prerequisite:
>  rsync version should be > 3.1.0
>
> Workaround:
> gluster volume geo-replication  ::
> config rsync-options "--ignore-missing-args"
>
> Thanks,
> Kotresh HR
>
>
>
>
> On Thu, May 30, 2019 at 5:39 PM deepu srinivasan <
> sdeep...@gmail.com> wrote:
>
>> Hi
>> We were evaluating Gluster geo Replication between two DCs one is
>> in US west and one is in US east. We took multiple trials for 
>> different
>> file size.
>> The Geo Replication tends to stop replicating but while checking
>> the status it appears to be in Active state. But the slave volume 
>> did not
>> increase in size.
>> So we have restarted the geo-replication session and checked the
>> status. The status was in an active state and it was in History 
>> Crawl for a
>> long time. We have enabled the DEBUG mode in logging and checked for 
>> any
>> error.
>> There was around 2000 file appeared for syncing candidate. The
>> Rsync process starts but the rsync did not happen in the slave 
>> volume.
>> Every time the rsync process appears in the "ps auxxx" list but the
>> replication did not happen in the slave end. What would be the cause 
>> of
>> this problem? Is there anyway to debug it?
>>
>> We have also checked the strace of the rync program.
>> it displays something like this
>>
>> "write(2, "rsync: link_stat \"/tmp/gsyncd-au"..., 128"
>>
>>
>> We are using the below specs
>>
>> Gluster version - 4.1.7
>> Sync mode - rsync
>> Volume - 1x3 in each end (master and slave)
>> Intranet Bandwidth - 10 Gig
>>
>
>
> --
> Thanks and Regards,
> Kotresh H R
>

>>
>> --
>> Thanks and Regards,
>> Kotresh H R
>>
>

 --
 Thanks and Regards,
 Kotresh H R

>>>
>>
>> --
>> Thanks and Regards,
>> Kotresh H R
>>
>
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Geo Replication stops replicating

[deepu srinivasan]

Have already added the path in bashrc . Still in faulty state

On Tue, Jun 4, 2019, 5:27 PM Kotresh Hiremath Ravishankar <
khire...@redhat.com> wrote:

> could you please try adding /usr/sbin to $PATH for user 'sas'? If it's
> bash, add 'export PATH=/usr/sbin:$PATH' in
> /home/sas/.bashrc
>
> On Tue, Jun 4, 2019 at 5:24 PM deepu srinivasan 
> wrote:
>
>> Hi Kortesh
>> Please find the logs of the above error
>> *Master log snippet*
>>
>>> [2019-06-04 11:52:09.254731] I [resource(worker
>>> /home/sas/gluster/data/code-misc):1379:connect_remote] SSH: Initializing
>>> SSH connection between master and slave...
>>>  [2019-06-04 11:52:09.308923] D [repce(worker
>>> /home/sas/gluster/data/code-misc):196:push] RepceClient: call
>>> 89724:139652759443264:1559649129.31 __repce_version__() ...
>>>  [2019-06-04 11:52:09.602792] E [syncdutils(worker
>>> /home/sas/gluster/data/code-misc):311:log_raise_exception] :
>>> connection to peer is broken
>>>  [2019-06-04 11:52:09.603312] E [syncdutils(worker
>>> /home/sas/gluster/data/code-misc):805:errlog] Popen: command returned error
>>>   cmd=ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i
>>> /var/lib/ glusterd/geo-replication/secret.pem -p 22 -oControlMaster=auto -S
>>> /tmp/gsyncd-aux-ssh-4aL2tc/d893f66e0addc32f7d0080bb503f5185.sock
>>> sas@192.168.185.107 /usr/libexec/glusterfs/gsyncd slave code-misc sas@
>>>   192.168.185.107::code-misc --master-node 192.168.185.106
>>> --master-node-id 851b64d0-d885-4ae9-9b38-ab5b15db0fec --master-brick
>>> /home/sas/gluster/data/code-misc --local-node 192.168.185.122 --local-node-
>>>   id bcaa7af6-c3a1-4411-8e99-4ebecb32eb6a --slave-timeout 120
>>> --slave-log-level DEBUG --slave-gluster-log-level INFO
>>> --slave-gluster-command-dir /usr/sbin   error=1
>>>  [2019-06-04 11:52:09.614996] I [repce(agent
>>> /home/sas/gluster/data/code-misc):97:service_loop] RepceServer: terminating
>>> on reaching EOF.
>>>  [2019-06-04 11:52:09.615545] D [monitor(monitor):271:monitor] Monitor:
>>> worker(/home/sas/gluster/data/code-misc) connected
>>>  [2019-06-04 11:52:09.616528] I [monitor(monitor):278:monitor] Monitor:
>>> worker died in startup phase brick=/home/sas/gluster/data/code-misc
>>>  [2019-06-04 11:52:09.619391] I
>>> [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status
>>> Change status=Faulty
>>>
>>
>> *Slave log snippet*
>>
>>> [2019-06-04 11:50:09.782668] E [syncdutils(slave
>>> 192.168.185.106/home/sas/gluster/data/code-misc):809:logerr] Popen:
>>> /usr/sbin/gluster> 2 : failed with this errno (No such file or directory)
>>> [2019-06-04 11:50:11.188167] W [gsyncd(slave
>>> 192.168.185.125/home/sas/gluster/data/code-misc):305:main] :
>>> Session config file not exists, using the default config
>>> path=/var/lib/glusterd/geo-replication/code-misc_192.168.185.107_code-misc/gsyncd.conf
>>> [2019-06-04 11:50:11.201070] I [resource(slave
>>> 192.168.185.125/home/sas/gluster/data/code-misc):1098:connect] GLUSTER:
>>> Mounting gluster volume locally...
>>> [2019-06-04 11:50:11.271231] E [resource(slave
>>> 192.168.185.125/home/sas/gluster/data/code-misc):1006:handle_mounter]
>>> MountbrokerMounter: glusterd answered mnt=
>>> [2019-06-04 11:50:11.271998] E [syncdutils(slave
>>> 192.168.185.125/home/sas/gluster/data/code-misc):805:errlog] Popen:
>>> command returned error cmd=/usr/sbin/gluster --remote-host=localhost
>>> system:: mount sas user-map-root=sas aux-gfid-mount acl log-level=INFO
>>> log-file=/var/log/glusterfs/geo-replication-slaves/code-misc_192.168.185.107_code-misc/mnt-192.168.185.125-home-sas-gluster-data-code-misc.log
>>> volfile-server=localhost volfile-id=code-misc client-pid=-1 error=1
>>> [2019-06-04 11:50:11.272113] E [syncdutils(slave
>>> 192.168.185.125/home/sas/gluster/data/code-misc):809:logerr] Popen:
>>> /usr/sbin/gluster> 2 : failed with this errno (No such file or directory)
>>
>>
>> On Tue, Jun 4, 2019 at 5:10 PM deepu srinivasan 
>> wrote:
>>
>>> Hi
>>> As discussed I have upgraded gluster from 4.1 to 6.2 version. But the
>>> Geo replication failed to start.
>>> Stays in faulty state
>>>
>>> On Fri, May 31, 2019, 5:32 PM deepu srinivasan 
>>> wrote:
>>>
 Checked the data. It remains in 2708. No progress.

 On Fri, May 31, 2019 at 4:36 PM Kotresh Hiremath Ravishankar <
 khire...@redhat.com> wrote:

> That means it could be working and the defunct process might be some
> old zombie one. Could you check, that data progress ?
>
> On Fri, May 31, 2019 at 4:29 PM deepu srinivasan 
> wrote:
>
>> Hi
>> When i change the rsync option the rsync process doesnt seem to start
>> . Only a defunt process is listed in ps aux. Only when i set rsync option
>> to " " and restart all the process the rsync process is listed in ps aux.
>>
>>
>> On Fri, May 31, 2019 at 4:23 PM Kotresh Hiremath Ravishankar <
>> khire...@redhat.com> wrote:
>>
>>> Yes, rsync config option should have fixed this issue.
>>>
>>> 

Re: [Gluster-users] Geo Replication stops replicating

[deepu srinivasan]

Hi Kortesh
Please find the logs of the above error
*Master log snippet*

> [2019-06-04 11:52:09.254731] I [resource(worker
> /home/sas/gluster/data/code-misc):1379:connect_remote] SSH: Initializing
> SSH connection between master and slave...
>  [2019-06-04 11:52:09.308923] D [repce(worker
> /home/sas/gluster/data/code-misc):196:push] RepceClient: call
> 89724:139652759443264:1559649129.31 __repce_version__() ...
>  [2019-06-04 11:52:09.602792] E [syncdutils(worker
> /home/sas/gluster/data/code-misc):311:log_raise_exception] :
> connection to peer is broken
>  [2019-06-04 11:52:09.603312] E [syncdutils(worker
> /home/sas/gluster/data/code-misc):805:errlog] Popen: command returned error
>   cmd=ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i
> /var/lib/ glusterd/geo-replication/secret.pem -p 22 -oControlMaster=auto -S
> /tmp/gsyncd-aux-ssh-4aL2tc/d893f66e0addc32f7d0080bb503f5185.sock
> sas@192.168.185.107 /usr/libexec/glusterfs/gsyncd slave code-misc sas@
> 192.168.185.107::code-misc --master-node 192.168.185.106 --master-node-id
> 851b64d0-d885-4ae9-9b38-ab5b15db0fec --master-brick
> /home/sas/gluster/data/code-misc --local-node 192.168.185.122 --local-node-
>   id bcaa7af6-c3a1-4411-8e99-4ebecb32eb6a --slave-timeout 120
> --slave-log-level DEBUG --slave-gluster-log-level INFO
> --slave-gluster-command-dir /usr/sbin   error=1
>  [2019-06-04 11:52:09.614996] I [repce(agent
> /home/sas/gluster/data/code-misc):97:service_loop] RepceServer: terminating
> on reaching EOF.
>  [2019-06-04 11:52:09.615545] D [monitor(monitor):271:monitor] Monitor:
> worker(/home/sas/gluster/data/code-misc) connected
>  [2019-06-04 11:52:09.616528] I [monitor(monitor):278:monitor] Monitor:
> worker died in startup phase brick=/home/sas/gluster/data/code-misc
>  [2019-06-04 11:52:09.619391] I
> [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status
> Change status=Faulty
>

*Slave log snippet*

> [2019-06-04 11:50:09.782668] E [syncdutils(slave
> 192.168.185.106/home/sas/gluster/data/code-misc):809:logerr] Popen:
> /usr/sbin/gluster> 2 : failed with this errno (No such file or directory)
> [2019-06-04 11:50:11.188167] W [gsyncd(slave
> 192.168.185.125/home/sas/gluster/data/code-misc):305:main] : Session
> config file not exists, using the default config
> path=/var/lib/glusterd/geo-replication/code-misc_192.168.185.107_code-misc/gsyncd.conf
> [2019-06-04 11:50:11.201070] I [resource(slave
> 192.168.185.125/home/sas/gluster/data/code-misc):1098:connect] GLUSTER:
> Mounting gluster volume locally...
> [2019-06-04 11:50:11.271231] E [resource(slave
> 192.168.185.125/home/sas/gluster/data/code-misc):1006:handle_mounter]
> MountbrokerMounter: glusterd answered mnt=
> [2019-06-04 11:50:11.271998] E [syncdutils(slave
> 192.168.185.125/home/sas/gluster/data/code-misc):805:errlog] Popen:
> command returned error cmd=/usr/sbin/gluster --remote-host=localhost
> system:: mount sas user-map-root=sas aux-gfid-mount acl log-level=INFO
> log-file=/var/log/glusterfs/geo-replication-slaves/code-misc_192.168.185.107_code-misc/mnt-192.168.185.125-home-sas-gluster-data-code-misc.log
> volfile-server=localhost volfile-id=code-misc client-pid=-1 error=1
> [2019-06-04 11:50:11.272113] E [syncdutils(slave
> 192.168.185.125/home/sas/gluster/data/code-misc):809:logerr] Popen:
> /usr/sbin/gluster> 2 : failed with this errno (No such file or directory)


On Tue, Jun 4, 2019 at 5:10 PM deepu srinivasan  wrote:

> Hi
> As discussed I have upgraded gluster from 4.1 to 6.2 version. But the Geo
> replication failed to start.
> Stays in faulty state
>
> On Fri, May 31, 2019, 5:32 PM deepu srinivasan  wrote:
>
>> Checked the data. It remains in 2708. No progress.
>>
>> On Fri, May 31, 2019 at 4:36 PM Kotresh Hiremath Ravishankar <
>> khire...@redhat.com> wrote:
>>
>>> That means it could be working and the defunct process might be some old
>>> zombie one. Could you check, that data progress ?
>>>
>>> On Fri, May 31, 2019 at 4:29 PM deepu srinivasan 
>>> wrote:
>>>
 Hi
 When i change the rsync option the rsync process doesnt seem to start .
 Only a defunt process is listed in ps aux. Only when i set rsync option to
 " " and restart all the process the rsync process is listed in ps aux.


 On Fri, May 31, 2019 at 4:23 PM Kotresh Hiremath Ravishankar <
 khire...@redhat.com> wrote:

> Yes, rsync config option should have fixed this issue.
>
> Could you share the output of the following?
>
> 1. gluster volume geo-replication  ::
> config rsync-options
> 2. ps -ef | grep rsync
>
> On Fri, May 31, 2019 at 4:11 PM deepu srinivasan 
> wrote:
>
>> Done.
>> We got the following result .
>>
>>> 1559298781.338234 write(2, "rsync: link_stat
>>> \"/tmp/gsyncd-aux-mount-EEJ_sY/.gfid/3fa6aed8-802e-4efe-9903-8bc171176d88\"
>>> failed: No such file or directory (2)", 128
>>
>> seems like a file is missing ?
>>
>> On Fri, May 31, 2019 

Re: [Gluster-users] Geo Replication stops replicating

[deepu srinivasan]

Checked the data. It remains in 2708. No progress.

On Fri, May 31, 2019 at 4:36 PM Kotresh Hiremath Ravishankar <
khire...@redhat.com> wrote:

> That means it could be working and the defunct process might be some old
> zombie one. Could you check, that data progress ?
>
> On Fri, May 31, 2019 at 4:29 PM deepu srinivasan 
> wrote:
>
>> Hi
>> When i change the rsync option the rsync process doesnt seem to start .
>> Only a defunt process is listed in ps aux. Only when i set rsync option to
>> " " and restart all the process the rsync process is listed in ps aux.
>>
>>
>> On Fri, May 31, 2019 at 4:23 PM Kotresh Hiremath Ravishankar <
>> khire...@redhat.com> wrote:
>>
>>> Yes, rsync config option should have fixed this issue.
>>>
>>> Could you share the output of the following?
>>>
>>> 1. gluster volume geo-replication  ::
>>> config rsync-options
>>> 2. ps -ef | grep rsync
>>>
>>> On Fri, May 31, 2019 at 4:11 PM deepu srinivasan 
>>> wrote:
>>>
 Done.
 We got the following result .

> 1559298781.338234 write(2, "rsync: link_stat
> \"/tmp/gsyncd-aux-mount-EEJ_sY/.gfid/3fa6aed8-802e-4efe-9903-8bc171176d88\"
> failed: No such file or directory (2)", 128

 seems like a file is missing ?

 On Fri, May 31, 2019 at 3:25 PM Kotresh Hiremath Ravishankar <
 khire...@redhat.com> wrote:

> Hi,
>
> Could you take the strace with with more string size? The argument
> strings are truncated.
>
> strace -s 500 -ttt -T -p 
>
> On Fri, May 31, 2019 at 3:17 PM deepu srinivasan 
> wrote:
>
>> Hi Kotresh
>> The above-mentioned work around did not work properly.
>>
>> On Fri, May 31, 2019 at 3:16 PM deepu srinivasan 
>> wrote:
>>
>>> Hi Kotresh
>>> We have tried the above-mentioned rsync option and we are planning
>>> to have the version upgrade to 6.0.
>>>
>>> On Fri, May 31, 2019 at 11:04 AM Kotresh Hiremath Ravishankar <
>>> khire...@redhat.com> wrote:
>>>
 Hi,

 This looks like the hang because stderr buffer filled up with
 errors messages and no one reading it.
 I think this issue is fixed in latest releases. As a workaround,
 you can do following and check if it works.

 Prerequisite:
  rsync version should be > 3.1.0

 Workaround:
 gluster volume geo-replication  ::
 config rsync-options "--ignore-missing-args"

 Thanks,
 Kotresh HR




 On Thu, May 30, 2019 at 5:39 PM deepu srinivasan <
 sdeep...@gmail.com> wrote:

> Hi
> We were evaluating Gluster geo Replication between two DCs one is
> in US west and one is in US east. We took multiple trials for 
> different
> file size.
> The Geo Replication tends to stop replicating but while checking
> the status it appears to be in Active state. But the slave volume did 
> not
> increase in size.
> So we have restarted the geo-replication session and checked the
> status. The status was in an active state and it was in History Crawl 
> for a
> long time. We have enabled the DEBUG mode in logging and checked for 
> any
> error.
> There was around 2000 file appeared for syncing candidate. The
> Rsync process starts but the rsync did not happen in the slave volume.
> Every time the rsync process appears in the "ps auxxx" list but the
> replication did not happen in the slave end. What would be the cause 
> of
> this problem? Is there anyway to debug it?
>
> We have also checked the strace of the rync program.
> it displays something like this
>
> "write(2, "rsync: link_stat \"/tmp/gsyncd-au"..., 128"
>
>
> We are using the below specs
>
> Gluster version - 4.1.7
> Sync mode - rsync
> Volume - 1x3 in each end (master and slave)
> Intranet Bandwidth - 10 Gig
>


 --
 Thanks and Regards,
 Kotresh H R

>>>
>
> --
> Thanks and Regards,
> Kotresh H R
>

>>>
>>> --
>>> Thanks and Regards,
>>> Kotresh H R
>>>
>>
>
> --
> Thanks and Regards,
> Kotresh H R
>
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Geo Replication stops replicating

[deepu srinivasan]

Hi
When i change the rsync option the rsync process doesnt seem to start .
Only a defunt process is listed in ps aux. Only when i set rsync option to
" " and restart all the process the rsync process is listed in ps aux.


On Fri, May 31, 2019 at 4:23 PM Kotresh Hiremath Ravishankar <
khire...@redhat.com> wrote:

> Yes, rsync config option should have fixed this issue.
>
> Could you share the output of the following?
>
> 1. gluster volume geo-replication  ::
> config rsync-options
> 2. ps -ef | grep rsync
>
> On Fri, May 31, 2019 at 4:11 PM deepu srinivasan 
> wrote:
>
>> Done.
>> We got the following result .
>>
>>> 1559298781.338234 write(2, "rsync: link_stat
>>> \"/tmp/gsyncd-aux-mount-EEJ_sY/.gfid/3fa6aed8-802e-4efe-9903-8bc171176d88\"
>>> failed: No such file or directory (2)", 128
>>
>> seems like a file is missing ?
>>
>> On Fri, May 31, 2019 at 3:25 PM Kotresh Hiremath Ravishankar <
>> khire...@redhat.com> wrote:
>>
>>> Hi,
>>>
>>> Could you take the strace with with more string size? The argument
>>> strings are truncated.
>>>
>>> strace -s 500 -ttt -T -p 
>>>
>>> On Fri, May 31, 2019 at 3:17 PM deepu srinivasan 
>>> wrote:
>>>
 Hi Kotresh
 The above-mentioned work around did not work properly.

 On Fri, May 31, 2019 at 3:16 PM deepu srinivasan 
 wrote:

> Hi Kotresh
> We have tried the above-mentioned rsync option and we are planning to
> have the version upgrade to 6.0.
>
> On Fri, May 31, 2019 at 11:04 AM Kotresh Hiremath Ravishankar <
> khire...@redhat.com> wrote:
>
>> Hi,
>>
>> This looks like the hang because stderr buffer filled up with errors
>> messages and no one reading it.
>> I think this issue is fixed in latest releases. As a workaround, you
>> can do following and check if it works.
>>
>> Prerequisite:
>>  rsync version should be > 3.1.0
>>
>> Workaround:
>> gluster volume geo-replication  ::
>> config rsync-options "--ignore-missing-args"
>>
>> Thanks,
>> Kotresh HR
>>
>>
>>
>>
>> On Thu, May 30, 2019 at 5:39 PM deepu srinivasan 
>> wrote:
>>
>>> Hi
>>> We were evaluating Gluster geo Replication between two DCs one is in
>>> US west and one is in US east. We took multiple trials for different 
>>> file
>>> size.
>>> The Geo Replication tends to stop replicating but while checking the
>>> status it appears to be in Active state. But the slave volume did not
>>> increase in size.
>>> So we have restarted the geo-replication session and checked the
>>> status. The status was in an active state and it was in History Crawl 
>>> for a
>>> long time. We have enabled the DEBUG mode in logging and checked for any
>>> error.
>>> There was around 2000 file appeared for syncing candidate. The Rsync
>>> process starts but the rsync did not happen in the slave volume. Every 
>>> time
>>> the rsync process appears in the "ps auxxx" list but the replication did
>>> not happen in the slave end. What would be the cause of this problem? Is
>>> there anyway to debug it?
>>>
>>> We have also checked the strace of the rync program.
>>> it displays something like this
>>>
>>> "write(2, "rsync: link_stat \"/tmp/gsyncd-au"..., 128"
>>>
>>>
>>> We are using the below specs
>>>
>>> Gluster version - 4.1.7
>>> Sync mode - rsync
>>> Volume - 1x3 in each end (master and slave)
>>> Intranet Bandwidth - 10 Gig
>>>
>>
>>
>> --
>> Thanks and Regards,
>> Kotresh H R
>>
>
>>>
>>> --
>>> Thanks and Regards,
>>> Kotresh H R
>>>
>>
>
> --
> Thanks and Regards,
> Kotresh H R
>
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Geo Replication stops replicating

[deepu srinivasan]

Done.
We got the following result .

> 1559298781.338234 write(2, "rsync: link_stat
> \"/tmp/gsyncd-aux-mount-EEJ_sY/.gfid/3fa6aed8-802e-4efe-9903-8bc171176d88\"
> failed: No such file or directory (2)", 128

seems like a file is missing ?

On Fri, May 31, 2019 at 3:25 PM Kotresh Hiremath Ravishankar <
khire...@redhat.com> wrote:

> Hi,
>
> Could you take the strace with with more string size? The argument strings
> are truncated.
>
> strace -s 500 -ttt -T -p 
>
> On Fri, May 31, 2019 at 3:17 PM deepu srinivasan 
> wrote:
>
>> Hi Kotresh
>> The above-mentioned work around did not work properly.
>>
>> On Fri, May 31, 2019 at 3:16 PM deepu srinivasan 
>> wrote:
>>
>>> Hi Kotresh
>>> We have tried the above-mentioned rsync option and we are planning to
>>> have the version upgrade to 6.0.
>>>
>>> On Fri, May 31, 2019 at 11:04 AM Kotresh Hiremath Ravishankar <
>>> khire...@redhat.com> wrote:
>>>
 Hi,

 This looks like the hang because stderr buffer filled up with errors
 messages and no one reading it.
 I think this issue is fixed in latest releases. As a workaround, you
 can do following and check if it works.

 Prerequisite:
  rsync version should be > 3.1.0

 Workaround:
 gluster volume geo-replication  ::
 config rsync-options "--ignore-missing-args"

 Thanks,
 Kotresh HR




 On Thu, May 30, 2019 at 5:39 PM deepu srinivasan 
 wrote:

> Hi
> We were evaluating Gluster geo Replication between two DCs one is in
> US west and one is in US east. We took multiple trials for different file
> size.
> The Geo Replication tends to stop replicating but while checking the
> status it appears to be in Active state. But the slave volume did not
> increase in size.
> So we have restarted the geo-replication session and checked the
> status. The status was in an active state and it was in History Crawl for 
> a
> long time. We have enabled the DEBUG mode in logging and checked for any
> error.
> There was around 2000 file appeared for syncing candidate. The Rsync
> process starts but the rsync did not happen in the slave volume. Every 
> time
> the rsync process appears in the "ps auxxx" list but the replication did
> not happen in the slave end. What would be the cause of this problem? Is
> there anyway to debug it?
>
> We have also checked the strace of the rync program.
> it displays something like this
>
> "write(2, "rsync: link_stat \"/tmp/gsyncd-au"..., 128"
>
>
> We are using the below specs
>
> Gluster version - 4.1.7
> Sync mode - rsync
> Volume - 1x3 in each end (master and slave)
> Intranet Bandwidth - 10 Gig
>


 --
 Thanks and Regards,
 Kotresh H R

>>>
>
> --
> Thanks and Regards,
> Kotresh H R
>
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Geo Replication stops replicating

[deepu srinivasan]

Hi Kotresh
We have tried the above-mentioned rsync option and we are planning to have
the version upgrade to 6.0.

On Fri, May 31, 2019 at 11:04 AM Kotresh Hiremath Ravishankar <
khire...@redhat.com> wrote:

> Hi,
>
> This looks like the hang because stderr buffer filled up with errors
> messages and no one reading it.
> I think this issue is fixed in latest releases. As a workaround, you can
> do following and check if it works.
>
> Prerequisite:
>  rsync version should be > 3.1.0
>
> Workaround:
> gluster volume geo-replication  :: config
> rsync-options "--ignore-missing-args"
>
> Thanks,
> Kotresh HR
>
>
>
>
> On Thu, May 30, 2019 at 5:39 PM deepu srinivasan 
> wrote:
>
>> Hi
>> We were evaluating Gluster geo Replication between two DCs one is in US
>> west and one is in US east. We took multiple trials for different file
>> size.
>> The Geo Replication tends to stop replicating but while checking the
>> status it appears to be in Active state. But the slave volume did not
>> increase in size.
>> So we have restarted the geo-replication session and checked the status.
>> The status was in an active state and it was in History Crawl for a long
>> time. We have enabled the DEBUG mode in logging and checked for any error.
>> There was around 2000 file appeared for syncing candidate. The Rsync
>> process starts but the rsync did not happen in the slave volume. Every time
>> the rsync process appears in the "ps auxxx" list but the replication did
>> not happen in the slave end. What would be the cause of this problem? Is
>> there anyway to debug it?
>>
>> We have also checked the strace of the rync program.
>> it displays something like this
>>
>> "write(2, "rsync: link_stat \"/tmp/gsyncd-au"..., 128"
>>
>>
>> We are using the below specs
>>
>> Gluster version - 4.1.7
>> Sync mode - rsync
>> Volume - 1x3 in each end (master and slave)
>> Intranet Bandwidth - 10 Gig
>>
>
>
> --
> Thanks and Regards,
> Kotresh H R
>
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Geo Replication stops replicating

[deepu srinivasan]

Hi Kotresh
The above-mentioned work around did not work properly.

On Fri, May 31, 2019 at 3:16 PM deepu srinivasan  wrote:

> Hi Kotresh
> We have tried the above-mentioned rsync option and we are planning to have
> the version upgrade to 6.0.
>
> On Fri, May 31, 2019 at 11:04 AM Kotresh Hiremath Ravishankar <
> khire...@redhat.com> wrote:
>
>> Hi,
>>
>> This looks like the hang because stderr buffer filled up with errors
>> messages and no one reading it.
>> I think this issue is fixed in latest releases. As a workaround, you can
>> do following and check if it works.
>>
>> Prerequisite:
>>  rsync version should be > 3.1.0
>>
>> Workaround:
>> gluster volume geo-replication  :: config
>> rsync-options "--ignore-missing-args"
>>
>> Thanks,
>> Kotresh HR
>>
>>
>>
>>
>> On Thu, May 30, 2019 at 5:39 PM deepu srinivasan 
>> wrote:
>>
>>> Hi
>>> We were evaluating Gluster geo Replication between two DCs one is in US
>>> west and one is in US east. We took multiple trials for different file
>>> size.
>>> The Geo Replication tends to stop replicating but while checking the
>>> status it appears to be in Active state. But the slave volume did not
>>> increase in size.
>>> So we have restarted the geo-replication session and checked the status.
>>> The status was in an active state and it was in History Crawl for a long
>>> time. We have enabled the DEBUG mode in logging and checked for any error.
>>> There was around 2000 file appeared for syncing candidate. The Rsync
>>> process starts but the rsync did not happen in the slave volume. Every time
>>> the rsync process appears in the "ps auxxx" list but the replication did
>>> not happen in the slave end. What would be the cause of this problem? Is
>>> there anyway to debug it?
>>>
>>> We have also checked the strace of the rync program.
>>> it displays something like this
>>>
>>> "write(2, "rsync: link_stat \"/tmp/gsyncd-au"..., 128"
>>>
>>>
>>> We are using the below specs
>>>
>>> Gluster version - 4.1.7
>>> Sync mode - rsync
>>> Volume - 1x3 in each end (master and slave)
>>> Intranet Bandwidth - 10 Gig
>>>
>>
>>
>> --
>> Thanks and Regards,
>> Kotresh H R
>>
>
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Geo Replication stops replicating

[Sunny Kumar]

You should not have used this one:
>
> gluster-mountbroker remove --volume code-misc --user sas

-- This one is to remove volume/user from mount broker.

Please try setting up mount broker once again.

-Sunny

On Thu, Jun 6, 2019 at 5:28 PM deepu srinivasan  wrote:
>
> Hi Sunny
> Please find the logs attached
>>
>> The message "E [MSGID: 106061] 
>> [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option 
>> mountbroker-root' missing in glusterd vol file" repeated 13 times between 
>> [2019-06-06 11:51:43.986788] and [2019-06-06 11:52:32.764546]
>>
>> The message "W [MSGID: 106176] 
>> [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: unsuccessful 
>> mount request [No such file or directory]" repeated 13 times between 
>> [2019-06-06 11:51:43.986798] and [2019-06-06 11:52:32.764548]
>>
>> The message "I [MSGID: 106488] 
>> [glusterd-handler.c:1559:__glusterd_handle_cli_get_volume] 0-management: 
>> Received get vol req" repeated 2 times between [2019-06-06 11:53:07.064332] 
>> and [2019-06-06 11:53:07.303978]
>>
>> [2019-06-06 11:55:35.624320] I [MSGID: 106495] 
>> [glusterd-handler.c:3137:__glusterd_handle_getwd] 0-glusterd: Received getwd 
>> req
>>
>> [2019-06-06 11:55:35.884345] I [MSGID: 106131] 
>> [glusterd-proc-mgmt.c:86:glusterd_proc_stop] 0-management: quotad already 
>> stopped
>>
>> [2019-06-06 11:55:35.884373] I [MSGID: 106568] 
>> [glusterd-svc-mgmt.c:253:glusterd_svc_stop] 0-management: quotad service is 
>> stopped
>>
>> [2019-06-06 11:55:35.884459] I [MSGID: 106131] 
>> [glusterd-proc-mgmt.c:86:glusterd_proc_stop] 0-management: bitd already 
>> stopped
>>
>> [2019-06-06 11:55:35.884473] I [MSGID: 106568] 
>> [glusterd-svc-mgmt.c:253:glusterd_svc_stop] 0-management: bitd service is 
>> stopped
>>
>> [2019-06-06 11:55:35.884554] I [MSGID: 106131] 
>> [glusterd-proc-mgmt.c:86:glusterd_proc_stop] 0-management: scrub already 
>> stopped
>>
>> [2019-06-06 11:55:35.884567] I [MSGID: 106568] 
>> [glusterd-svc-mgmt.c:253:glusterd_svc_stop] 0-management: scrub service is 
>> stopped
>>
>> [2019-06-06 11:55:35.893823] I [run.c:242:runner_log] 
>> (-->/usr/lib64/glusterfs/6.2/xlator/mgmt/glusterd.so(+0xe8e1a) 
>> [0x7f7380d60e1a] 
>> -->/usr/lib64/glusterfs/6.2/xlator/mgmt/glusterd.so(+0xe88e5) 
>> [0x7f7380d608e5] -->/lib64/libglusterfs.so.0(runner_log+0x115) 
>> [0x7f738cbc5df5] ) 0-management: Ran script: 
>> /var/lib/glusterd/hooks/1/set/post/S30samba-set.sh --volname=code-misc -o 
>> features.read-only=on --gd-workdir=/var/lib/glusterd
>>
>> [2019-06-06 11:55:35.900465] I [run.c:242:runner_log] 
>> (-->/usr/lib64/glusterfs/6.2/xlator/mgmt/glusterd.so(+0xe8e1a) 
>> [0x7f7380d60e1a] 
>> -->/usr/lib64/glusterfs/6.2/xlator/mgmt/glusterd.so(+0xe88e5) 
>> [0x7f7380d608e5] -->/lib64/libglusterfs.so.0(runner_log+0x115) 
>> [0x7f738cbc5df5] ) 0-management: Ran script: 
>> /var/lib/glusterd/hooks/1/set/post/S32gluster_enable_shared_storage.sh 
>> --volname=code-misc -o features.read-only=on --gd-workdir=/var/lib/glusterd
>>
>> [2019-06-06 11:55:43.485284] I [MSGID: 106488] 
>> [glusterd-handler.c:1559:__glusterd_handle_cli_get_volume] 0-management: 
>> Received get vol req
>>
>> The message "I [MSGID: 106488] 
>> [glusterd-handler.c:1559:__glusterd_handle_cli_get_volume] 0-management: 
>> Received get vol req" repeated 2 times between [2019-06-06 11:55:43.485284] 
>> and [2019-06-06 11:55:43.512321]
>>
>> [2019-06-06 11:55:44.055419] I [MSGID: 106496] 
>> [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: Received mount 
>> req
>>
>> [2019-06-06 11:55:44.055473] E [MSGID: 106061] 
>> [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option 
>> mountbroker-root' missing in glusterd vol file
>>
>> [2019-06-06 11:55:44.055483] W [MSGID: 106176] 
>> [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: unsuccessful 
>> mount request [No such file or directory]
>>
>> [2019-06-06 11:55:44.056695] I [MSGID: 106496] 
>> [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: Received mount 
>> req
>>
>> [2019-06-06 11:55:44.056725] E [MSGID: 106061] 
>> [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option 
>> mountbroker-root' missing in glusterd vol file
>>
>> [2019-06-06 11:55:44.056734] W [MSGID: 106176] 
>> [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: unsuccessful 
>> mount request [No such file or directory]
>>
>> [2019-06-06 11:55:44.057522] I [MSGID: 106496] 
>> [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: Received mount 
>> req
>>
>> [2019-06-06 11:55:44.057552] E [MSGID: 106061] 
>> [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option 
>> mountbroker-root' missing in glusterd vol file
>>
>> [2019-06-06 11:55:44.057562] W [MSGID: 106176] 
>> [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: unsuccessful 
>> mount request [No such file or directory]
>>
>> [2019-06-06 11:55:54.655681] I [MSGID: 106496] 
>> [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: 

Re: [Gluster-users] Geo Replication stops replicating

[Sunny Kumar]

Whats current trackback please share.

-Sunny


On Thu, Jun 6, 2019 at 4:53 PM deepu srinivasan  wrote:
>
> Hi Sunny
> I have changed the file in /usr/libexec/glusterfs/peer_mountbroker.py as 
> mentioned in the patch.
> Now the "gluster-mountbroker status" command is working fine. But the 
> geo-replication seems to be in the faulty state still.
>
>
> Thankyou
> Deepak
>
> On Thu, Jun 6, 2019 at 4:10 PM Sunny Kumar  wrote:
>>
>> Above error can be tracked here:
>>
>> https://bugzilla.redhat.com/show_bug.cgi?id=1709248
>>
>> and patch link:
>> https://review.gluster.org/#/c/glusterfs/+/22716/
>>
>> You can apply patch and test it however its waiting on regression to
>> pass and merge.
>>
>> -Sunny
>>
>>
>> On Thu, Jun 6, 2019 at 4:00 PM deepu srinivasan  wrote:
>> >
>> > Hi
>> > I have followed the following steps to create the geo-replication but the 
>> > status seems to be in a faulty state.
>> >
>> > Steps :
>> >
>> > Installed cluster version 5.6 in totally six nodes.
>> >>
>> >> glusterfs 5.6
>> >>
>> >> Repository revision: git://git.gluster.org/glusterfs.git
>> >>
>> >> Copyright (c) 2006-2016 Red Hat, Inc. 
>> >>
>> >> GlusterFS comes with ABSOLUTELY NO WARRANTY.
>> >>
>> >> It is licensed to you under your choice of the GNU Lesser
>> >>
>> >> General Public License, version 3 or any later version (LGPLv3
>> >>
>> >> or later), or the GNU General Public License, version 2 (GPLv2),
>> >>
>> >> in all cases as published by the Free Software Foundation
>> >
>> >
>> > peer_probed the first three nodes and second three nodes.
>> >
>> >
>> >
>> > Added new volume in both the clusters
>> >
>> >
>> >
>> > execute gluster-mountbroker commands and restarted glusterd.
>> >>
>> >> gluster-mountbroker setup /var/mountbroker-root sas
>> >>
>> >> gluster-mountbroker remove --volume code-misc --user sas
>> >
>> >
>> > configured a passwordless sssh from master to slave
>> >>
>> >> ssh-keygen; ssh-copy-id sas@192.168.185.107
>> >
>> > created a common pem pub file
>> >>
>> >> gluster system:: execute gsec_create
>> >
>> > created geo-replication session.
>> >>
>> >> gluster volume geo-replication code-misc sas@192.168.185.107::code-misc 
>> >> create push-pem
>> >
>> >  executed the following command in slave
>> >>
>> >> /usr/libexec/glusterfs/set_geo_rep_pem_keys.sh sas code-misc code-misc
>> >
>> > started the gluster geo-replication.
>> >>
>> >> gluster volume geo-replication code-misc sas@192.168.185.107::code-misc 
>> >> start
>> >
>> >
>> > Now the geo-replication works fine.
>> > Tested with 2000 files All seems to sync finely.
>> >
>> > Now I updated all the node to version 6.2 by using rpms which were built 
>> > by the source code in a docker container in my personal machine.
>> >
>> >
>> >> gluster --version
>> >>
>> >> glusterfs 6.2
>> >>
>> >> Repository revision: git://git.gluster.org/glusterfs.git
>> >>
>> >> Copyright (c) 2006-2016 Red Hat, Inc. 
>> >>
>> >> GlusterFS comes with ABSOLUTELY NO WARRANTY.
>> >>
>> >> It is licensed to you under your choice of the GNU Lesser
>> >>
>> >> General Public License, version 3 or any later version (LGPLv3
>> >>
>> >> or later), or the GNU General Public License, version 2 (GPLv2),
>> >>
>> >> in all cases as published by the Free Software Foundation.
>> >
>> >
>> > I have stopped the glusterd daemons in all the node along with the volume 
>> > and geo-replication.
>> > Now I started the daemons, volume and geo-replication session the status 
>> > seems to be faulty.
>> > Also noted that the result of "gluster-mountbroker status" command always 
>> > end in python exception like this
>> >>
>> >> Traceback (most recent call last):
>> >>
>> >>   File "/usr/sbin/gluster-mountbroker", line 396, in 
>> >>
>> >> runcli()
>> >>
>> >>   File "/usr/lib/python2.7/site-packages/gluster/cliutils/cliutils.py", 
>> >> line 225, in runcli
>> >>
>> >> cls.run(args)
>> >>
>> >>   File "/usr/sbin/gluster-mountbroker", line 275, in run
>> >>
>> >> out = execute_in_peers("node-status")
>> >>
>> >>   File "/usr/lib/python2.7/site-packages/gluster/cliutils/cliutils.py", 
>> >> line 127, in execute_in_peers
>> >>
>> >> raise GlusterCmdException((rc, out, err, " ".join(cmd)))
>> >>
>> >> gluster.cliutils.cliutils.GlusterCmdException: (1, '', 'Unable to end. 
>> >> Error : Success\n', 'gluster system:: execute mountbroker.py node-status')
>> >
>> >
>> > Is it I or everyone gets an error for gluster-mountbroker command for 
>> > gluster version greater than 6.0?. Please help.
>> >
>> > Thank you
>> > Deepak
>> >
>> >
>> > On Thu, Jun 6, 2019 at 10:35 AM Sunny Kumar  wrote:
>> >>
>> >> Hi,
>> >>
>> >> Updated link for documentation :
>> >>
>> >> --  
>> >> https://docs.gluster.org/en/latest/Administrator%20Guide/Geo%20Replication/
>> >>
>> >> You can use this tool as well:
>> >> http://aravindavk.in/blog/gluster-georep-tools/
>> >>
>> >> -Sunny
>> >>
>> >> On Thu, Jun 6, 2019 at 10:29 AM Kotresh Hiremath Ravishankar
>> 

Re: [Gluster-users] Geo Replication stops replicating

[Sunny Kumar]

Above error can be tracked here:

https://bugzilla.redhat.com/show_bug.cgi?id=1709248

and patch link:
https://review.gluster.org/#/c/glusterfs/+/22716/

You can apply patch and test it however its waiting on regression to
pass and merge.

-Sunny


On Thu, Jun 6, 2019 at 4:00 PM deepu srinivasan  wrote:
>
> Hi
> I have followed the following steps to create the geo-replication but the 
> status seems to be in a faulty state.
>
> Steps :
>
> Installed cluster version 5.6 in totally six nodes.
>>
>> glusterfs 5.6
>>
>> Repository revision: git://git.gluster.org/glusterfs.git
>>
>> Copyright (c) 2006-2016 Red Hat, Inc. 
>>
>> GlusterFS comes with ABSOLUTELY NO WARRANTY.
>>
>> It is licensed to you under your choice of the GNU Lesser
>>
>> General Public License, version 3 or any later version (LGPLv3
>>
>> or later), or the GNU General Public License, version 2 (GPLv2),
>>
>> in all cases as published by the Free Software Foundation
>
>
> peer_probed the first three nodes and second three nodes.
>
>
>
> Added new volume in both the clusters
>
>
>
> execute gluster-mountbroker commands and restarted glusterd.
>>
>> gluster-mountbroker setup /var/mountbroker-root sas
>>
>> gluster-mountbroker remove --volume code-misc --user sas
>
>
> configured a passwordless sssh from master to slave
>>
>> ssh-keygen; ssh-copy-id sas@192.168.185.107
>
> created a common pem pub file
>>
>> gluster system:: execute gsec_create
>
> created geo-replication session.
>>
>> gluster volume geo-replication code-misc sas@192.168.185.107::code-misc 
>> create push-pem
>
>  executed the following command in slave
>>
>> /usr/libexec/glusterfs/set_geo_rep_pem_keys.sh sas code-misc code-misc
>
> started the gluster geo-replication.
>>
>> gluster volume geo-replication code-misc sas@192.168.185.107::code-misc start
>
>
> Now the geo-replication works fine.
> Tested with 2000 files All seems to sync finely.
>
> Now I updated all the node to version 6.2 by using rpms which were built by 
> the source code in a docker container in my personal machine.
>
>
>> gluster --version
>>
>> glusterfs 6.2
>>
>> Repository revision: git://git.gluster.org/glusterfs.git
>>
>> Copyright (c) 2006-2016 Red Hat, Inc. 
>>
>> GlusterFS comes with ABSOLUTELY NO WARRANTY.
>>
>> It is licensed to you under your choice of the GNU Lesser
>>
>> General Public License, version 3 or any later version (LGPLv3
>>
>> or later), or the GNU General Public License, version 2 (GPLv2),
>>
>> in all cases as published by the Free Software Foundation.
>
>
> I have stopped the glusterd daemons in all the node along with the volume and 
> geo-replication.
> Now I started the daemons, volume and geo-replication session the status 
> seems to be faulty.
> Also noted that the result of "gluster-mountbroker status" command always end 
> in python exception like this
>>
>> Traceback (most recent call last):
>>
>>   File "/usr/sbin/gluster-mountbroker", line 396, in 
>>
>> runcli()
>>
>>   File "/usr/lib/python2.7/site-packages/gluster/cliutils/cliutils.py", line 
>> 225, in runcli
>>
>> cls.run(args)
>>
>>   File "/usr/sbin/gluster-mountbroker", line 275, in run
>>
>> out = execute_in_peers("node-status")
>>
>>   File "/usr/lib/python2.7/site-packages/gluster/cliutils/cliutils.py", line 
>> 127, in execute_in_peers
>>
>> raise GlusterCmdException((rc, out, err, " ".join(cmd)))
>>
>> gluster.cliutils.cliutils.GlusterCmdException: (1, '', 'Unable to end. Error 
>> : Success\n', 'gluster system:: execute mountbroker.py node-status')
>
>
> Is it I or everyone gets an error for gluster-mountbroker command for gluster 
> version greater than 6.0?. Please help.
>
> Thank you
> Deepak
>
>
> On Thu, Jun 6, 2019 at 10:35 AM Sunny Kumar  wrote:
>>
>> Hi,
>>
>> Updated link for documentation :
>>
>> --  
>> https://docs.gluster.org/en/latest/Administrator%20Guide/Geo%20Replication/
>>
>> You can use this tool as well:
>> http://aravindavk.in/blog/gluster-georep-tools/
>>
>> -Sunny
>>
>> On Thu, Jun 6, 2019 at 10:29 AM Kotresh Hiremath Ravishankar
>>  wrote:
>> >
>> > Hi,
>> >
>> > I think the steps to setup non-root geo-rep is not followed properly. The 
>> > following entry is missing in glusterd vol file which is required.
>> >
>> > The message "E [MSGID: 106061] 
>> > [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option 
>> > mountbroker-root' missing in glusterd vol file" repeated 33 times between 
>> > [2019-06-05 08:50:46.361384] and [2019-06-05 08:52:34.019757]
>> >
>> > Could you please the steps from below?
>> >
>> > https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.4/html-single/administration_guide/index#Setting_Up_the_Environment_for_a_Secure_Geo-replication_Slave
>> >
>> > And let us know if you still face the issue.
>> >
>> >
>> >
>> >
>> > On Thu, Jun 6, 2019 at 10:24 AM deepu srinivasan  
>> > wrote:
>> >>
>> >> Hi Kotresh, Sunny
>> >> I Have mailed the logs I found in one of 

Re: [Gluster-users] Geo Replication stops replicating

[Sunny Kumar]

Hi,

Updated link for documentation :

--  https://docs.gluster.org/en/latest/Administrator%20Guide/Geo%20Replication/

You can use this tool as well:
http://aravindavk.in/blog/gluster-georep-tools/

-Sunny

On Thu, Jun 6, 2019 at 10:29 AM Kotresh Hiremath Ravishankar
 wrote:
>
> Hi,
>
> I think the steps to setup non-root geo-rep is not followed properly. The 
> following entry is missing in glusterd vol file which is required.
>
> The message "E [MSGID: 106061] [glusterd-mountbroker.c:555:glusterd_do_mount] 
> 0-management: 'option mountbroker-root' missing in glusterd vol file" 
> repeated 33 times between [2019-06-05 08:50:46.361384] and [2019-06-05 
> 08:52:34.019757]
>
> Could you please the steps from below?
>
> https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.4/html-single/administration_guide/index#Setting_Up_the_Environment_for_a_Secure_Geo-replication_Slave
>
> And let us know if you still face the issue.
>
>
>
>
> On Thu, Jun 6, 2019 at 10:24 AM deepu srinivasan  wrote:
>>
>> Hi Kotresh, Sunny
>> I Have mailed the logs I found in one of the slave machines. Is there 
>> anything to do with permission? Please help.
>>
>> On Wed, Jun 5, 2019 at 2:28 PM deepu srinivasan  wrote:
>>>
>>> Hi Kotresh, Sunny
>>> Found this log in the slave machine.

 [2019-06-05 08:49:10.632583] I [MSGID: 106488] 
 [glusterd-handler.c:1559:__glusterd_handle_cli_get_volume] 0-management: 
 Received get vol req

 The message "I [MSGID: 106488] 
 [glusterd-handler.c:1559:__glusterd_handle_cli_get_volume] 0-management: 
 Received get vol req" repeated 2 times between [2019-06-05 
 08:49:10.632583] and [2019-06-05 08:49:10.670863]

 The message "I [MSGID: 106496] 
 [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: Received 
 mount req" repeated 34 times between [2019-06-05 08:48:41.005398] and 
 [2019-06-05 08:50:37.254063]

 The message "E [MSGID: 106061] 
 [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option 
 mountbroker-root' missing in glusterd vol file" repeated 34 times between 
 [2019-06-05 08:48:41.005434] and [2019-06-05 08:50:37.254079]

 The message "W [MSGID: 106176] 
 [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: unsuccessful 
 mount request [No such file or directory]" repeated 34 times between 
 [2019-06-05 08:48:41.005444] and [2019-06-05 08:50:37.254080]

 [2019-06-05 08:50:46.361347] I [MSGID: 106496] 
 [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: Received 
 mount req

 [2019-06-05 08:50:46.361384] E [MSGID: 106061] 
 [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option 
 mountbroker-root' missing in glusterd vol file

 [2019-06-05 08:50:46.361419] W [MSGID: 106176] 
 [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: unsuccessful 
 mount request [No such file or directory]

 The message "I [MSGID: 106496] 
 [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: Received 
 mount req" repeated 33 times between [2019-06-05 08:50:46.361347] and 
 [2019-06-05 08:52:34.019741]

 The message "E [MSGID: 106061] 
 [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option 
 mountbroker-root' missing in glusterd vol file" repeated 33 times between 
 [2019-06-05 08:50:46.361384] and [2019-06-05 08:52:34.019757]

 The message "W [MSGID: 106176] 
 [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: unsuccessful 
 mount request [No such file or directory]" repeated 33 times between 
 [2019-06-05 08:50:46.361419] and [2019-06-05 08:52:34.019758]

 [2019-06-05 08:52:44.426839] I [MSGID: 106496] 
 [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: Received 
 mount req

 [2019-06-05 08:52:44.426886] E [MSGID: 106061] 
 [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option 
 mountbroker-root' missing in glusterd vol file

 [2019-06-05 08:52:44.426896] W [MSGID: 106176] 
 [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: unsuccessful 
 mount request [No such file or directory]
>>>
>>>
>>> On Wed, Jun 5, 2019 at 1:06 AM deepu srinivasan  wrote:

 Thankyou Kotresh

 On Tue, Jun 4, 2019, 11:20 PM Kotresh Hiremath Ravishankar 
  wrote:
>
> Ccing Sunny, who was investing similar issue.
>
> On Tue, Jun 4, 2019 at 5:46 PM deepu srinivasan  
> wrote:
>>
>> Have already added the path in bashrc . Still in faulty state
>>
>> On Tue, Jun 4, 2019, 5:27 PM Kotresh Hiremath Ravishankar 
>>  wrote:
>>>
>>> could you please try adding /usr/sbin to $PATH for user 'sas'? If it's 
>>> bash, add 'export PATH=/usr/sbin:$PATH' in
>>> /home/sas/.bashrc
>>>
>>> On Tue, Jun 4, 2019 at 5:24 PM deepu srinivasan  
>>> wrote:

 Hi 

Re: [Gluster-users] Geo Replication stops replicating

[Kotresh Hiremath Ravishankar]

Hi,

I think the steps to setup non-root geo-rep is not followed properly. The
following entry is missing in glusterd vol file which is required.

The message "E [MSGID: 106061]
[glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option
mountbroker-root' missing in glusterd vol file" repeated 33 times between
[2019-06-05 08:50:46.361384] and [2019-06-05 08:52:34.019757]

Could you please the steps from below?

https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.4/html-single/administration_guide/index#Setting_Up_the_Environment_for_a_Secure_Geo-replication_Slave

And let us know if you still face the issue.




On Thu, Jun 6, 2019 at 10:24 AM deepu srinivasan  wrote:

> Hi Kotresh, Sunny
> I Have mailed the logs I found in one of the slave machines. Is there
> anything to do with permission? Please help.
>
> On Wed, Jun 5, 2019 at 2:28 PM deepu srinivasan 
> wrote:
>
>> Hi Kotresh, Sunny
>> Found this log in the slave machine.
>>
>>> [2019-06-05 08:49:10.632583] I [MSGID: 106488]
>>> [glusterd-handler.c:1559:__glusterd_handle_cli_get_volume] 0-management:
>>> Received get vol req
>>>
>>> The message "I [MSGID: 106488]
>>> [glusterd-handler.c:1559:__glusterd_handle_cli_get_volume] 0-management:
>>> Received get vol req" repeated 2 times between [2019-06-05 08:49:10.632583]
>>> and [2019-06-05 08:49:10.670863]
>>>
>>> The message "I [MSGID: 106496]
>>> [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: Received
>>> mount req" repeated 34 times between [2019-06-05 08:48:41.005398] and
>>> [2019-06-05 08:50:37.254063]
>>>
>>> The message "E [MSGID: 106061]
>>> [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option
>>> mountbroker-root' missing in glusterd vol file" repeated 34 times between
>>> [2019-06-05 08:48:41.005434] and [2019-06-05 08:50:37.254079]
>>>
>>> The message "W [MSGID: 106176]
>>> [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: unsuccessful
>>> mount request [No such file or directory]" repeated 34 times between
>>> [2019-06-05 08:48:41.005444] and [2019-06-05 08:50:37.254080]
>>>
>>> [2019-06-05 08:50:46.361347] I [MSGID: 106496]
>>> [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: Received
>>> mount req
>>>
>>> [2019-06-05 08:50:46.361384] E [MSGID: 106061]
>>> [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option
>>> mountbroker-root' missing in glusterd vol file
>>>
>>> [2019-06-05 08:50:46.361419] W [MSGID: 106176]
>>> [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: unsuccessful
>>> mount request [No such file or directory]
>>>
>>> The message "I [MSGID: 106496]
>>> [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: Received
>>> mount req" repeated 33 times between [2019-06-05 08:50:46.361347] and
>>> [2019-06-05 08:52:34.019741]
>>>
>>> The message "E [MSGID: 106061]
>>> [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option
>>> mountbroker-root' missing in glusterd vol file" repeated 33 times between
>>> [2019-06-05 08:50:46.361384] and [2019-06-05 08:52:34.019757]
>>>
>>> The message "W [MSGID: 106176]
>>> [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: unsuccessful
>>> mount request [No such file or directory]" repeated 33 times between
>>> [2019-06-05 08:50:46.361419] and [2019-06-05 08:52:34.019758]
>>>
>>> [2019-06-05 08:52:44.426839] I [MSGID: 106496]
>>> [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: Received
>>> mount req
>>>
>>> [2019-06-05 08:52:44.426886] E [MSGID: 106061]
>>> [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option
>>> mountbroker-root' missing in glusterd vol file
>>>
>>> [2019-06-05 08:52:44.426896] W [MSGID: 106176]
>>> [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: unsuccessful
>>> mount request [No such file or directory]
>>>
>>
>> On Wed, Jun 5, 2019 at 1:06 AM deepu srinivasan 
>> wrote:
>>
>>> Thankyou Kotresh
>>>
>>> On Tue, Jun 4, 2019, 11:20 PM Kotresh Hiremath Ravishankar <
>>> khire...@redhat.com> wrote:
>>>
 Ccing Sunny, who was investing similar issue.

 On Tue, Jun 4, 2019 at 5:46 PM deepu srinivasan 
 wrote:

> Have already added the path in bashrc . Still in faulty state
>
> On Tue, Jun 4, 2019, 5:27 PM Kotresh Hiremath Ravishankar <
> khire...@redhat.com> wrote:
>
>> could you please try adding /usr/sbin to $PATH for user 'sas'? If
>> it's bash, add 'export PATH=/usr/sbin:$PATH' in
>> /home/sas/.bashrc
>>
>> On Tue, Jun 4, 2019 at 5:24 PM deepu srinivasan 
>> wrote:
>>
>>> Hi Kortesh
>>> Please find the logs of the above error
>>> *Master log snippet*
>>>
 [2019-06-04 11:52:09.254731] I [resource(worker
 /home/sas/gluster/data/code-misc):1379:connect_remote] SSH: 
 Initializing
 SSH connection between master and slave...
  [2019-06-04 11:52:09.308923] D [repce(worker
 /home/sas/gluster/data/code-misc):196:push] RepceClient: 

Re: [Gluster-users] Geo Replication stops replicating

[Kotresh Hiremath Ravishankar]

Ccing Sunny, who was investing similar issue.

On Tue, Jun 4, 2019 at 5:46 PM deepu srinivasan  wrote:

> Have already added the path in bashrc . Still in faulty state
>
> On Tue, Jun 4, 2019, 5:27 PM Kotresh Hiremath Ravishankar <
> khire...@redhat.com> wrote:
>
>> could you please try adding /usr/sbin to $PATH for user 'sas'? If it's
>> bash, add 'export PATH=/usr/sbin:$PATH' in
>> /home/sas/.bashrc
>>
>> On Tue, Jun 4, 2019 at 5:24 PM deepu srinivasan 
>> wrote:
>>
>>> Hi Kortesh
>>> Please find the logs of the above error
>>> *Master log snippet*
>>>
 [2019-06-04 11:52:09.254731] I [resource(worker
 /home/sas/gluster/data/code-misc):1379:connect_remote] SSH: Initializing
 SSH connection between master and slave...
  [2019-06-04 11:52:09.308923] D [repce(worker
 /home/sas/gluster/data/code-misc):196:push] RepceClient: call
 89724:139652759443264:1559649129.31 __repce_version__() ...
  [2019-06-04 11:52:09.602792] E [syncdutils(worker
 /home/sas/gluster/data/code-misc):311:log_raise_exception] :
 connection to peer is broken
  [2019-06-04 11:52:09.603312] E [syncdutils(worker
 /home/sas/gluster/data/code-misc):805:errlog] Popen: command returned error
   cmd=ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i
 /var/lib/ glusterd/geo-replication/secret.pem -p 22 -oControlMaster=auto -S
 /tmp/gsyncd-aux-ssh-4aL2tc/d893f66e0addc32f7d0080bb503f5185.sock
 sas@192.168.185.107 /usr/libexec/glusterfs/gsyncd slave code-misc sas@
   192.168.185.107::code-misc --master-node 192.168.185.106
 --master-node-id 851b64d0-d885-4ae9-9b38-ab5b15db0fec --master-brick
 /home/sas/gluster/data/code-misc --local-node 192.168.185.122 --local-node-
   id bcaa7af6-c3a1-4411-8e99-4ebecb32eb6a --slave-timeout 120
 --slave-log-level DEBUG --slave-gluster-log-level INFO
 --slave-gluster-command-dir /usr/sbin   error=1
  [2019-06-04 11:52:09.614996] I [repce(agent
 /home/sas/gluster/data/code-misc):97:service_loop] RepceServer: terminating
 on reaching EOF.
  [2019-06-04 11:52:09.615545] D [monitor(monitor):271:monitor] Monitor:
 worker(/home/sas/gluster/data/code-misc) connected
  [2019-06-04 11:52:09.616528] I [monitor(monitor):278:monitor] Monitor:
 worker died in startup phase brick=/home/sas/gluster/data/code-misc
  [2019-06-04 11:52:09.619391] I
 [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status
 Change status=Faulty

>>>
>>> *Slave log snippet*
>>>
 [2019-06-04 11:50:09.782668] E [syncdutils(slave
 192.168.185.106/home/sas/gluster/data/code-misc):809:logerr] Popen:
 /usr/sbin/gluster> 2 : failed with this errno (No such file or directory)
 [2019-06-04 11:50:11.188167] W [gsyncd(slave
 192.168.185.125/home/sas/gluster/data/code-misc):305:main] :
 Session config file not exists, using the default config
 path=/var/lib/glusterd/geo-replication/code-misc_192.168.185.107_code-misc/gsyncd.conf
 [2019-06-04 11:50:11.201070] I [resource(slave
 192.168.185.125/home/sas/gluster/data/code-misc):1098:connect]
 GLUSTER: Mounting gluster volume locally...
 [2019-06-04 11:50:11.271231] E [resource(slave
 192.168.185.125/home/sas/gluster/data/code-misc):1006:handle_mounter]
 MountbrokerMounter: glusterd answered mnt=
 [2019-06-04 11:50:11.271998] E [syncdutils(slave
 192.168.185.125/home/sas/gluster/data/code-misc):805:errlog] Popen:
 command returned error cmd=/usr/sbin/gluster --remote-host=localhost
 system:: mount sas user-map-root=sas aux-gfid-mount acl log-level=INFO
 log-file=/var/log/glusterfs/geo-replication-slaves/code-misc_192.168.185.107_code-misc/mnt-192.168.185.125-home-sas-gluster-data-code-misc.log
 volfile-server=localhost volfile-id=code-misc client-pid=-1 error=1
 [2019-06-04 11:50:11.272113] E [syncdutils(slave
 192.168.185.125/home/sas/gluster/data/code-misc):809:logerr] Popen:
 /usr/sbin/gluster> 2 : failed with this errno (No such file or directory)
>>>
>>>
>>> On Tue, Jun 4, 2019 at 5:10 PM deepu srinivasan 
>>> wrote:
>>>
 Hi
 As discussed I have upgraded gluster from 4.1 to 6.2 version. But the
 Geo replication failed to start.
 Stays in faulty state

 On Fri, May 31, 2019, 5:32 PM deepu srinivasan 
 wrote:

> Checked the data. It remains in 2708. No progress.
>
> On Fri, May 31, 2019 at 4:36 PM Kotresh Hiremath Ravishankar <
> khire...@redhat.com> wrote:
>
>> That means it could be working and the defunct process might be some
>> old zombie one. Could you check, that data progress ?
>>
>> On Fri, May 31, 2019 at 4:29 PM deepu srinivasan 
>> wrote:
>>
>>> Hi
>>> When i change the rsync option the rsync process doesnt seem to
>>> start . Only a defunt process is listed in ps aux. Only when i set rsync
>>> option to " " and restart all the process the rsync process is listed 
>>> 

Re: [Gluster-users] Geo Replication stops replicating

[Kotresh Hiremath Ravishankar]

could you please try adding /usr/sbin to $PATH for user 'sas'? If it's
bash, add 'export PATH=/usr/sbin:$PATH' in
/home/sas/.bashrc

On Tue, Jun 4, 2019 at 5:24 PM deepu srinivasan  wrote:

> Hi Kortesh
> Please find the logs of the above error
> *Master log snippet*
>
>> [2019-06-04 11:52:09.254731] I [resource(worker
>> /home/sas/gluster/data/code-misc):1379:connect_remote] SSH: Initializing
>> SSH connection between master and slave...
>>  [2019-06-04 11:52:09.308923] D [repce(worker
>> /home/sas/gluster/data/code-misc):196:push] RepceClient: call
>> 89724:139652759443264:1559649129.31 __repce_version__() ...
>>  [2019-06-04 11:52:09.602792] E [syncdutils(worker
>> /home/sas/gluster/data/code-misc):311:log_raise_exception] :
>> connection to peer is broken
>>  [2019-06-04 11:52:09.603312] E [syncdutils(worker
>> /home/sas/gluster/data/code-misc):805:errlog] Popen: command returned error
>>   cmd=ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i
>> /var/lib/ glusterd/geo-replication/secret.pem -p 22 -oControlMaster=auto -S
>> /tmp/gsyncd-aux-ssh-4aL2tc/d893f66e0addc32f7d0080bb503f5185.sock
>> sas@192.168.185.107 /usr/libexec/glusterfs/gsyncd slave code-misc sas@
>> 192.168.185.107::code-misc --master-node 192.168.185.106
>> --master-node-id 851b64d0-d885-4ae9-9b38-ab5b15db0fec --master-brick
>> /home/sas/gluster/data/code-misc --local-node 192.168.185.122 --local-node-
>>   id bcaa7af6-c3a1-4411-8e99-4ebecb32eb6a --slave-timeout 120
>> --slave-log-level DEBUG --slave-gluster-log-level INFO
>> --slave-gluster-command-dir /usr/sbin   error=1
>>  [2019-06-04 11:52:09.614996] I [repce(agent
>> /home/sas/gluster/data/code-misc):97:service_loop] RepceServer: terminating
>> on reaching EOF.
>>  [2019-06-04 11:52:09.615545] D [monitor(monitor):271:monitor] Monitor:
>> worker(/home/sas/gluster/data/code-misc) connected
>>  [2019-06-04 11:52:09.616528] I [monitor(monitor):278:monitor] Monitor:
>> worker died in startup phase brick=/home/sas/gluster/data/code-misc
>>  [2019-06-04 11:52:09.619391] I
>> [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status
>> Change status=Faulty
>>
>
> *Slave log snippet*
>
>> [2019-06-04 11:50:09.782668] E [syncdutils(slave
>> 192.168.185.106/home/sas/gluster/data/code-misc):809:logerr] Popen:
>> /usr/sbin/gluster> 2 : failed with this errno (No such file or directory)
>> [2019-06-04 11:50:11.188167] W [gsyncd(slave
>> 192.168.185.125/home/sas/gluster/data/code-misc):305:main] :
>> Session config file not exists, using the default config
>> path=/var/lib/glusterd/geo-replication/code-misc_192.168.185.107_code-misc/gsyncd.conf
>> [2019-06-04 11:50:11.201070] I [resource(slave
>> 192.168.185.125/home/sas/gluster/data/code-misc):1098:connect] GLUSTER:
>> Mounting gluster volume locally...
>> [2019-06-04 11:50:11.271231] E [resource(slave
>> 192.168.185.125/home/sas/gluster/data/code-misc):1006:handle_mounter]
>> MountbrokerMounter: glusterd answered mnt=
>> [2019-06-04 11:50:11.271998] E [syncdutils(slave
>> 192.168.185.125/home/sas/gluster/data/code-misc):805:errlog] Popen:
>> command returned error cmd=/usr/sbin/gluster --remote-host=localhost
>> system:: mount sas user-map-root=sas aux-gfid-mount acl log-level=INFO
>> log-file=/var/log/glusterfs/geo-replication-slaves/code-misc_192.168.185.107_code-misc/mnt-192.168.185.125-home-sas-gluster-data-code-misc.log
>> volfile-server=localhost volfile-id=code-misc client-pid=-1 error=1
>> [2019-06-04 11:50:11.272113] E [syncdutils(slave
>> 192.168.185.125/home/sas/gluster/data/code-misc):809:logerr] Popen:
>> /usr/sbin/gluster> 2 : failed with this errno (No such file or directory)
>
>
> On Tue, Jun 4, 2019 at 5:10 PM deepu srinivasan 
> wrote:
>
>> Hi
>> As discussed I have upgraded gluster from 4.1 to 6.2 version. But the Geo
>> replication failed to start.
>> Stays in faulty state
>>
>> On Fri, May 31, 2019, 5:32 PM deepu srinivasan 
>> wrote:
>>
>>> Checked the data. It remains in 2708. No progress.
>>>
>>> On Fri, May 31, 2019 at 4:36 PM Kotresh Hiremath Ravishankar <
>>> khire...@redhat.com> wrote:
>>>
 That means it could be working and the defunct process might be some
 old zombie one. Could you check, that data progress ?

 On Fri, May 31, 2019 at 4:29 PM deepu srinivasan 
 wrote:

> Hi
> When i change the rsync option the rsync process doesnt seem to start
> . Only a defunt process is listed in ps aux. Only when i set rsync option
> to " " and restart all the process the rsync process is listed in ps aux.
>
>
> On Fri, May 31, 2019 at 4:23 PM Kotresh Hiremath Ravishankar <
> khire...@redhat.com> wrote:
>
>> Yes, rsync config option should have fixed this issue.
>>
>> Could you share the output of the following?
>>
>> 1. gluster volume geo-replication  ::
>> config rsync-options
>> 2. ps -ef | grep rsync
>>
>> On Fri, May 31, 2019 at 4:11 PM deepu srinivasan 
>> wrote:
>>
>>> Done.
>>> We 

Re: [Gluster-users] Geo Replication stops replicating

[Kotresh Hiremath Ravishankar]

That means it could be working and the defunct process might be some old
zombie one. Could you check, that data progress ?

On Fri, May 31, 2019 at 4:29 PM deepu srinivasan  wrote:

> Hi
> When i change the rsync option the rsync process doesnt seem to start .
> Only a defunt process is listed in ps aux. Only when i set rsync option to
> " " and restart all the process the rsync process is listed in ps aux.
>
>
> On Fri, May 31, 2019 at 4:23 PM Kotresh Hiremath Ravishankar <
> khire...@redhat.com> wrote:
>
>> Yes, rsync config option should have fixed this issue.
>>
>> Could you share the output of the following?
>>
>> 1. gluster volume geo-replication  ::
>> config rsync-options
>> 2. ps -ef | grep rsync
>>
>> On Fri, May 31, 2019 at 4:11 PM deepu srinivasan 
>> wrote:
>>
>>> Done.
>>> We got the following result .
>>>
 1559298781.338234 write(2, "rsync: link_stat
 \"/tmp/gsyncd-aux-mount-EEJ_sY/.gfid/3fa6aed8-802e-4efe-9903-8bc171176d88\"
 failed: No such file or directory (2)", 128
>>>
>>> seems like a file is missing ?
>>>
>>> On Fri, May 31, 2019 at 3:25 PM Kotresh Hiremath Ravishankar <
>>> khire...@redhat.com> wrote:
>>>
 Hi,

 Could you take the strace with with more string size? The argument
 strings are truncated.

 strace -s 500 -ttt -T -p 

 On Fri, May 31, 2019 at 3:17 PM deepu srinivasan 
 wrote:

> Hi Kotresh
> The above-mentioned work around did not work properly.
>
> On Fri, May 31, 2019 at 3:16 PM deepu srinivasan 
> wrote:
>
>> Hi Kotresh
>> We have tried the above-mentioned rsync option and we are planning to
>> have the version upgrade to 6.0.
>>
>> On Fri, May 31, 2019 at 11:04 AM Kotresh Hiremath Ravishankar <
>> khire...@redhat.com> wrote:
>>
>>> Hi,
>>>
>>> This looks like the hang because stderr buffer filled up with errors
>>> messages and no one reading it.
>>> I think this issue is fixed in latest releases. As a workaround, you
>>> can do following and check if it works.
>>>
>>> Prerequisite:
>>>  rsync version should be > 3.1.0
>>>
>>> Workaround:
>>> gluster volume geo-replication  ::
>>> config rsync-options "--ignore-missing-args"
>>>
>>> Thanks,
>>> Kotresh HR
>>>
>>>
>>>
>>>
>>> On Thu, May 30, 2019 at 5:39 PM deepu srinivasan 
>>> wrote:
>>>
 Hi
 We were evaluating Gluster geo Replication between two DCs one is
 in US west and one is in US east. We took multiple trials for different
 file size.
 The Geo Replication tends to stop replicating but while checking
 the status it appears to be in Active state. But the slave volume did 
 not
 increase in size.
 So we have restarted the geo-replication session and checked the
 status. The status was in an active state and it was in History Crawl 
 for a
 long time. We have enabled the DEBUG mode in logging and checked for 
 any
 error.
 There was around 2000 file appeared for syncing candidate. The
 Rsync process starts but the rsync did not happen in the slave volume.
 Every time the rsync process appears in the "ps auxxx" list but the
 replication did not happen in the slave end. What would be the cause of
 this problem? Is there anyway to debug it?

 We have also checked the strace of the rync program.
 it displays something like this

 "write(2, "rsync: link_stat \"/tmp/gsyncd-au"..., 128"


 We are using the below specs

 Gluster version - 4.1.7
 Sync mode - rsync
 Volume - 1x3 in each end (master and slave)
 Intranet Bandwidth - 10 Gig

>>>
>>>
>>> --
>>> Thanks and Regards,
>>> Kotresh H R
>>>
>>

 --
 Thanks and Regards,
 Kotresh H R

>>>
>>
>> --
>> Thanks and Regards,
>> Kotresh H R
>>
>

-- 
Thanks and Regards,
Kotresh H R
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Geo Replication stops replicating

[Kotresh Hiremath Ravishankar]

Yes, rsync config option should have fixed this issue.

Could you share the output of the following?

1. gluster volume geo-replication  ::
config rsync-options
2. ps -ef | grep rsync

On Fri, May 31, 2019 at 4:11 PM deepu srinivasan  wrote:

> Done.
> We got the following result .
>
>> 1559298781.338234 write(2, "rsync: link_stat
>> \"/tmp/gsyncd-aux-mount-EEJ_sY/.gfid/3fa6aed8-802e-4efe-9903-8bc171176d88\"
>> failed: No such file or directory (2)", 128
>
> seems like a file is missing ?
>
> On Fri, May 31, 2019 at 3:25 PM Kotresh Hiremath Ravishankar <
> khire...@redhat.com> wrote:
>
>> Hi,
>>
>> Could you take the strace with with more string size? The argument
>> strings are truncated.
>>
>> strace -s 500 -ttt -T -p 
>>
>> On Fri, May 31, 2019 at 3:17 PM deepu srinivasan 
>> wrote:
>>
>>> Hi Kotresh
>>> The above-mentioned work around did not work properly.
>>>
>>> On Fri, May 31, 2019 at 3:16 PM deepu srinivasan 
>>> wrote:
>>>
 Hi Kotresh
 We have tried the above-mentioned rsync option and we are planning to
 have the version upgrade to 6.0.

 On Fri, May 31, 2019 at 11:04 AM Kotresh Hiremath Ravishankar <
 khire...@redhat.com> wrote:

> Hi,
>
> This looks like the hang because stderr buffer filled up with errors
> messages and no one reading it.
> I think this issue is fixed in latest releases. As a workaround, you
> can do following and check if it works.
>
> Prerequisite:
>  rsync version should be > 3.1.0
>
> Workaround:
> gluster volume geo-replication  ::
> config rsync-options "--ignore-missing-args"
>
> Thanks,
> Kotresh HR
>
>
>
>
> On Thu, May 30, 2019 at 5:39 PM deepu srinivasan 
> wrote:
>
>> Hi
>> We were evaluating Gluster geo Replication between two DCs one is in
>> US west and one is in US east. We took multiple trials for different file
>> size.
>> The Geo Replication tends to stop replicating but while checking the
>> status it appears to be in Active state. But the slave volume did not
>> increase in size.
>> So we have restarted the geo-replication session and checked the
>> status. The status was in an active state and it was in History Crawl 
>> for a
>> long time. We have enabled the DEBUG mode in logging and checked for any
>> error.
>> There was around 2000 file appeared for syncing candidate. The Rsync
>> process starts but the rsync did not happen in the slave volume. Every 
>> time
>> the rsync process appears in the "ps auxxx" list but the replication did
>> not happen in the slave end. What would be the cause of this problem? Is
>> there anyway to debug it?
>>
>> We have also checked the strace of the rync program.
>> it displays something like this
>>
>> "write(2, "rsync: link_stat \"/tmp/gsyncd-au"..., 128"
>>
>>
>> We are using the below specs
>>
>> Gluster version - 4.1.7
>> Sync mode - rsync
>> Volume - 1x3 in each end (master and slave)
>> Intranet Bandwidth - 10 Gig
>>
>
>
> --
> Thanks and Regards,
> Kotresh H R
>

>>
>> --
>> Thanks and Regards,
>> Kotresh H R
>>
>

-- 
Thanks and Regards,
Kotresh H R
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Geo Replication stops replicating

[Kotresh Hiremath Ravishankar]

Hi,

Could you take the strace with with more string size? The argument strings
are truncated.

strace -s 500 -ttt -T -p 

On Fri, May 31, 2019 at 3:17 PM deepu srinivasan  wrote:

> Hi Kotresh
> The above-mentioned work around did not work properly.
>
> On Fri, May 31, 2019 at 3:16 PM deepu srinivasan 
> wrote:
>
>> Hi Kotresh
>> We have tried the above-mentioned rsync option and we are planning to
>> have the version upgrade to 6.0.
>>
>> On Fri, May 31, 2019 at 11:04 AM Kotresh Hiremath Ravishankar <
>> khire...@redhat.com> wrote:
>>
>>> Hi,
>>>
>>> This looks like the hang because stderr buffer filled up with errors
>>> messages and no one reading it.
>>> I think this issue is fixed in latest releases. As a workaround, you can
>>> do following and check if it works.
>>>
>>> Prerequisite:
>>>  rsync version should be > 3.1.0
>>>
>>> Workaround:
>>> gluster volume geo-replication  ::
>>> config rsync-options "--ignore-missing-args"
>>>
>>> Thanks,
>>> Kotresh HR
>>>
>>>
>>>
>>>
>>> On Thu, May 30, 2019 at 5:39 PM deepu srinivasan 
>>> wrote:
>>>
 Hi
 We were evaluating Gluster geo Replication between two DCs one is in US
 west and one is in US east. We took multiple trials for different file
 size.
 The Geo Replication tends to stop replicating but while checking the
 status it appears to be in Active state. But the slave volume did not
 increase in size.
 So we have restarted the geo-replication session and checked the
 status. The status was in an active state and it was in History Crawl for a
 long time. We have enabled the DEBUG mode in logging and checked for any
 error.
 There was around 2000 file appeared for syncing candidate. The Rsync
 process starts but the rsync did not happen in the slave volume. Every time
 the rsync process appears in the "ps auxxx" list but the replication did
 not happen in the slave end. What would be the cause of this problem? Is
 there anyway to debug it?

 We have also checked the strace of the rync program.
 it displays something like this

 "write(2, "rsync: link_stat \"/tmp/gsyncd-au"..., 128"


 We are using the below specs

 Gluster version - 4.1.7
 Sync mode - rsync
 Volume - 1x3 in each end (master and slave)
 Intranet Bandwidth - 10 Gig

>>>
>>>
>>> --
>>> Thanks and Regards,
>>> Kotresh H R
>>>
>>

-- 
Thanks and Regards,
Kotresh H R
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Geo Replication stops replicating

[Kotresh Hiremath Ravishankar]

Hi,

This looks like the hang because stderr buffer filled up with errors
messages and no one reading it.
I think this issue is fixed in latest releases. As a workaround, you can do
following and check if it works.

Prerequisite:
 rsync version should be > 3.1.0

Workaround:
gluster volume geo-replication  :: config
rsync-options "--ignore-missing-args"

Thanks,
Kotresh HR




On Thu, May 30, 2019 at 5:39 PM deepu srinivasan  wrote:

> Hi
> We were evaluating Gluster geo Replication between two DCs one is in US
> west and one is in US east. We took multiple trials for different file
> size.
> The Geo Replication tends to stop replicating but while checking the
> status it appears to be in Active state. But the slave volume did not
> increase in size.
> So we have restarted the geo-replication session and checked the status.
> The status was in an active state and it was in History Crawl for a long
> time. We have enabled the DEBUG mode in logging and checked for any error.
> There was around 2000 file appeared for syncing candidate. The Rsync
> process starts but the rsync did not happen in the slave volume. Every time
> the rsync process appears in the "ps auxxx" list but the replication did
> not happen in the slave end. What would be the cause of this problem? Is
> there anyway to debug it?
>
> We have also checked the strace of the rync program.
> it displays something like this
>
> "write(2, "rsync: link_stat \"/tmp/gsyncd-au"..., 128"
>
>
> We are using the below specs
>
> Gluster version - 4.1.7
> Sync mode - rsync
> Volume - 1x3 in each end (master and slave)
> Intranet Bandwidth - 10 Gig
>


-- 
Thanks and Regards,
Kotresh H R
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users