As requested you will find below the last few 100 lines of the geo-rep log file
from the master node.
FILE:
/var/log/glusterfs/geo-replication/cloud-pro/ssh%3A%2F%2Froot%40192.168.20.107%3Agluster%3A%2F%2F127.0.0.1%3Acloud-pro-geo.log
[2016-09-13 19:39:48.686060] I [syncdutils(/data/cloud-pro/brick):220:finalize]
: exiting.
[2016-09-13 19:39:48.688112] I [repce(agent):92:service_loop] RepceServer:
terminating on reaching EOF.
[2016-09-13 19:39:48.688461] I [syncdutils(agent):220:finalize] : exiting.
[2016-09-13 19:39:49.518510] I [monitor(monitor):343:monitor] Monitor:
worker(/data/cloud-pro/brick) died in startup phase
[2016-09-13 19:39:59.675405] I [monitor(monitor):266:monitor] Monitor:
[2016-09-13 19:39:59.675740] I [monitor(monitor):267:monitor] Monitor: starting
gsyncd worker
[2016-09-13 19:39:59.768181] I [gsyncd(/data/cloud-pro/brick):710:main_i]
: syncing: gluster://localhost:cloud-pro ->
ssh://r...@gfs1geo.domain.tld:gluster://localhost:cloud-pro-geo
[2016-09-13 19:39:59.768640] I [changelogagent(agent):73:__init__]
ChangelogAgent: Agent listining...
[2016-09-13 19:40:02.554076] I
[master(/data/cloud-pro/brick):83:gmaster_builder] : setting up xsync
change detection mode
[2016-09-13 19:40:02.554500] I [master(/data/cloud-pro/brick):367:__init__]
_GMaster: using 'rsync' as the sync engine
[2016-09-13 19:40:02.555332] I
[master(/data/cloud-pro/brick):83:gmaster_builder] : setting up changelog
change detection mode
[2016-09-13 19:40:02.555600] I [master(/data/cloud-pro/brick):367:__init__]
_GMaster: using 'rsync' as the sync engine
[2016-09-13 19:40:02.556711] I
[master(/data/cloud-pro/brick):83:gmaster_builder] : setting up
changeloghistory change detection mode
[2016-09-13 19:40:02.556983] I [master(/data/cloud-pro/brick):367:__init__]
_GMaster: using 'rsync' as the sync engine
[2016-09-13 19:40:04.692655] I [master(/data/cloud-pro/brick):1249:register]
_GMaster: xsync temp directory:
/var/lib/misc/glusterfsd/cloud-pro/ssh%3A%2F%2Froot%40192.168.20.107%3Agluster%3A%2F%2F127.0.0.1%3Acloud-pro-geo/6b844f56a11ecd24d5e36242f045e58c/xsync
[2016-09-13 19:40:04.692945] I
[resource(/data/cloud-pro/brick):1491:service_loop] GLUSTER: Register time:
1473795604
[2016-09-13 19:40:04.724827] I [master(/data/cloud-pro/brick):510:crawlwrap]
_GMaster: primary master with volume id d99af2fa-439b-4a21-bf3a-38f3849f87ec ...
[2016-09-13 19:40:04.734505] I [master(/data/cloud-pro/brick):519:crawlwrap]
_GMaster: crawl interval: 1 seconds
[2016-09-13 19:40:04.754487] I [master(/data/cloud-pro/brick):1163:crawl]
_GMaster: starting history crawl... turns: 1, stime: (1473688054, 0), etime:
1473795604
[2016-09-13 19:40:05.757395] I [master(/data/cloud-pro/brick):1192:crawl]
_GMaster: slave's time: (1473688054, 0)
[2016-09-13 19:40:05.832189] E [repce(/data/cloud-pro/brick):207:__call__]
RepceClient: call 28167:140561674450688:1473795605.78 (entry_ops) failed on
peer with OSError
[2016-09-13 19:40:05.832458] E
[syncdutils(/data/cloud-pro/brick):276:log_raise_exception] : FAIL:
Traceback (most recent call last):
File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/gsyncd.py", line
201, in main
main_i()
File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/gsyncd.py", line
720, in main_i
local.service_loop(*[r for r in [remote] if r])
File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/resource.py", line
1497, in service_loop
g3.crawlwrap(oneshot=True)
File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/master.py", line
571, in crawlwrap
self.crawl()
File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/master.py", line
1201, in crawl
self.changelogs_batch_process(changes)
File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/master.py", line
1107, in changelogs_batch_process
self.process(batch)
File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/master.py", line
992, in process
self.process_change(change, done, retry)
File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/master.py", line
934, in process_change
failures = self.slave.server.entry_ops(entries)
File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/repce.py", line
226, in __call__
return self.ins(self.meth, *a)
File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/repce.py", line
208, in __call__
raise res
OSError: [Errno 39] Directory not empty
[2016-09-13 19:40:05.834218] I [syncdutils(/data/cloud-pro/brick):220:finalize]
: exiting.
[2016-09-13 19:40:05.836263] I [repce(agent):92:service_loop] RepceServer:
terminating on reaching EOF.
[2016-09-13 19:40:05.836611] I [syncdutils(agent):220:finalize] : exiting.
[2016-09-13 19:40:06.558499] I [monitor(monitor):343:monitor] Monitor:
worker(/data/cloud-pro/brick) died in startup phase
[2016-09-13 19:40:16.715217] I [monitor(monitor):266:monitor] Monitor:
[2016-09-13 19:40:16.71551