[ceph-users] Re: Recovering from a Failed Disk (replication 1)

[vladimir franciz blando]

The OSD even when it's down, I can still access it's contents, looks like I
need to check out ceph-objectstore-tool.

# idweight  type name   up/down reweight
-1  98.44   root default
-2  32.82   host ceph-node-1
0   3.64osd.0   up  1
1   3.64osd.1   up  1
2   3.64osd.2   up  1
3   3.64osd.3   up  1
4   3.64osd.4   up  1
5   3.64osd.5   up  1
6   3.7 osd.6   up  1
7   3.64osd.7   up  1
8   3.64osd.8   up  1
-3  32.74   host ceph-node-2
10  3.64osd.10  down1
11  3.64osd.11  up  1
12  3.64osd.12  up  1
13  3.64osd.13  up  1
15  3.64osd.15  up  1
16  3.64osd.16  up  1
17  3.64osd.17  up  1
27  3.63osd.27  up  1
14  3.63osd.14  DNE
-4  32.88   host ceph-node-3
18  3.7 osd.18  up  1
19  3.7 osd.19  up  1
20  3.64osd.20  up  1
21  3.64osd.21  up  1
22  3.64osd.22  up  1
23  3.64osd.23  up  1
24  3.64osd.24  up  1
25  3.64osd.25  up  1
26  3.64osd.26  up  1

[root@ceph-node-2 ceph-10]# pwd
/var/lib/ceph/osd/ceph-10
[root@ceph-node-2 ceph-10]#
[root@ceph-node-2 ceph-10]# ll
total 52
-rw-r--r--.  1 root root  499 Aug 12  2014 activate.monmap
-rw-r--r--.  1 root root3 Aug 12  2014 active
-rw-r--r--.  1 root root   37 Aug 12  2014 ceph_fsid
drwxr-xr-x. 91 root root 8192 Oct 19 11:56 current
-rw-r--r--.  1 root root   37 Aug 12  2014 fsid
lrwxrwxrwx.  1 root root9 Feb 26  2017 journal -> /dev/sdj2
-rw---.  1 root root   57 Aug 12  2014 keyring
-rw-r--r--.  1 root root   21 Aug 12  2014 magic
-rw-r--r--.  1 root root6 Aug 12  2014 ready
-rw-r--r--.  1 root root4 Aug 12  2014 store_version
-rw-r--r--.  1 root root   42 Aug 12  2014 superblock
-rw-r--r--.  1 root root0 Oct 17 10:23 sysvinit
-rw-r--r--.  1 root root3 Aug 12  2014 whoami
[root@ceph-node-2 ceph-10]




- Vlad

ᐧ

On Thu, Oct 17, 2019 at 11:56 AM Ashley Merrick 
wrote:

> I think your better off doing the DD method, you can export and import a
> PG at a time (ceph-objectstore-tool)
>
> But if the disk is failing a DD is probably your best method.
>
>
>
>  On Thu, 17 Oct 2019 11:44:20 +0800 *vladimir franciz blando
> >* wrote 
>
> Sorry for not being clear, when I say healthy disk, I mean those are
> already an OSD, so I need to transfer the data from the failed OSD to the
> other OSDs that are healthy.
>
> - Vlad
>
> ᐧ
>
> On Thu, Oct 17, 2019 at 11:31 AM Konstantin Shalygin 
> wrote:
>
>
> On 10/17/19 10:29 AM, vladimir franciz blando wrote:
> > I have a not ideal setup on one of my cluster,  3 ceph  nodes but
> > using replication 1 on all pools (don't ask me why replication 1, it's
> > a long story).
> >
> > So it has come to this situation that a disk keeps on crashing,
> > possible a hardware failure and I need to recover from that.
> >
> > What's my best option for me to recover the data from the failed disk
> > and transfer it to the other healthy disks?
> >
> > This cluster is using Firefly
>
>
> `dd if=/dev/old_drive of=/dev/new_drive` I guess.
>
>
>
> k
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] rgw multisite failover

[Frank R]

I am looking to change an RGW multisite deployment so that the secondary
will become master. This is meant to be a permanent change.

Per:
https://docs.ceph.com/docs/mimic/radosgw/multisite/

I need to:

1. Stop RGW daemons on the current master end.

On a secondary RGW node:
2. radosgw-admin zone modify --rgw-zone={zone-name} --master --default
3. radosgw-admin period update --commit
4. systemctl restart ceph-radosgw@rgw.`hostname -s`

Since I want the former master to be secondary permanently do I need to do
anything after restarting the RGW daemons on the old master end?
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] ceph balancer do not start

[Jan Peters]

Hello,

i use ceph 12.2.12 and would like to activate the ceph balancer.

unfortunately no redistribution of the PGs is started:

ceph balancer status
{
"active": true,
"plans": [],
"mode": "crush-compat"
}

ceph balancer eval
current cluster score 0.023776 (lower is better)


ceph config-key dump
{
"initial_mon_keyring": "AQBLchlbABAA+5CuVU+8MB69xfc3xAXkjQ==",
"mgr/balancer/active": "1",
"mgr/balancer/max_misplaced:": "0.01",
"mgr/balancer/mode": "crush-compat"
}


What am I not doing correctly?

best regards
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: CephFS and 32-bit Inode Numbers

[Darrell Enns]

Does your 32-bit application actually use the inode numbers? Or is it just 
trying to read other metadata (such as filenames in a directory, file sizes, 
etc)? If it's the latter, you could use LD_PRELOAD to wrap the calls and return 
fake/mangled inode numbers (since the application doesn't care about them 
anyway).

See here for more details: https://www.tcm.phy.cam.ac.uk/sw/inodes64.html

-Original Message-
From: Yan, Zheng  
Sent: Wednesday, October 16, 2019 8:07 PM
To: Dan van der Ster 
Cc: ceph-users 
Subject: [ceph-users] Re: CephFS and 32-bit Inode Numbers

On Mon, Oct 14, 2019 at 7:19 PM Dan van der Ster  wrote:
>
> Hi all,
>
> One of our users has some 32-bit commercial software that they want to 
> use with CephFS, but it's not working because our inode numbers are 
> too large. E.g. his application gets a "file too big" error trying to 
> stat inode 0x40008445FB3.
>
> I'm aware that CephFS is offsets the inode numbers by (mds_rank + 1) * 
> 2^40; in the case above the file is managed by mds.3.
>
> Did anyone see this same issue and find a workaround? (I read that 
> GlusterFS has an enable-in32 client option -- does CephFS have 
> something like that planned?)
>

ceph-fuse has client_use_faked_inos option. When it is enabled, ceph-fuse maps 
64bits inode numbers to 32bits. It works as long as client has less than 2^32 
inodes cached. So far there is no kernel client counterpart.

Regards
Yan, Zheng

> Thanks!
>
> Dan
> ___
> ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an 
> email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to 
ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] OSD node suddenly slow to responding cmd

[Amudhan P]

Hi,

I am using Ceph nautilus cluster and i found one of my OSD node running 3
OSD's service suddenly went down and it was very slow typing command. I
killed ceph-osd process and system become normal and started all OSD
service.

After that it becomes normal I figure out that due low memory the system
was slow but there is no OOM in any of the logs.

I can see heartbeat_check and suicide time out in below msg in OSD logs,
but why did this happened? what could be the issue?

*osd.1*

2019-10-16 20:49:38.433 7f246f302700  0 bluestore(/var/lib/ceph/osd/ceph-1)
log_latency_fn slow operation observed for _txc_committed_kv, latency =
6.20965s, txc = 0x55
a2541f1500
2019-10-16 20:50:05.233 7f247bf7d700  1 heartbeat_map is_healthy
'OSD::osd_op_tp thread 0x7f245e4e6700' had timed out after 15
2019-10-16 20:50:06.721 7f247af7b700  1 heartbeat_map is_healthy
'OSD::osd_op_tp thread 0x7f245e4e6700' had timed out after 15
2019-10-16 20:53:57.881 7f2475d15700 -1 osd.1 2035 heartbeat_check: no
reply from 192.168.100.113:6805 osd.2 since back 2019-10-16 20:49:44.711088
front 2019-10-16 20:4
9:47.684146 (oldest deadline 2019-10-16 20:50:07.937387)
2019-10-16 20:55:26.997 7f2475d15700 -1 osd.1 2035 heartbeat_check: no
reply from 192.168.100.111:6821 osd.3 since back 2019-10-16 20:49:44.711031
front 2019-10-16 20:4
9:44.711495 (oldest deadline 2019-10-16 20:50:07.937387)
2019-10-16 20:56:25.513 7f2475d15700 -1 osd.1 2035 heartbeat_check: no
reply from 192.168.100.112:6805 osd.4 since back 2019-10-16 20:49:44.484031
front 2019-10-16 20:4
9:44.711337 (oldest deadline 2019-10-16 20:50:03.162338)
2019-10-16 20:57:52.557 7f2475d15700 -1 osd.1 2035 heartbeat_check: no
reply from 192.168.100.113:6813 osd.5 since back 2019-10-16 20:49:44.484064
front 2019-10-16 20:4
9:44.711504 (oldest deadline 2019-10-16 20:50:03.162338)
2019-10-16 20:59:22.837 7f2475d15700 -1 osd.1 2035 heartbeat_check: no
reply from 192.168.100.111:6805 osd.6 since back 2019-10-16 20:49:44.711483
front 2019-10-16 20:4
9:44.483337 (oldest deadline 2019-10-16 20:50:03.162338)
2019-10-16 21:01:18.761 7f2475d15700 -1 osd.1 2035 heartbeat_check: no
reply from 192.168.100.112:6821 osd.7 since back 2019-10-16 20:49:44.711275
front 2019-10-16 20:4
9:44.711352 (oldest deadline 2019-10-16 20:50:07.937387)
2019-10-16 21:02:34.021 7f2475d15700 -1 osd.1 2035 heartbeat_check: no
reply from 192.168.100.113:6821 osd.8 since back 2019-10-16 20:49:44.711517
front 2019-10-16 20:4
9:44.711470 (oldest deadline 2019-10-16 20:50:07.937387)
2019-10-16 21:04:26.361 7f247b77c700  1 heartbeat_map is_healthy
'OSD::osd_op_tp thread 0x7f245d4e4700' had timed out after 60
2019-10-16 21:04:46.225 7f247b77c700  1 heartbeat_map is_healthy
'OSD::osd_op_tp thread 0x7f245e4e6700' had timed out after 15
2019-10-16 21:05:01.453 7f247b77c700  1 heartbeat_map is_healthy
'OSD::osd_op_tp thread 0x7f245e4e6700' had suicide timed out after 150
2019-10-16 21:34:06.365 7f0a6105cf80  0 set uid:gid to 64045:64045
(ceph:ceph)
2019-10-16 21:34:06.365 7f0a6105cf80  0 ceph version 14.2.4
(75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable), process
ceph-osd, pid 203416
2019-10-16 21:34:06.365 7f0a6105cf80  0 pidfile_write: ignore empty
--pid-file


*osd.4*
2019-10-16 20:53:57.881 7f7ef6010700 -1 osd.4 2035 heartbeat_check: no
reply from 192.168.100.112:6813 osd.1 since back 2019-10-16 20:49:41.268550
front 2019-10-16 20:49:41.268546 (oldest deadline 2019-10-16
20:50:07.683860)
2019-10-16 20:55:26.997 7f7ef6010700 -1 osd.4 2035 heartbeat_check: no
reply from 192.168.100.113:6805 osd.2 since back 2019-10-16 20:49:44.484184
front 2019-10-16 20:49:44.483961 (oldest deadline 2019-10-16
20:50:07.683860)
2019-10-16 20:56:25.513 7f7ef6010700 -1 osd.4 2035 heartbeat_check: no
reply from 192.168.100.111:6821 osd.3 since back 2019-10-16 20:49:41.192151
front 2019-10-16 20:49:41.191856 (oldest deadline 2019-10-16
20:50:07.683860)
2019-10-16 20:57:52.557 7f7ef6010700 -1 osd.4 2035 heartbeat_check: no
reply from 192.168.100.113:6813 osd.5 since back 2019-10-16 20:49:44.484278
front 2019-10-16 20:49:47.920080 (oldest deadline 2019-10-16
20:50:07.683860)
2019-10-16 20:59:22.837 7f7ef6010700 -1 osd.4 2035 heartbeat_check: no
reply from 192.168.100.111:6805 osd.6 since back 2019-10-16 20:49:41.268846
front 2019-10-16 20:49:41.192139 (oldest deadline 2019-10-16
20:50:07.683860)
2019-10-16 21:01:16.561 7f7ef6010700 -1 osd.4 2035 heartbeat_check: no
reply from 192.168.100.112:6821 osd.7 since back 2019-10-16 20:49:41.380152
front 2019-10-16 20:49:41.380178 (oldest deadline 2019-10-16
20:50:07.683860)
2019-10-16 21:02:15.621 7f7ef6010700 -1 osd.4 2035 heartbeat_check: no
reply from 192.168.100.113:6821 osd.8 since back 2019-10-16 20:49:43.862767
front 2019-10-16 20:49:44.484153 (oldest deadline 2019-10-16
20:50:07.683860)
2019-10-16 21:04:14.997 7f7efba77700  1 heartbeat_map is_healthy
'OSD::osd_op_tp thread 0x7f7edd7df700' had timed out after 60
2019-10-16 21:04:31.689 7f7efba77700  1 heartbeat_map is_healthy
'OSD::osd_op_tp thr

[ceph-users] mds log showing msg with HANGUP

[Amudhan P]

Hi,

I am getting below error msg in ceph nautilus cluster, do I need to worry
about this?

Oct 14 06:25:02 mon01 ceph-mds[35067]: 2019-10-14 06:25:02.209 7f55a4c48700
-1 received  signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds
ceph-osd ceph-fuse
Oct 14 06:25:02 mon01 ceph-mds[35067]: 2019-10-14 06:25:02.253 7f55a4c48700
-1 received  signal: Hangup from  (PID: 244322) UID: 0
Oct 15 06:25:01 mon01 ceph-mds[35067]: 2019-10-15 06:25:01.988 7f55a4c48700
-1 received  signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds
ceph-osd ceph-fuse
Oct 15 06:25:02 mon01 ceph-mds[35067]: 2019-10-15 06:25:02.040 7f55a4c48700
-1 received  signal: Hangup from pkill -1 -x
ceph-mon|ceph-mgr|ceph-mds|ceph-osd|ceph-fuse|r
Oct 16 06:25:02 mon01 ceph-mds[35067]: 2019-10-16 06:25:02.646 7f55a4c48700
-1 received  signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds
ceph-osd ceph-fuse
Oct 16 06:25:02 mon01 ceph-mds[35067]: 2019-10-16 06:25:02.678 7f55a4c48700
-1 received  signal: Hangup from  (PID: 305528) UID: 0
Oct 17 06:25:02 mon01 ceph-mds[35067]: 2019-10-17 06:25:02.337 7f55a4c48700
-1 received  signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds
ceph-osd ceph-fuse
Oct 17 06:25:02 mon01 ceph-mds[35067]: 2019-10-17 06:25:02.381 7f55a4c48700
-1 received  signal: Hangup from  (PID: 374957) UID: 0
Oct 18 06:25:01 mon01 ceph-mds[35067]: 2019-10-18 06:25:01.947 7f55a4c48700
-1 received  signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds
ceph-osd ceph-fuse
Oct 18 06:25:02 mon01 ceph-mds[35067]: 2019-10-18 06:25:02.015 7f55a4c48700
-1 Fail to open '/proc/436318/cmdline' error = (2) No such file or directory
Oct 18 06:25:02 mon01 ceph-mds[35067]: 2019-10-18 06:25:02.015 7f55a4c48700
-1 received  signal: Hangup from  (PID: 436318) UID: 0
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Change device class in EC profile

[Frank Schilder]

Hi Maks,

thanks for looking at this. Unfortunately, this does not answer the question.

After steps 1-3 you are exactly in the same situation as I am, namely that the 
profile attached to the pool is outdated and, therefore, contains invalid 
information that will be confusing. To check, execute

ceph osd pool get POOL_NAME erasure_code_profile

and then do

ceph osd erasure-code-profile get PROFILE_NAME_RETURNED_ABOVE

This will not match the new crush rule unless there was no change. Note that 
you cannot *set* the EC profile on a pool to match the profile you created your 
new crush rule from. Hence, the pool info will drag outdated information along.

Best regards,

=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14


From: Maks Kowalik 
Sent: 18 October 2019 14:01
To: Frank Schilder
Subject: Re: [ceph-users] Change device class in EC profile

I managed to achieve a similar goal (old EC profile didn't use classes at all) 
by:
1. Creating a new EC profile.
2. Creating new crush rule based on the EC profile.
3. Changing crush rule for a pool to a newly created one.

Kind regards,
Maks

pt., 18 paź 2019 o 13:07 Frank Schilder mailto:fr...@dtu.dk>> 
napisał(a):
I recently moved an EC pool from HDD to SSD by changing the device class in the 
crush rule. I would like to complete this operation by cleaning up a dirty 
trail. The EC profile attached to this pool is called sr-ec-6-2-hdd and it is 
easy enough to rename that to sr-ec-6-2-ssd. However, the profile itself 
contains the device class as well:

crush-device-class=hdd
[...]

I can already see the confusion this will cause in the future. Is this 
situation one of the few instances where using the --force option is warranted 
to change the device class of a profile, as in:

osd erasure-code-profile set sr-ec-6-2-ssd crush-device-class=ssd --force

If not, how can I change the device class of the profile?

Many thanks and best regards,

=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to 
ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: iscsi gate install

[Fyodor Ustinov]

Hi!

Thank you very much!

It remains to understand why this link is not in the documentation. :)

- Original Message -
> From: "Torben Hørup" 
> To: "Fyodor Ustinov" 
> Cc: "ceph-users" 
> Sent: Friday, 18 October, 2019 15:03:09
> Subject: Re: [ceph-users] iscsi gate install

> Take a look at https://shaman.ceph.com/repos/tcmu-runner/
> 
> /Torben
> 
> On 18.10.2019 11:55, Fyodor Ustinov wrote:
>> Hi!
>> 
>> CEPH documentation requre "tcmu-runner-1.4.0 or newer package", but I
>> can not find this package for Centos.
>> 
>> Maybe someone knows where to download this package?
>> 
>> WBR,
>> Fyodor.
>> ___
>> ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: iscsi gate install

[Torben Hørup]

Take a look at https://shaman.ceph.com/repos/tcmu-runner/

/Torben

On 18.10.2019 11:55, Fyodor Ustinov wrote:

Hi!

CEPH documentation requre "tcmu-runner-1.4.0 or newer package", but I
can not find this package for Centos.

Maybe someone knows where to download this package?

WBR,
Fyodor.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Change device class in EC profile

[Frank Schilder]

I recently moved an EC pool from HDD to SSD by changing the device class in the 
crush rule. I would like to complete this operation by cleaning up a dirty 
trail. The EC profile attached to this pool is called sr-ec-6-2-hdd and it is 
easy enough to rename that to sr-ec-6-2-ssd. However, the profile itself 
contains the device class as well:

crush-device-class=hdd
[...]

I can already see the confusion this will cause in the future. Is this 
situation one of the few instances where using the --force option is warranted 
to change the device class of a profile, as in:

osd erasure-code-profile set sr-ec-6-2-ssd crush-device-class=ssd --force

If not, how can I change the device class of the profile?

Many thanks and best regards,

=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] iscsi gate install

[Fyodor Ustinov]

Hi!

CEPH documentation requre "tcmu-runner-1.4.0 or newer package", but I can not 
find this package for Centos.

Maybe someone knows where to download this package?

WBR,
Fyodor.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Monitor unable to join existing cluster, stuck at probing

[Mathijs Smit]

Thank you for taking time to reply to my issue.

I have increased the log level to 10/10 for both the messenger and monitor 
debug and see the following pattern return in the logs. However I do not 
understand the severe high level log that is produced to deduct the problem.

My I again ask for advice?

Log output:

2019-10-18 10:58:28.962 7fd81fc02700  4 mon.mon4@-1(probing) e0 probe_timeout 
0x55de1e9c51a0
2019-10-18 10:58:28.962 7fd81fc02700 10 mon.mon4@-1(probing) e0 bootstrap
2019-10-18 10:58:28.962 7fd81fc02700 10 mon.mon4@-1(probing) e0 
sync_reset_requester
2019-10-18 10:58:28.962 7fd81fc02700 10 mon.mon4@-1(probing) e0 
unregister_cluster_logger - not registered
2019-10-18 10:58:28.962 7fd81fc02700 10 mon.mon4@-1(probing) e0 
cancel_probe_timeout (none scheduled)
2019-10-18 10:58:28.962 7fd81fc02700 10 mon.mon4@-1(probing) e0 _reset
2019-10-18 10:58:28.962 7fd81fc02700 10 mon.mon4@-1(probing) e0 
cancel_probe_timeout (none scheduled)
2019-10-18 10:58:28.962 7fd81fc02700 10 mon.mon4@-1(probing) e0 timecheck_finish
2019-10-18 10:58:28.962 7fd81fc02700 10 mon.mon4@-1(probing) e0 
scrub_event_cancel
2019-10-18 10:58:28.962 7fd81fc02700 10 mon.mon4@-1(probing) e0 scrub_reset
2019-10-18 10:58:28.962 7fd81fc02700 10 mon.mon4@-1(probing) e0 
cancel_probe_timeout (none scheduled)
2019-10-18 10:58:28.962 7fd81fc02700 10 mon.mon4@-1(probing) e0 
reset_probe_timeout 0x55de1e9c5260 after 2 seconds
2019-10-18 10:58:28.962 7fd81fc02700 10 mon.mon4@-1(probing) e0 probing other 
monitors
2019-10-18 10:58:28.962 7fd81fc02700  1 -- 10.200.1.104:6789/0 _send_message--> 
mon.0 10.200.1.101:6789/0 -- mon_probe(probe 
aaf1547b-8944-4f48-b354-93659202c6fe name mon4 new) v6 -- ?+0 0x55de1e9e5400
2019-10-18 10:58:28.962 7fd81fc02700  1 -- 10.200.1.104:6789/0 --> 
10.200.1.101:6789/0 -- mon_probe(probe aaf1547b-8944-4f48-b354-93659202c6fe 
name mon4 new) v6 -- 0x55de1e9e5400 con 0
2019-10-18 10:58:28.962 7fd81fc02700  1 -- 10.200.1.104:6789/0 _send_message--> 
mon.1 10.200.1.102:6789/0 -- mon_probe(probe 
aaf1547b-8944-4f48-b354-93659202c6fe name mon4 new) v6 -- ?+0 0x55de1e9e5680
2019-10-18 10:58:28.962 7fd81fc02700  1 -- 10.200.1.104:6789/0 --> 
10.200.1.102:6789/0 -- mon_probe(probe aaf1547b-8944-4f48-b354-93659202c6fe 
name mon4 new) v6 -- 0x55de1e9e5680 con 0
2019-10-18 10:58:28.962 7fd81fc02700  1 -- 10.200.1.104:6789/0 _send_message--> 
mon.2 10.200.1.103:6789/0 -- mon_probe(probe 
aaf1547b-8944-4f48-b354-93659202c6fe name mon4 new) v6 -- ?+0 0x55de1e9e5900
2019-10-18 10:58:28.962 7fd81fc02700  1 -- 10.200.1.104:6789/0 --> 
10.200.1.103:6789/0 -- mon_probe(probe aaf1547b-8944-4f48-b354-93659202c6fe 
name mon4 new) v6 -- 0x55de1e9e5900 con 0
2019-10-18 10:58:28.962 7fd81abf8700 10 -- 10.200.1.104:6789/0 >> 
10.200.1.101:6789/0 conn(0x55de1e7d3e00 :-1 s=STATE_OPEN pgs=2274435 cs=1 
l=0).handle_write
2019-10-18 10:58:28.962 7fd819bf6700 10 -- 10.200.1.104:6789/0 >> 
10.200.1.102:6789/0 conn(0x55de1e7d4400 :-1 s=STATE_OPEN pgs=2284339 cs=1 
l=0).handle_write
2019-10-18 10:58:28.962 7fd81a3f7700 10 -- 10.200.1.104:6789/0 >> 
10.200.1.103:6789/0 conn(0x55de1e7d4a00 :-1 s=STATE_OPEN pgs=2288108 cs=1 
l=0).handle_write
2019-10-18 10:58:28.963 7fd81abf8700 10 -- 10.200.1.104:6789/0 >> 
10.200.1.101:6789/0 conn(0x55de1e7d3e00 :-1 s=STATE_OPEN pgs=2274435 cs=1 
l=0)._try_send sent bytes 136 remaining bytes 0
2019-10-18 10:58:28.963 7fd81abf8700 10 -- 10.200.1.104:6789/0 >> 
10.200.1.101:6789/0 conn(0x55de1e7d3e00 :-1 s=STATE_OPEN pgs=2274435 cs=1 
l=0).write_message sending 0x55de1e9e5400 done.
2019-10-18 10:58:28.963 7fd81a3f7700 10 -- 10.200.1.104:6789/0 >> 
10.200.1.103:6789/0 conn(0x55de1e7d4a00 :-1 s=STATE_OPEN pgs=2288108 cs=1 
l=0)._try_send sent bytes 136 remaining bytes 0
2019-10-18 10:58:28.963 7fd81a3f7700 10 -- 10.200.1.104:6789/0 >> 
10.200.1.103:6789/0 conn(0x55de1e7d4a00 :-1 s=STATE_OPEN pgs=2288108 cs=1 
l=0).write_message sending 0x55de1e9e5900 done.
2019-10-18 10:58:28.963 7fd819bf6700 10 -- 10.200.1.104:6789/0 >> 
10.200.1.102:6789/0 conn(0x55de1e7d4400 :-1 s=STATE_OPEN pgs=2284339 cs=1 
l=0)._try_send sent bytes 136 remaining bytes 0
2019-10-18 10:58:28.963 7fd819bf6700 10 -- 10.200.1.104:6789/0 >> 
10.200.1.102:6789/0 conn(0x55de1e7d4400 :-1 s=STATE_OPEN pgs=2284339 cs=1 
l=0).write_message sending 0x55de1e9e5680 done.
2019-10-18 10:58:28.963 7fd81abf8700 10 -- 10.200.1.104:6789/0 >> 
10.200.1.101:6789/0 conn(0x55de1e7d3e00 :-1 s=STATE_OPEN_TAG_ACK pgs=2274435 
cs=1 l=0).handle_ack got ack seq 20 >= 20 on 0x55de1e9e5400 mon_probe(probe 
aaf1547b-8944-4f48-b354-93659202c6fe name mon4 new) v6
2019-10-18 10:58:28.963 7fd81a3f7700 10 -- 10.200.1.104:6789/0 >> 
10.200.1.103:6789/0 conn(0x55de1e7d4a00 :-1 s=STATE_OPEN_TAG_ACK pgs=2288108 
cs=1 l=0).handle_ack got ack seq 20 >= 20 on 0x55de1e9e5900 mon_probe(probe 
aaf1547b-8944-4f48-b354-93659202c6fe name mon4 new) v6
2019-10-18 10:58:28.963 7fd819bf6700 10 -- 10.200.1.104:6789/0 >> 
10.200.1.102:6789/0 conn(0x55de1e7d4400 :-1 s=STATE_OPEN_TAG_ACK pgs=2284339 
cs