RE: [PATCH v4 2/2] net/colo-compare.c: handling of the full primary or secondary queue

2020-04-09 Thread Zhang, Chen


> -Original Message-
> From: Derek Su 
> Sent: Thursday, April 9, 2020 3:11 PM
> To: Zhang, Chen 
> Cc: Lukas Straub ; Derek Su ;
> qemu-devel@nongnu.org; lizhij...@cn.fujitsu.com; chy...@qnap.com;
> jasow...@redhat.com; ctch...@qnap.com
> Subject: Re: [PATCH v4 2/2] net/colo-compare.c: handling of the full primary
> or secondary queue
> 
> Hello, Zhang and Lukas
> 
> Sure, after my re-test, the performance is hurt. Will update it later.
> 
> By the way, could I also move the "error_report("colo compare
> primary/secondary queue size too big, drop packet");" to trace?

Yes.

> The use of error_report is a little strange and make a flood in log.
> 
> May I  also make "MAX_QUEUE_SIZE"  be user-configurable in this series?

I have this patch internal and will send it later.

Thanks
Zhang Chen

> 
> Thanks,
> Derek Su
> 
> 
> 
> 
> Zhang, Chen  於 2020年4月9日 週四 下午2:59
> 寫道:
> >
> >
> >
> > > -Original Message-
> > > From: Lukas Straub 
> > > Sent: Thursday, April 9, 2020 3:19 AM
> > > To: Derek Su 
> > > Cc: qemu-devel@nongnu.org; lizhij...@cn.fujitsu.com;
> > > chy...@qnap.com; jasow...@redhat.com; ctch...@qnap.com; Zhang,
> Chen
> > > ; jwsu1...@gmail.com
> > > Subject: Re: [PATCH v4 2/2] net/colo-compare.c: handling of the full
> > > primary or secondary queue
> > >
> > > On Sat, 28 Mar 2020 20:46:46 +0800
> > > Derek Su  wrote:
> > >
> > > > The pervious handling of the full primary or queue is only
> > > > dropping the packet. If there are lots of clients to the guest VM, the
> "drop"
> > > > will lead to the lost of the networking connection until next
> > > > checkpoint.
> > > >
> > > > To address the issue, this patch drops the packet firstly.
> > > > Then, do checkpoint and flush packets.
> > > >
> > > > Signed-off-by: Derek Su 
> > >
> > > Hello,
> > > I had a look at this again and did some benchmarking.
> > > First just qemu 5.0-rc1 with my bugfixes (
> > > https://lists.nongnu.org/archive/html/qemu-devel/2020-
> > > 04/msg01432.html ) Then qemu 5.0-rc1 with my bugfixes and this patch
> > > series.
> > >
> > > This commit hurts performance too much:
> > > Client-to-server bandwidth falls from ~45.9 Mbit/s to 22.9 Mbit/s.
> > > Server-to-client bandwidth falls from ~6.3 Mbit/s to just ~674 Kbit/s.
> > > Average latency rises from ~197ms to ~397ms.
> > >
> > > Meanwhile the packet loss without this commit is negligible, only
> > > 1-2 ping packets got lost during each test run.
> > >
> > > Instead I think we should just turn the error message into a trace
> > > so it doesn't flood the logs.
> >
> > We re-test this patch, Lukas is right.
> > Sorry for the original idea, looks like it did not show better performance 
> > in
> the test.
> > Agree with Lukas's comments. Derek, can you please change it?
> >
> > Thanks
> > Zhang Chen
> >
> >
> > >
> > > Regards,
> > > Lukas Straub


Re: [PATCH v4 2/2] net/colo-compare.c: handling of the full primary or secondary queue

2020-04-09 Thread Derek Su
Hello, Zhang and Lukas

Sure, after my re-test, the performance is hurt. Will update it later.

By the way, could I also move the "error_report("colo compare
primary/secondary queue size too big, drop packet");" to trace?
The use of error_report is a little strange and make a flood in log.

May I  also make "MAX_QUEUE_SIZE"  be user-configurable in this series?

Thanks,
Derek Su




Zhang, Chen  於 2020年4月9日 週四 下午2:59寫道:
>
>
>
> > -Original Message-
> > From: Lukas Straub 
> > Sent: Thursday, April 9, 2020 3:19 AM
> > To: Derek Su 
> > Cc: qemu-devel@nongnu.org; lizhij...@cn.fujitsu.com; chy...@qnap.com;
> > jasow...@redhat.com; ctch...@qnap.com; Zhang, Chen
> > ; jwsu1...@gmail.com
> > Subject: Re: [PATCH v4 2/2] net/colo-compare.c: handling of the full primary
> > or secondary queue
> >
> > On Sat, 28 Mar 2020 20:46:46 +0800
> > Derek Su  wrote:
> >
> > > The pervious handling of the full primary or queue is only dropping
> > > the packet. If there are lots of clients to the guest VM, the "drop"
> > > will lead to the lost of the networking connection until next
> > > checkpoint.
> > >
> > > To address the issue, this patch drops the packet firstly.
> > > Then, do checkpoint and flush packets.
> > >
> > > Signed-off-by: Derek Su 
> >
> > Hello,
> > I had a look at this again and did some benchmarking.
> > First just qemu 5.0-rc1 with my bugfixes
> > ( https://lists.nongnu.org/archive/html/qemu-devel/2020-
> > 04/msg01432.html ) Then qemu 5.0-rc1 with my bugfixes and this patch
> > series.
> >
> > This commit hurts performance too much:
> > Client-to-server bandwidth falls from ~45.9 Mbit/s to 22.9 Mbit/s.
> > Server-to-client bandwidth falls from ~6.3 Mbit/s to just ~674 Kbit/s.
> > Average latency rises from ~197ms to ~397ms.
> >
> > Meanwhile the packet loss without this commit is negligible, only 1-2 ping
> > packets got lost during each test run.
> >
> > Instead I think we should just turn the error message into a trace so it
> > doesn't flood the logs.
>
> We re-test this patch, Lukas is right.
> Sorry for the original idea, looks like it did not show better performance in 
> the test.
> Agree with Lukas's comments. Derek, can you please change it?
>
> Thanks
> Zhang Chen
>
>
> >
> > Regards,
> > Lukas Straub



RE: [PATCH v4 2/2] net/colo-compare.c: handling of the full primary or secondary queue

2020-04-09 Thread Zhang, Chen



> -Original Message-
> From: Lukas Straub 
> Sent: Thursday, April 9, 2020 3:19 AM
> To: Derek Su 
> Cc: qemu-devel@nongnu.org; lizhij...@cn.fujitsu.com; chy...@qnap.com;
> jasow...@redhat.com; ctch...@qnap.com; Zhang, Chen
> ; jwsu1...@gmail.com
> Subject: Re: [PATCH v4 2/2] net/colo-compare.c: handling of the full primary
> or secondary queue
> 
> On Sat, 28 Mar 2020 20:46:46 +0800
> Derek Su  wrote:
> 
> > The pervious handling of the full primary or queue is only dropping
> > the packet. If there are lots of clients to the guest VM, the "drop"
> > will lead to the lost of the networking connection until next
> > checkpoint.
> >
> > To address the issue, this patch drops the packet firstly.
> > Then, do checkpoint and flush packets.
> >
> > Signed-off-by: Derek Su 
> 
> Hello,
> I had a look at this again and did some benchmarking.
> First just qemu 5.0-rc1 with my bugfixes
> ( https://lists.nongnu.org/archive/html/qemu-devel/2020-
> 04/msg01432.html ) Then qemu 5.0-rc1 with my bugfixes and this patch
> series.
> 
> This commit hurts performance too much:
> Client-to-server bandwidth falls from ~45.9 Mbit/s to 22.9 Mbit/s.
> Server-to-client bandwidth falls from ~6.3 Mbit/s to just ~674 Kbit/s.
> Average latency rises from ~197ms to ~397ms.
> 
> Meanwhile the packet loss without this commit is negligible, only 1-2 ping
> packets got lost during each test run.
> 
> Instead I think we should just turn the error message into a trace so it
> doesn't flood the logs.

We re-test this patch, Lukas is right.
Sorry for the original idea, looks like it did not show better performance in 
the test.
Agree with Lukas's comments. Derek, can you please change it?

Thanks
Zhang Chen


> 
> Regards,
> Lukas Straub



Re: [PATCH v4 2/2] net/colo-compare.c: handling of the full primary or secondary queue

2020-04-08 Thread Lukas Straub
On Sat, 28 Mar 2020 20:46:46 +0800
Derek Su  wrote:

> The pervious handling of the full primary or queue is only dropping
> the packet. If there are lots of clients to the guest VM,
> the "drop" will lead to the lost of the networking connection
> until next checkpoint.
> 
> To address the issue, this patch drops the packet firstly.
> Then, do checkpoint and flush packets.
> 
> Signed-off-by: Derek Su 

Hello,
I had a look at this again and did some benchmarking.
First just qemu 5.0-rc1 with my bugfixes ( 
https://lists.nongnu.org/archive/html/qemu-devel/2020-04/msg01432.html )
Then qemu 5.0-rc1 with my bugfixes and this patch series.

This commit hurts performance too much:
Client-to-server bandwidth falls from ~45.9 Mbit/s to 22.9 Mbit/s.
Server-to-client bandwidth falls from ~6.3 Mbit/s to just ~674 Kbit/s.
Average latency rises from ~197ms to ~397ms.

Meanwhile the packet loss without this commit is negligible,
only 1-2 ping packets got lost during each test run.

Instead I think we should just turn the error message
into a trace so it doesn't flood the logs.

Regards,
Lukas Straub
5.0-rc1 with bugfixes:

*** iperf -c 192.168.178.65 ***

[  3] local 192.168.178.68 port 33418 connected with 192.168.178.65 port 5001   
64 bytes from 192.168.178.65: icmp_seq=53 ttl=64 
time=2.57 ms
[ ID] Interval   Transfer Bandwidth 
64 bytes from 192.168.178.65: icmp_seq=54 ttl=64 
time=142 ms
[  3]  0.0-10.2 sec  60.1 MBytes  49.4 Mbits/sec
64 bytes from 192.168.178.65: icmp_seq=55 ttl=64 
time=2.64 ms

[  3] Sent 892 datagrams
rtt min/avg/max/mdev = 1.920/235.514/1810.653/411.394 
ms, pipe 4

[  3] local 192.168.178.68 port 33420 connected with 192.168.178.65 port 5001   
64 bytes from 192.168.178.65: icmp_seq=55 ttl=64 
time=24.3 ms
[ ID] Interval   Transfer Bandwidth 
64 bytes from 192.168.178.65: icmp_seq=56 ttl=64 
time=505 ms
[  3]  0.0-10.3 sec  51.0 MBytes  41.7 Mbits/sec
64 bytes from 192.168.178.65: icmp_seq=57 ttl=64 
time=11.7 ms

[  3] Sent 892 datagrams
rtt min/avg/max/mdev = 1.962/218.713/1365.510/344.164 
ms, pipe 3

[  3] local 192.168.178.68 port 33422 connected with 192.168.178.65 port 5001   
64 bytes from 192.168.178.65: icmp_seq=54 ttl=64 
time=49.2 ms
[ ID] Interval   Transfer Bandwidth 
64 bytes from 192.168.178.65: icmp_seq=55 ttl=64 
time=587 ms
[  3]  0.0-10.8 sec  60.1 MBytes  46.6 Mbits/sec
64 bytes from 192.168.178.65: icmp_seq=56 ttl=64 
time=167 ms

[  3] Sent 892 datagrams
rtt min/avg/max/mdev = 1.573/276.375/1384.085/362.027 
ms, pipe 3


*** iperf -c 192.168.178.65 -d ***

[  4] local 192.168.178.68 port 33424 connected with 192.168.178.65 port 5001   
64 bytes from 192.168.178.65: icmp_seq=60 ttl=64 
time=2.27 ms
[  5] local 192.168.178.68 port 5001 connected with 192.168.178.65 port 48440   
^C
[ ID] Interval   Transfer Bandwidth 
--- 192.168.178.65 ping statistics ---
[  4]  0.0-10.1 sec  60.8 MBytes  50.4 Mbits/sec
60 packets transmitted, 59 received, 1.7% packet 
loss, time 29595ms
[  5]  0.0-11.7 sec  11.2 MBytes  8.05 Mbits/sec
rtt min/avg/max/mdev = 1.785/254.258/1352.916/395.070 
ms, pipe 3

[  5] local 192.168.178.68 port 33426 connected with 192.168.178.65 port 5001   
64 bytes from 192.168.178.65: icmp_seq=59 ttl=64 
time=2.98 ms
[  4] local 192.168.178.68 port 5001 connected with 192.168.178.65 port 48442   
^C
[ ID] Interval   Transfer Bandwidth 
--- 192.168.178.65 ping statistics ---
[  5]  0.0-10.3 sec  58.6 MBytes  47.6 Mbits/sec
59 packets transmitted, 59 received, 0% packet loss, 
time 29088ms
[  4]  0.0-11.9 sec  13.8 MBytes  9.77 Mbits/sec
rtt min/avg/max/mdev = 2.026/207.507/1518.013/331.972 
ms, pipe 4

[  5] local 192.168.178.68 port 33432 connected with 192.168.178.65 port 5001   
64 bytes from 192.168.178.65: icmp_seq=93 ttl=64 
time=12.6 ms
[  4] local 192.168.178.68 port 5001 connected with 192.168.178.65 port 

Re: [PATCH v4 2/2] net/colo-compare.c: handling of the full primary or secondary queue

2020-04-05 Thread Lukas Straub
On Sat, 28 Mar 2020 20:46:46 +0800
Derek Su  wrote:

> The pervious handling of the full primary or queue is only dropping
> the packet. If there are lots of clients to the guest VM,
> the "drop" will lead to the lost of the networking connection
> until next checkpoint.
> 
> To address the issue, this patch drops the packet firstly.
> Then, do checkpoint and flush packets.
> 
> Signed-off-by: Derek Su 

Looks good and works well in my tests.
Reviewed-by: Lukas Straub 
Tested-by: Lukas Straub 

Regards,
Lukas Straub

> ---
>  net/colo-compare.c | 39 ---
>  1 file changed, 28 insertions(+), 11 deletions(-)
> 
> diff --git a/net/colo-compare.c b/net/colo-compare.c
> index cdd87b2aa8..fe8779cf2d 100644
> --- a/net/colo-compare.c
> +++ b/net/colo-compare.c
> @@ -125,6 +125,12 @@ static const char *colo_mode[] = {
>  [SECONDARY_IN] = "secondary",
>  };
>  
> +enum {
> +QUEUE_INSERT_ERR = -1,
> +QUEUE_INSERT_OK = 0,
> +QUEUE_INSERT_FULL = 1,
> +};
> +
>  static int compare_chr_send(CompareState *s,
>  const uint8_t *buf,
>  uint32_t size,
> @@ -211,8 +217,10 @@ static int colo_insert_packet(GQueue *queue, Packet 
> *pkt, uint32_t *max_ack)
>  }
>  
>  /*
> - * Return 0 on success, if return -1 means the pkt
> - * is unsupported(arp and ipv6) and will be sent later
> + * Return QUEUE_INSERT_OK on success.
> + * If return QUEUE_INSERT_FULL means list is full, and
> + * QUEUE_INSERT_ERR means the pkt is unsupported(arp and ipv6) and
> + * will be sent later
>   */
>  static int packet_enqueue(CompareState *s, int mode, Connection **con)
>  {
> @@ -234,7 +242,7 @@ static int packet_enqueue(CompareState *s, int mode, 
> Connection **con)
>  if (parse_packet_early(pkt)) {
>  packet_destroy(pkt, NULL);
>  pkt = NULL;
> -return -1;
> +return QUEUE_INSERT_ERR;
>  }
>  fill_connection_key(pkt, );
>  
> @@ -258,11 +266,12 @@ static int packet_enqueue(CompareState *s, int mode, 
> Connection **con)
>   "drop packet", colo_mode[mode]);
>  packet_destroy(pkt, NULL);
>  pkt = NULL;
> +return QUEUE_INSERT_FULL;
>  }
>  
>  *con = conn;
>  
> -return 0;
> +return QUEUE_INSERT_OK;
>  }
>  
>  static inline bool after(uint32_t seq1, uint32_t seq2)
> @@ -995,17 +1004,21 @@ static void compare_pri_rs_finalize(SocketReadState 
> *pri_rs)
>  {
>  CompareState *s = container_of(pri_rs, CompareState, pri_rs);
>  Connection *conn = NULL;
> +int ret;
>  
> -if (packet_enqueue(s, PRIMARY_IN, )) {
> +ret = packet_enqueue(s, PRIMARY_IN, );
> +if (ret == QUEUE_INSERT_OK) {
> +/* compare packet in the specified connection */
> +colo_compare_connection(conn, s);
> +} else if (ret == QUEUE_INSERT_FULL) {
> +colo_compare_inconsistency_notify(s);
> +} else {
>  trace_colo_compare_main("primary: unsupported packet in");
>  compare_chr_send(s,
>   pri_rs->buf,
>   pri_rs->packet_len,
>   pri_rs->vnet_hdr_len,
>   false);
> -} else {
> -/* compare packet in the specified connection */
> -colo_compare_connection(conn, s);
>  }
>  }
>  
> @@ -1013,12 +1026,16 @@ static void compare_sec_rs_finalize(SocketReadState 
> *sec_rs)
>  {
>  CompareState *s = container_of(sec_rs, CompareState, sec_rs);
>  Connection *conn = NULL;
> +int ret;
>  
> -if (packet_enqueue(s, SECONDARY_IN, )) {
> -trace_colo_compare_main("secondary: unsupported packet in");
> -} else {
> +ret = packet_enqueue(s, SECONDARY_IN, );
> +if (ret == QUEUE_INSERT_OK) {
>  /* compare packet in the specified connection */
>  colo_compare_connection(conn, s);
> +} else if (ret == QUEUE_INSERT_FULL) {
> +colo_compare_inconsistency_notify(s);
> +} else {
> +trace_colo_compare_main("secondary: unsupported packet in");
>  }
>  }
>  



pgp7uyYoK4C26.pgp
Description: OpenPGP digital signature


RE: [PATCH v4 2/2] net/colo-compare.c: handling of the full primary or secondary queue

2020-03-30 Thread Zhang, Chen


> Subject: [PATCH v4 2/2] net/colo-compare.c: handling of the full primary or
> secondary queue
> 
> The pervious handling of the full primary or queue is only dropping the
> packet. If there are lots of clients to the guest VM, the "drop" will lead to 
> the
> lost of the networking connection until next checkpoint.
> 
> To address the issue, this patch drops the packet firstly.
> Then, do checkpoint and flush packets.
> 
> Signed-off-by: Derek Su 

Looks good for me.
Reviewed-by: Zhang Chen 

Thanks
Zhang Chen

> ---
>  net/colo-compare.c | 39 ---
>  1 file changed, 28 insertions(+), 11 deletions(-)
> 
> diff --git a/net/colo-compare.c b/net/colo-compare.c index
> cdd87b2aa8..fe8779cf2d 100644
> --- a/net/colo-compare.c
> +++ b/net/colo-compare.c
> @@ -125,6 +125,12 @@ static const char *colo_mode[] = {
>  [SECONDARY_IN] = "secondary",
>  };
> 
> +enum {
> +QUEUE_INSERT_ERR = -1,
> +QUEUE_INSERT_OK = 0,
> +QUEUE_INSERT_FULL = 1,
> +};
> +
>  static int compare_chr_send(CompareState *s,
>  const uint8_t *buf,
>  uint32_t size, @@ -211,8 +217,10 @@ static int
> colo_insert_packet(GQueue *queue, Packet *pkt, uint32_t *max_ack)  }
> 
>  /*
> - * Return 0 on success, if return -1 means the pkt
> - * is unsupported(arp and ipv6) and will be sent later
> + * Return QUEUE_INSERT_OK on success.
> + * If return QUEUE_INSERT_FULL means list is full, and
> + * QUEUE_INSERT_ERR means the pkt is unsupported(arp and ipv6) and
> + * will be sent later
>   */
>  static int packet_enqueue(CompareState *s, int mode, Connection **con)
> { @@ -234,7 +242,7 @@ static int packet_enqueue(CompareState *s, int
> mode, Connection **con)
>  if (parse_packet_early(pkt)) {
>  packet_destroy(pkt, NULL);
>  pkt = NULL;
> -return -1;
> +return QUEUE_INSERT_ERR;
>  }
>  fill_connection_key(pkt, );
> 
> @@ -258,11 +266,12 @@ static int packet_enqueue(CompareState *s, int
> mode, Connection **con)
>   "drop packet", colo_mode[mode]);
>  packet_destroy(pkt, NULL);
>  pkt = NULL;
> +return QUEUE_INSERT_FULL;
>  }
> 
>  *con = conn;
> 
> -return 0;
> +return QUEUE_INSERT_OK;
>  }
> 
>  static inline bool after(uint32_t seq1, uint32_t seq2) @@ -995,17 +1004,21
> @@ static void compare_pri_rs_finalize(SocketReadState *pri_rs)  {
>  CompareState *s = container_of(pri_rs, CompareState, pri_rs);
>  Connection *conn = NULL;
> +int ret;
> 
> -if (packet_enqueue(s, PRIMARY_IN, )) {
> +ret = packet_enqueue(s, PRIMARY_IN, );
> +if (ret == QUEUE_INSERT_OK) {
> +/* compare packet in the specified connection */
> +colo_compare_connection(conn, s);
> +} else if (ret == QUEUE_INSERT_FULL) {
> +colo_compare_inconsistency_notify(s);
> +} else {
>  trace_colo_compare_main("primary: unsupported packet in");
>  compare_chr_send(s,
>   pri_rs->buf,
>   pri_rs->packet_len,
>   pri_rs->vnet_hdr_len,
>   false);
> -} else {
> -/* compare packet in the specified connection */
> -colo_compare_connection(conn, s);
>  }
>  }
> 
> @@ -1013,12 +1026,16 @@ static void
> compare_sec_rs_finalize(SocketReadState *sec_rs)  {
>  CompareState *s = container_of(sec_rs, CompareState, sec_rs);
>  Connection *conn = NULL;
> +int ret;
> 
> -if (packet_enqueue(s, SECONDARY_IN, )) {
> -trace_colo_compare_main("secondary: unsupported packet in");
> -} else {
> +ret = packet_enqueue(s, SECONDARY_IN, );
> +if (ret == QUEUE_INSERT_OK) {
>  /* compare packet in the specified connection */
>  colo_compare_connection(conn, s);
> +} else if (ret == QUEUE_INSERT_FULL) {
> +colo_compare_inconsistency_notify(s);
> +} else {
> +trace_colo_compare_main("secondary: unsupported packet in");
>  }
>  }
> 
> --
> 2.17.1




[PATCH v4 2/2] net/colo-compare.c: handling of the full primary or secondary queue

2020-03-28 Thread Derek Su
The pervious handling of the full primary or queue is only dropping
the packet. If there are lots of clients to the guest VM,
the "drop" will lead to the lost of the networking connection
until next checkpoint.

To address the issue, this patch drops the packet firstly.
Then, do checkpoint and flush packets.

Signed-off-by: Derek Su 
---
 net/colo-compare.c | 39 ---
 1 file changed, 28 insertions(+), 11 deletions(-)

diff --git a/net/colo-compare.c b/net/colo-compare.c
index cdd87b2aa8..fe8779cf2d 100644
--- a/net/colo-compare.c
+++ b/net/colo-compare.c
@@ -125,6 +125,12 @@ static const char *colo_mode[] = {
 [SECONDARY_IN] = "secondary",
 };
 
+enum {
+QUEUE_INSERT_ERR = -1,
+QUEUE_INSERT_OK = 0,
+QUEUE_INSERT_FULL = 1,
+};
+
 static int compare_chr_send(CompareState *s,
 const uint8_t *buf,
 uint32_t size,
@@ -211,8 +217,10 @@ static int colo_insert_packet(GQueue *queue, Packet *pkt, 
uint32_t *max_ack)
 }
 
 /*
- * Return 0 on success, if return -1 means the pkt
- * is unsupported(arp and ipv6) and will be sent later
+ * Return QUEUE_INSERT_OK on success.
+ * If return QUEUE_INSERT_FULL means list is full, and
+ * QUEUE_INSERT_ERR means the pkt is unsupported(arp and ipv6) and
+ * will be sent later
  */
 static int packet_enqueue(CompareState *s, int mode, Connection **con)
 {
@@ -234,7 +242,7 @@ static int packet_enqueue(CompareState *s, int mode, 
Connection **con)
 if (parse_packet_early(pkt)) {
 packet_destroy(pkt, NULL);
 pkt = NULL;
-return -1;
+return QUEUE_INSERT_ERR;
 }
 fill_connection_key(pkt, );
 
@@ -258,11 +266,12 @@ static int packet_enqueue(CompareState *s, int mode, 
Connection **con)
  "drop packet", colo_mode[mode]);
 packet_destroy(pkt, NULL);
 pkt = NULL;
+return QUEUE_INSERT_FULL;
 }
 
 *con = conn;
 
-return 0;
+return QUEUE_INSERT_OK;
 }
 
 static inline bool after(uint32_t seq1, uint32_t seq2)
@@ -995,17 +1004,21 @@ static void compare_pri_rs_finalize(SocketReadState 
*pri_rs)
 {
 CompareState *s = container_of(pri_rs, CompareState, pri_rs);
 Connection *conn = NULL;
+int ret;
 
-if (packet_enqueue(s, PRIMARY_IN, )) {
+ret = packet_enqueue(s, PRIMARY_IN, );
+if (ret == QUEUE_INSERT_OK) {
+/* compare packet in the specified connection */
+colo_compare_connection(conn, s);
+} else if (ret == QUEUE_INSERT_FULL) {
+colo_compare_inconsistency_notify(s);
+} else {
 trace_colo_compare_main("primary: unsupported packet in");
 compare_chr_send(s,
  pri_rs->buf,
  pri_rs->packet_len,
  pri_rs->vnet_hdr_len,
  false);
-} else {
-/* compare packet in the specified connection */
-colo_compare_connection(conn, s);
 }
 }
 
@@ -1013,12 +1026,16 @@ static void compare_sec_rs_finalize(SocketReadState 
*sec_rs)
 {
 CompareState *s = container_of(sec_rs, CompareState, sec_rs);
 Connection *conn = NULL;
+int ret;
 
-if (packet_enqueue(s, SECONDARY_IN, )) {
-trace_colo_compare_main("secondary: unsupported packet in");
-} else {
+ret = packet_enqueue(s, SECONDARY_IN, );
+if (ret == QUEUE_INSERT_OK) {
 /* compare packet in the specified connection */
 colo_compare_connection(conn, s);
+} else if (ret == QUEUE_INSERT_FULL) {
+colo_compare_inconsistency_notify(s);
+} else {
+trace_colo_compare_main("secondary: unsupported packet in");
 }
 }
 
-- 
2.17.1