SOLUÇÕES GRÁFICAS
SOLUÇÕES GRÁFICAS
[PATCH RFC v2] ipvs: add genetlink cmd to dump all services and destinations
A common operation for userspace applications managing ipvs is to dump all
services and all destinations and then sort out what needs to be done.
Previously this could only be accomplished by issuing 1 netlink
IPVS_CMD_GET_SERVICE dump command followed by N IPVS_CMD_GET_DEST dump
commands. For a dynamic system with a very large number of services this
could be cause a performance impact.
This patch introduces a new way of dumping all services and destinations
with the new IPVS_CMD_GET_SERVICE_DEST command. A dump call for this
command will send the services as IPVS_CMD_NEW_SERVICE messages each
containing a nested array of destinations with the new IPVS_SVC_ATTR_DESTS
and IPVS_DESTS_ATTR_DEST attributes. Services may be repeated if their
destinations do not fit in a single packet, user space should be
responsible for merging the destinations from each repeated service.
It's also possible to dump a single service and its destinations by sending
a IPVS_CMD_ATTR_SERVICE argument to the dump command.
Signed-off-by: Cezar Sa Espinola
---
Changes in v2:
- Send destinations nested in a service.
This patch should be safer now as every packet starts with a service
message. Also Julian's idea of adding more coordinates for tab and row
showed a nice performance improvement. I think the same idea could also
be applied to the existing ip_vs_genl_dump_services, I can send a patch
for it later.
I also verified that each packet can fit somewhere around 50
destinations on my system. So I was able to verify that it works as
expected even when services are repeated including the additional
destinations.
As before a patched ipvsadm for the message format in this commit is
available in [1] and benchmarks in [2] were executed comparing runtime
of "ipvsadm -Sn" using the new command vs the old command. Here are the
benchmark summary:
tcp svcs | fw svcs | dsts | run time
- | --- | |
1000 | 0 |4 | -62.12%
2000 | 0 |2 | -72.97%
8000 | 0 |2 | -73.47%
16000 | 0 |1 | -81.79%
100 | 0 | 100 | -9.09%
8000 |8000 |1 | -81.10%
[1] - https://github.com/cezarsa/ipvsadm/compare/master...dump-svc-ds
[2] - https://github.com/cezarsa/ipvsadm-validate#benchmark-results
include/uapi/linux/ip_vs.h | 17 ++-
net/netfilter/ipvs/ip_vs_ctl.c | 188 ++---
2 files changed, 189 insertions(+), 16 deletions(-)
diff --git a/include/uapi/linux/ip_vs.h b/include/uapi/linux/ip_vs.h
index 4102ddcb4e14..ce9bfa03b61b 100644
--- a/include/uapi/linux/ip_vs.h
+++ b/include/uapi/linux/ip_vs.h
@@ -331,6 +331,8 @@ enum {
IPVS_CMD_ZERO, /* zero all counters and stats */
IPVS_CMD_FLUSH, /* flush services and dests */
+ IPVS_CMD_GET_SERVICE_DEST, /* get service and destination info */
+
__IPVS_CMD_MAX,
};
@@ -374,15 +376,28 @@ enum {
IPVS_SVC_ATTR_STATS64, /* nested attribute for service stats */
+ IPVS_SVC_ATTR_DESTS,/* nested destinations */
+
__IPVS_SVC_ATTR_MAX,
};
#define IPVS_SVC_ATTR_MAX (__IPVS_SVC_ATTR_MAX - 1)
+enum {
+ IPVS_DESTS_ATTR_UNSPEC = 0,
+
+ IPVS_DESTS_ATTR_DEST, /* nested destination */
+
+ __IPVS_DESTS_ATTR_MAX,
+};
+
+#define IPVS_DESTS_ATTR_MAX (__IPVS_DESTS_ATTR_MAX - 1)
+
/*
* Attributes used to describe a destination (real server)
*
- * Used inside nested attribute IPVS_CMD_ATTR_DEST
+ * Used inside nested attribute IPVS_CMD_ATTR_DEST and
+ * IPVS_DESTS_ATTR_DEST
*/
enum {
IPVS_DEST_ATTR_UNSPEC = 0,
diff --git a/net/netfilter/ipvs/ip_vs_ctl.c b/net/netfilter/ipvs/ip_vs_ctl.c
index e279ded4e306..9f40627f74ef 100644
--- a/net/netfilter/ipvs/ip_vs_ctl.c
+++ b/net/netfilter/ipvs/ip_vs_ctl.c
@@ -2981,6 +2981,7 @@ static const struct nla_policy
ip_vs_svc_policy[IPVS_SVC_ATTR_MAX + 1] = {
[IPVS_SVC_ATTR_TIMEOUT] = { .type = NLA_U32 },
[IPVS_SVC_ATTR_NETMASK] = { .type = NLA_U32 },
[IPVS_SVC_ATTR_STATS] = { .type = NLA_NESTED },
+ [IPVS_SVC_ATTR_DESTS] = { .type = NLA_NESTED },
};
/* Policy used for attributes in nested attribute IPVS_CMD_ATTR_DEST */
@@ -3070,31 +3071,26 @@ static int ip_vs_genl_fill_stats64(struct sk_buff *skb,
int container_type,
return -EMSGSIZE;
}
-static int ip_vs_genl_fill_service(struct sk_buff *skb,
- struct ip_vs_service *svc)
+static int ip_vs_genl_put_service_attrs(struct sk_buff *skb,
+ struct ip_vs_service *svc)
{
struct ip_vs_scheduler *sched;
struct ip_vs_pe *pe;
- struct nlattr *nl_service;
struct ip_vs_flags flags = { .flags = svc->flags,
.mask = ~0 };
struct ip_vs_kstats kstats;
char *sched_name;
- nl_service = nla_nest_start_noflag(skb
[PATCH RFC v3] ipvs: add genetlink cmd to dump all services and destinations
A common operation for userspace applications managing ipvs is to dump all
services and all destinations and then sort out what needs to be done.
Previously this could only be accomplished by issuing 1 netlink
IPVS_CMD_GET_SERVICE dump command followed by N IPVS_CMD_GET_DEST dump
commands. For a dynamic system with a very large number of services this
could be cause a performance impact.
This patch introduces a new way of dumping all services and destinations
with the new IPVS_CMD_GET_SERVICE_DEST command. A dump call for this
command will send the services as IPVS_CMD_NEW_SERVICE messages each
containing a nested array of destinations with the new IPVS_SVC_ATTR_DESTS
and IPVS_DESTS_ATTR_DEST attributes. Services may be repeated if their
destinations do not fit in a single packet, user space should be
responsible for merging the destinations from each repeated service.
It's also possible to dump a single service and its destinations by sending
a IPVS_CMD_ATTR_SERVICE argument to the dump command.
Signed-off-by: Cezar Sa Espinola
---
Changes in v2:
- Send destinations nested in a service.
Changes in v3:
- Avoid tab lookups at EOF and other requested changes.
include/uapi/linux/ip_vs.h | 17 ++-
net/netfilter/ipvs/ip_vs_ctl.c | 200 ++---
2 files changed, 201 insertions(+), 16 deletions(-)
diff --git a/include/uapi/linux/ip_vs.h b/include/uapi/linux/ip_vs.h
index 4102ddcb4e14..ce9bfa03b61b 100644
--- a/include/uapi/linux/ip_vs.h
+++ b/include/uapi/linux/ip_vs.h
@@ -331,6 +331,8 @@ enum {
IPVS_CMD_ZERO, /* zero all counters and stats */
IPVS_CMD_FLUSH, /* flush services and dests */
+ IPVS_CMD_GET_SERVICE_DEST, /* get service and destination info */
+
__IPVS_CMD_MAX,
};
@@ -374,15 +376,28 @@ enum {
IPVS_SVC_ATTR_STATS64, /* nested attribute for service stats */
+ IPVS_SVC_ATTR_DESTS,/* nested destinations */
+
__IPVS_SVC_ATTR_MAX,
};
#define IPVS_SVC_ATTR_MAX (__IPVS_SVC_ATTR_MAX - 1)
+enum {
+ IPVS_DESTS_ATTR_UNSPEC = 0,
+
+ IPVS_DESTS_ATTR_DEST, /* nested destination */
+
+ __IPVS_DESTS_ATTR_MAX,
+};
+
+#define IPVS_DESTS_ATTR_MAX (__IPVS_DESTS_ATTR_MAX - 1)
+
/*
* Attributes used to describe a destination (real server)
*
- * Used inside nested attribute IPVS_CMD_ATTR_DEST
+ * Used inside nested attribute IPVS_CMD_ATTR_DEST and
+ * IPVS_DESTS_ATTR_DEST
*/
enum {
IPVS_DEST_ATTR_UNSPEC = 0,
diff --git a/net/netfilter/ipvs/ip_vs_ctl.c b/net/netfilter/ipvs/ip_vs_ctl.c
index e279ded4e306..65456eaaf11f 100644
--- a/net/netfilter/ipvs/ip_vs_ctl.c
+++ b/net/netfilter/ipvs/ip_vs_ctl.c
@@ -2981,6 +2981,7 @@ static const struct nla_policy
ip_vs_svc_policy[IPVS_SVC_ATTR_MAX + 1] = {
[IPVS_SVC_ATTR_TIMEOUT] = { .type = NLA_U32 },
[IPVS_SVC_ATTR_NETMASK] = { .type = NLA_U32 },
[IPVS_SVC_ATTR_STATS] = { .type = NLA_NESTED },
+ [IPVS_SVC_ATTR_DESTS] = { .type = NLA_NESTED },
};
/* Policy used for attributes in nested attribute IPVS_CMD_ATTR_DEST */
@@ -3070,31 +3071,26 @@ static int ip_vs_genl_fill_stats64(struct sk_buff *skb,
int container_type,
return -EMSGSIZE;
}
-static int ip_vs_genl_fill_service(struct sk_buff *skb,
- struct ip_vs_service *svc)
+static int ip_vs_genl_put_service_attrs(struct sk_buff *skb,
+ struct ip_vs_service *svc)
{
struct ip_vs_scheduler *sched;
struct ip_vs_pe *pe;
- struct nlattr *nl_service;
struct ip_vs_flags flags = { .flags = svc->flags,
.mask = ~0 };
struct ip_vs_kstats kstats;
char *sched_name;
- nl_service = nla_nest_start_noflag(skb, IPVS_CMD_ATTR_SERVICE);
- if (!nl_service)
- return -EMSGSIZE;
-
if (nla_put_u16(skb, IPVS_SVC_ATTR_AF, svc->af))
- goto nla_put_failure;
+ return -EMSGSIZE;
if (svc->fwmark) {
if (nla_put_u32(skb, IPVS_SVC_ATTR_FWMARK, svc->fwmark))
- goto nla_put_failure;
+ return -EMSGSIZE;
} else {
if (nla_put_u16(skb, IPVS_SVC_ATTR_PROTOCOL, svc->protocol) ||
nla_put(skb, IPVS_SVC_ATTR_ADDR, sizeof(svc->addr),
&svc->addr) ||
nla_put_be16(skb, IPVS_SVC_ATTR_PORT, svc->port))
- goto nla_put_failure;
+ return -EMSGSIZE;
}
sched = rcu_dereference_protected(svc->scheduler, 1);
@@ -3105,11 +3101,26 @@ static int ip_vs_genl_fill_service(struct sk_buff *skb,
nla_put(skb, IPVS_SVC_ATTR_FLAGS, sizeof(flags), &flags) ||
nla_put_u32(skb, IPVS_SVC_ATTR_TIMEOUT, svc->timeout / HZ) ||
nla_put_be32(skb, IPVS_SVC_ATTR_N
[PATCH RFC] ipvs: add genetlink cmd to dump all services and destinations
A common operation for userspace applications managing ipvs is to dump
all services and all destinations and then sort out what needs to be
done. Previously this could only be accomplished by issuing 1 netlink
IPVS_CMD_GET_SERVICE dump command followed by N IPVS_CMD_GET_DEST dump
commands. For a dynamic system with a very large number of services this
could be cause a performance impact.
This patch introduces a new way of dumping all services and destinations
with the new IPVS_CMD_GET_SERVICE_DEST command. A dump call for this
command will send the services as IPVS_CMD_NEW_SERVICE messages
imediatelly followed by its destinations as multiple IPVS_CMD_NEW_DEST
messages. It's also possible to dump a single service and its
destinations by sending a IPVS_CMD_ATTR_SERVICE argument to the dump
command.
Signed-off-by: Cezar Sa Espinola
---
To ensure that this patch improves performance I decided to also patch
ipvsadm in order to run some benchmarks comparing 'ipvsadm -Sn' with the
unpatched version. The ipvsadm patch is available on github in [1] for
now but I intend to submit it if this RFC goes forward.
The benchmarks look nice and detailed results and some scripts to allow
reproducing then are available in another github repository [2]. The
summary of the benchmarks is:
svcs | dsts | run time compared to unpatched
- | | --
1000 |4 | -60.63%
2000 |2 | -71.10%
8000 |2 | -52.83%
16000 |1 | -54.13%
100 | 100 | -4.76%
[1] - https://github.com/cezarsa/ipvsadm/compare/master...dump-svc-ds
[2] - https://github.com/cezarsa/ipvsadm-validate#benchmark-results
include/uapi/linux/ip_vs.h | 2 +
net/netfilter/ipvs/ip_vs_ctl.c | 109 +
2 files changed, 111 insertions(+)
diff --git a/include/uapi/linux/ip_vs.h b/include/uapi/linux/ip_vs.h
index 4102ddcb4e14..353548cb7b81 100644
--- a/include/uapi/linux/ip_vs.h
+++ b/include/uapi/linux/ip_vs.h
@@ -331,6 +331,8 @@ enum {
IPVS_CMD_ZERO, /* zero all counters and stats */
IPVS_CMD_FLUSH, /* flush services and dests */
+ IPVS_CMD_GET_SERVICE_DEST, /* get service and destination info */
+
__IPVS_CMD_MAX,
};
diff --git a/net/netfilter/ipvs/ip_vs_ctl.c b/net/netfilter/ipvs/ip_vs_ctl.c
index e279ded4e306..09a7dd823dc0 100644
--- a/net/netfilter/ipvs/ip_vs_ctl.c
+++ b/net/netfilter/ipvs/ip_vs_ctl.c
@@ -3396,6 +3396,109 @@ static int ip_vs_genl_dump_dests(struct sk_buff *skb,
return skb->len;
}
+struct dump_services_dests_ctx {
+ struct ip_vs_service*last_svc;
+ int idx_svc;
+ int idx_dest;
+ int start_svc;
+ int start_dest;
+};
+
+static int ip_vs_genl_dump_service_destinations(struct sk_buff *skb,
+ struct netlink_callback *cb,
+ struct ip_vs_service *svc,
+ struct dump_services_dests_ctx
*ctx)
+{
+ struct ip_vs_dest *dest;
+
+ if (++ctx->idx_svc < ctx->start_svc)
+ return 0;
+
+ if (ctx->idx_svc == ctx->start_svc && ctx->last_svc != svc)
+ return 0;
+
+ if (ctx->idx_svc > ctx->start_svc) {
+ if (ip_vs_genl_dump_service(skb, svc, cb) < 0) {
+ ctx->idx_svc--;
+ return -EMSGSIZE;
+ }
+ ctx->last_svc = svc;
+ ctx->start_dest = 0;
+ }
+
+ ctx->idx_dest = 0;
+ list_for_each_entry(dest, &svc->destinations, n_list) {
+ if (++ctx->idx_dest <= ctx->start_dest)
+ continue;
+ if (ip_vs_genl_dump_dest(skb, dest, cb) < 0) {
+ ctx->idx_dest--;
+ return -EMSGSIZE;
+ }
+ }
+
+ return 0;
+}
+
+static int ip_vs_genl_dump_services_destinations(struct sk_buff *skb,
+struct netlink_callback *cb)
+{
+ /* Besides usual index based counters, saving a pointer to the last
+* dumped service is useful to ensure we only dump destinations that
+* belong to it, even when services are removed while the dump is still
+* running causing indexes to shift.
+*/
+ struct dump_services_dests_ctx ctx = {
+ .idx_svc = 0,
+ .idx_dest = 0,
+ .start_svc = cb->args[0],
+ .start_dest = cb->args[1],
+ .last_svc = (struct ip_vs_service *)(cb->args[2]),
+ };
+ struct net *net = sock_net(skb->sk);
+ struct netns_ipvs *ipvs = net_ipvs(net);
+ struct ip_vs_service *svc = NULL;
+ struct nlattr *attrs[IPVS_CMD_ATTR_MAX +
Re: [PATCH RFC] ipvs: add genetlink cmd to dump all services and destinations
Hi,
> > + if (ctx->idx_svc == ctx->start_svc && ctx->last_svc != svc)
> > + return 0;
> > +
> > + if (ctx->idx_svc > ctx->start_svc) {
> > + if (ip_vs_genl_dump_service(skb, svc, cb) < 0) {
> > + ctx->idx_svc--;
> > + return -EMSGSIZE;
> > + }
> > + ctx->last_svc = svc;
> > + ctx->start_dest = 0;
> > + }
> > +
> > + ctx->idx_dest = 0;
> > + list_for_each_entry(dest, &svc->destinations, n_list) {
> > + if (++ctx->idx_dest <= ctx->start_dest)
> > + continue;
> > + if (ip_vs_genl_dump_dest(skb, dest, cb) < 0) {
> > + ctx->idx_dest--;
>
> At this point idx_svc is incremented and we
> stop at the middle of dest list, so we need ctx->idx_svc-- too.
>
> And now what happens if all dests can not fit in a packet?
> We should start next packet with the same svc? And then
> user space should merge the dests when multiple packets
> start with same service?
My (maybe not so great) idea was to avoid repeating the svc on each
packet. It's possible for a packet to start with a destination and
user space must consider then as belonging to the last svc received on
the previous packet. The comparison "ctx->last_svc != svc" was
intended to ensure that a packet only starts with destinations if the
current service is the same as the svc we sent on the previous packet.
>
> The main points are:
>
> - the virtual services are in hash table, their order is
> not important, user space can sort them
>
> - order of dests in a service is important for the schedulers
>
> - every packet should contain info for svc, so that we can
> properly add dests to the right svc
Thanks, I will rework the patch with these points in mind. It does
sound safer to ensure every packet starts with service information.
> > +nla_put_failure:
> > + mutex_unlock(&__ip_vs_mutex);
> > + cb->args[0] = ctx.idx_svc;
> > + cb->args[1] = ctx.idx_dest;
> > + cb->args[2] = (long)ctx.last_svc;
>
> last_svc is used out of __ip_vs_mutex region,
> so it is not safe. We can get a reference count but this
> is bad if user space blocks.
I thought it would be relatively safe to store a pointer to the last
svc since I would only use it for pointer comparison and never
dereferencing it. But in retrospect it does look unsafe and fragile
and could probably lead to errors especially if services are modified
during a dump causing the stored pointer to point to a different
service.
> But even if we use just indexes it should be ok.
> If multiple agents are used in parallel it is not our
> problem. What can happen is that we can send duplicates
> or to skip entries (both svcs and dests). It is impossible
> to keep any kind of references to current entries or even
> keys to lookup them if another agent can remove them.
Got it. I noticed this behavior while writing this patch and even
created a few crude validation scripts running parallel agents and
checking the diff in [1].
[1] -
https://github.com/cezarsa/ipvsadm-validate/blob/37ebd39785b1e835c6d4b5c58aaca7be60d5e194/test.sh#L86-L87
Thanks a lot for the review,
--
Cezar Sá Espinola
