Re: RELENG_5 and FAST_IPSEC limits
Hi, On Thu, 17 Mar 2005 09:00:06 -0800 Sam Leffler [EMAIL PROTECTED] said: sam Possibly; I can't tell from the patch if locks are held across calls sam they should not be. I also worry about the effect of holding the various sam locks for an extended period of time (will it impact packet processing?) Since key_setdump() which is substantial function of KEYCTL_DUMPSA sysctl does in much the same way as key_dump(), I added locking in a similar manner. So, I believe period of time for holding the locks is differ little from key_dump(). However, sysctl required calling key_setdump() twice; 1st is for getting data size and 2nd is for actuall query. sam Are you suggesting KAME code can/will change to eliminate the use of sam PF_KEY sockets to query the SA db state? It seems that SADB_DUMP is useless on large system, and we need an alternative for it. Once we introduce an alternative, we don't need SADB_DUMP within our tree. However, SADB_DUMP is defined in RFC 2367 and deployed. So, we should not eliminate it for now. I hope it should be deprecated in the future. The reason I'm going to bring KEYCTL_DUMPSA sysctl into FreeBSD is that there is implementation and Racoon supports it as well. NetBSD has KEYCTL_DUMPSA already, and OpenBSD added similar sysctl recently. I wonder if KEYCTL_DUMPSA sysctl is good for alternative. We need to call sysctl twice for variable length data such as SA dump. It may become overhead. Further, data size is unassured to be same between these two sysctl call. If data grows, 2nd sysctl call will fail. In anyway, it solves a limitation of SADB_DUMP. Sincerely, -- Hajimu UMEMOTO @ Internet Mutual Aid Society Yokohama, Japan [EMAIL PROTECTED] [EMAIL PROTECTED],jp.}FreeBSD.org http://www.imasy.org/~ume/ ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: RELENG_5 and FAST_IPSEC limits
Hajimu UMEMOTO wrote: Hi, On Wed, 16 Mar 2005 10:17:14 -0800 Sam Leffler [EMAIL PROTECTED] said: sam Note the change lacks any locking so if your SA db is changing there's a sam good chance you'll blow up. Ah, yes. I forgot the fact that FAST_IPSEC is mpsafe. How about this? This is againt sys/netipsec/key.c with my previous patch applied. ...patch removed... Possibly; I can't tell from the patch if locks are held across calls they should not be. I also worry about the effect of holding the various locks for an extended period of time (will it impact packet processing?) Note that switching to a sysctl would also eliminate a problem in the PF_KEY socket code where the raw_cb list is walked w/o holding rawcb_mtx. Roselyn Lee at Vernier Networks hit this but we didn't apply a change she had (yet) because there appeared to be issues with LOR's between the raw cb and SA db locks. In general the PF_KEY code is desparately in need of a rewrite--if for nothing else but to isolate the ABI dependence between PF_KEY and the IPsec code. Been on my TODO list for several years now. Are you suggesting KAME code can/will change to eliminate the use of PF_KEY sockets to query the SA db state? Sam ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: RELENG_5 and FAST_IPSEC limits
Hi,
On Tue, 15 Mar 2005 13:23:07 -0800
Sam Leffler [EMAIL PROTECTED] said:
We are running into a case where there are too many SAs, and doing a
setkey -D would fail with a
recv: Resource temporarily unavailable
after displaying most of the associations.
Is there a way to get around this, or is there a hard limit ?
# setkey -D | grep ^172 | wc
186 3725096
When the remotes are renegotiating, and there are a lot of tunnels in
the state of mature and dying, this number can go up to 341, but not
higher. This also seems to send racoon into a hung state that we then
need to kill off and restart.
It was suggested in a post that /usr/src/sys/net/raw_cb.h get changed from
#define RAWSNDQ 8192
#define RAWRCVQ 8192
to something larger like
#define RAWSNDQ 24576
#define RAWRCVQ 24576
If this is the underlying issue, will it work on its own, or are there
other values that need to be tuned ? Will I need to recompile any
userland apps (e.g. racoon, setkey) and are there any other values I
would need to adjust
sam Looks like you're hitting the limit on returning status information
sam through a PF_KEY socket. FWIW this is not related to FAST_IPSEC; it's
sam an issue with PF_KEY and is common to both IPsec implementations.
sam Upping the raw socket buffer sizes should permit more information to be
sam returned but you may always exceed this limit as you can create more
sam SA's than can be reported in a single msg. Some groups have dealt with
sam this by changing the PF_KEY api, e.g. to report an incomplete msg so the
sam user-mode app can retrieve more data with additional reads. If upping
sam the socket buffer limits doesn't help then you might search for patches.
KAME/NetBSD does it throuth sysctl. Since sysctl API is slightly
different between NetBSD and FreeBSD, I didn't merge it, yet.
Please try attached patch. This is mainly taken from NetBSD.
There is corresponding code in racoon, already. Please make sure to
copy sys/netkey/key_var.h into /usr/include/netkey/ before recompiling
racoon.
Sincerely,
Index: sys/netipsec/key.c
diff -u -p sys/netipsec/key.c.orig sys/netipsec/key.c
--- sys/netipsec/key.c.orig Mon Feb 28 05:34:43 2005
+++ sys/netipsec/key.c Thu Mar 17 00:44:24 2005
@@ -392,6 +392,7 @@ static int key_spdflush __P((struct sock
const struct sadb_msghdr *));
static int key_spddump __P((struct socket *, struct mbuf *,
const struct sadb_msghdr *));
+static struct mbuf *key_setspddump __P((int *));
static struct mbuf *key_setdumpsp __P((struct secpolicy *,
u_int8_t, u_int32_t, u_int32_t));
static u_int key_getspreqmsglen __P((struct secpolicy *));
@@ -483,6 +484,7 @@ static int key_flush __P((struct socket
const struct sadb_msghdr *));
static int key_dump __P((struct socket *, struct mbuf *,
const struct sadb_msghdr *));
+static struct mbuf *key_setdump __P((u_int8_t, int *));
static int key_promisc __P((struct socket *, struct mbuf *,
const struct sadb_msghdr *));
static int key_senderror __P((struct socket *, struct mbuf *, int));
@@ -2396,6 +2398,52 @@ key_spddump(so, m, mhp)
}
static struct mbuf *
+key_setspddump(errorp)
+ int *errorp;
+{
+ struct secpolicy *sp;
+ int cnt;
+ u_int dir;
+ struct mbuf *m, *n;
+
+ /* search SPD entry and get buffer size. */
+ cnt = 0;
+ for (dir = 0; dir IPSEC_DIR_MAX; dir++) {
+ LIST_FOREACH(sp, sptree[dir], chain) {
+ cnt++;
+ }
+ }
+
+ if (cnt == 0) {
+ *errorp = ENOENT;
+ return (NULL);
+ }
+
+ m = NULL;
+ for (dir = 0; dir IPSEC_DIR_MAX; dir++) {
+ LIST_FOREACH(sp, sptree[dir], chain) {
+ --cnt;
+ n = key_setdumpsp(sp, SADB_X_SPDDUMP, cnt, 0);
+
+ if (!n) {
+ *errorp = ENOBUFS;
+ m_freem(m);
+ return (NULL);
+ }
+ if (!m)
+ m = n;
+ else {
+ m-m_pkthdr.len += n-m_pkthdr.len;
+ m_cat(m, n);
+ }
+ }
+ }
+
+ *errorp = 0;
+ return (m);
+}
+
+static struct mbuf *
key_setdumpsp(sp, type, seq, pid)
struct secpolicy *sp;
u_int8_t type;
@@ -6502,6 +6550,98 @@ key_dump(so, m, mhp)
return 0;
}
+static struct mbuf *
+key_setdump(req_satype, errorp)
+ u_int8_t req_satype;
+ int *errorp;
+{
+ struct secashead *sah;
+ struct secasvar *sav;
+ u_int16_t proto;
+ u_int stateidx;
+ u_int8_t satype;
+ u_int8_t state;
+ int cnt;
+ struct mbuf *m, *n;
+
+ /* map satype to proto */
+ if ((proto =
Re: RELENG_5 and FAST_IPSEC limits
Hajimu UMEMOTO wrote: KAME/NetBSD does it throuth sysctl. Since sysctl API is slightly different between NetBSD and FreeBSD, I didn't merge it, yet. Please try attached patch. This is mainly taken from NetBSD. There is corresponding code in racoon, already. Please make sure to copy sys/netkey/key_var.h into /usr/include/netkey/ before recompiling racoon. Note the change lacks any locking so if your SA db is changing there's a good chance you'll blow up. Sam ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: RELENG_5 and FAST_IPSEC limits
Hi,
On Wed, 16 Mar 2005 10:17:14 -0800
Sam Leffler [EMAIL PROTECTED] said:
sam Note the change lacks any locking so if your SA db is changing there's a
sam good chance you'll blow up.
Ah, yes. I forgot the fact that FAST_IPSEC is mpsafe.
How about this? This is againt sys/netipsec/key.c with my previous
patch applied.
Index: sys/netipsec/key.c
diff -u -p sys/netipsec/key.c.old sys/netipsec/key.c
--- sys/netipsec/key.c.old Thu Mar 17 03:52:18 2005
+++ sys/netipsec/key.c Thu Mar 17 04:01:50 2005
@@ -2408,6 +2408,7 @@ key_setspddump(errorp)
/* search SPD entry and get buffer size. */
cnt = 0;
+ SPTREE_LOCK();
for (dir = 0; dir IPSEC_DIR_MAX; dir++) {
LIST_FOREACH(sp, sptree[dir], chain) {
cnt++;
@@ -2415,6 +2416,7 @@ key_setspddump(errorp)
}
if (cnt == 0) {
+ SPTREE_UNLOCK();
*errorp = ENOENT;
return (NULL);
}
@@ -2426,6 +2428,7 @@ key_setspddump(errorp)
n = key_setdumpsp(sp, SADB_X_SPDDUMP, cnt, 0);
if (!n) {
+ SPTREE_UNLOCK();
*errorp = ENOBUFS;
m_freem(m);
return (NULL);
@@ -2438,6 +2441,7 @@ key_setspddump(errorp)
}
}
}
+ SPTREE_UNLOCK();
*errorp = 0;
return (m);
@@ -6572,6 +6576,7 @@ key_setdump(req_satype, errorp)
/* count sav entries to be sent to the userland. */
cnt = 0;
+ SAHTREE_LOCK();
LIST_FOREACH(sah, sahtree, chain) {
if (req_satype != SADB_SATYPE_UNSPEC
proto != sah-saidx.proto)
@@ -6588,6 +6593,7 @@ key_setdump(req_satype, errorp)
}
if (cnt == 0) {
+ SAHTREE_UNLOCK();
*errorp = ENOENT;
return (NULL);
}
@@ -6601,6 +6607,7 @@ key_setdump(req_satype, errorp)
/* map proto to satype */
if ((satype = key_proto2satype(sah-saidx.proto)) == 0) {
+ SAHTREE_UNLOCK();
m_freem(m);
*errorp = EINVAL;
return (NULL);
@@ -6614,6 +6621,7 @@ key_setdump(req_satype, errorp)
n = key_setdumpsa(sav, SADB_DUMP, satype,
--cnt, 0);
if (!n) {
+ SAHTREE_UNLOCK();
m_freem(m);
*errorp = ENOBUFS;
return (NULL);
@@ -6626,6 +6634,7 @@ key_setdump(req_satype, errorp)
}
}
}
+ SAHTREE_UNLOCK();
if (!m) {
*errorp = EINVAL;
Sincerely,
--
Hajimu UMEMOTO @ Internet Mutual Aid Society Yokohama, Japan
[EMAIL PROTECTED] [EMAIL PROTECTED],jp.}FreeBSD.org
http://www.imasy.org/~ume/
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to [EMAIL PROTECTED]
RELENG_5 and FAST_IPSEC limits
Hi, We are running into a case where there are too many SAs, and doing a setkey -D would fail with a recv: Resource temporarily unavailable after displaying most of the associations. Is there a way to get around this, or is there a hard limit ? # setkey -D | grep ^172 | wc 186 3725096 When the remotes are renegotiating, and there are a lot of tunnels in the state of mature and dying, this number can go up to 341, but not higher. This also seems to send racoon into a hung state that we then need to kill off and restart. It was suggested in a post that /usr/src/sys/net/raw_cb.h get changed from #define RAWSNDQ 8192 #define RAWRCVQ 8192 to something larger like #define RAWSNDQ 24576 #define RAWRCVQ 24576 If this is the underlying issue, will it work on its own, or are there other values that need to be tuned ? Will I need to recompile any userland apps (e.g. racoon, setkey) and are there any other values I would need to adjust ---Mike Mike Tancsa, tel +1 519 651 3400 Sentex Communications,[EMAIL PROTECTED] Providing Internet since 1994www.sentex.net Cambridge, Ontario Canada www.sentex.net/mike ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: RELENG_5 and FAST_IPSEC limits
Mike Tancsa wrote: Hi, We are running into a case where there are too many SAs, and doing a setkey -D would fail with a recv: Resource temporarily unavailable after displaying most of the associations. Is there a way to get around this, or is there a hard limit ? # setkey -D | grep ^172 | wc 186 3725096 When the remotes are renegotiating, and there are a lot of tunnels in the state of mature and dying, this number can go up to 341, but not higher. This also seems to send racoon into a hung state that we then need to kill off and restart. It was suggested in a post that /usr/src/sys/net/raw_cb.h get changed from #define RAWSNDQ 8192 #define RAWRCVQ 8192 to something larger like #define RAWSNDQ 24576 #define RAWRCVQ 24576 If this is the underlying issue, will it work on its own, or are there other values that need to be tuned ? Will I need to recompile any userland apps (e.g. racoon, setkey) and are there any other values I would need to adjust Looks like you're hitting the limit on returning status information through a PF_KEY socket. FWIW this is not related to FAST_IPSEC; it's an issue with PF_KEY and is common to both IPsec implementations. Upping the raw socket buffer sizes should permit more information to be returned but you may always exceed this limit as you can create more SA's than can be reported in a single msg. Some groups have dealt with this by changing the PF_KEY api, e.g. to report an incomplete msg so the user-mode app can retrieve more data with additional reads. If upping the socket buffer limits doesn't help then you might search for patches. Sam ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: RELENG_5 and FAST_IPSEC limits
At 04:23 PM 15/03/2005, Sam Leffler wrote:
Mike Tancsa wrote:
Hi,
We are running into a case where there are too many SAs, and doing a
setkey -D would fail with a
recv: Resource temporarily unavailable
after displaying most of the associations.
Is there a way to get around this, or is there a hard limit ?
# setkey -D | grep ^172 | wc
186 3725096
When the remotes are renegotiating, and there are a lot of tunnels in the
state of mature and dying, this number can go up to 341, but not
higher. This also seems to send racoon into a hung state that we then
need to kill off and restart.
It was suggested in a post that /usr/src/sys/net/raw_cb.h get changed from
#define RAWSNDQ 8192
#define RAWRCVQ 8192
to something larger like
#define RAWSNDQ 24576
#define RAWRCVQ 24576
If this is the underlying issue, will it work on its own, or are there
other values that need to be tuned ? Will I need to recompile any
userland apps (e.g. racoon, setkey) and are there any other values I
would need to adjust
Looks like you're hitting the limit on returning status information
through a PF_KEY socket. FWIW this is not related to FAST_IPSEC; it's an
issue with PF_KEY and is common to both IPsec implementations.
Upping the raw socket buffer sizes should permit more information to be
returned but you may always exceed this limit as you can create more SA's
than can be reported in a single msg. Some groups have dealt with this by
changing the PF_KEY api, e.g. to report an incomplete msg so the user-mode
app can retrieve more data with additional reads. If upping the socket
buffer limits doesn't help then you might search for patches.
Hi and Thanks for the response! We will start by increasing the 2
values. Are there any potential negative side effects of doing so that we
need to be aware of ? i.e. anything else that would need to be changed as a
result ?
I am pretty sure racoon was getting hung up in pfkey.c in this loop
while (1) {
if (msg)
racoon_free(msg);
msg = pk_recv(s, len);
if (msg == NULL) {
if (len 0)
goto done;
else
continue;
}
if (msg-sadb_msg_type != SADB_DUMP || msg-sadb_msg_pid
!= pid)
continue;
ml = msg-sadb_msg_len 3;
bl = buf ? buf-l : 0;
buf = vrealloc(buf, bl + ml);
if (buf == NULL) {
plog(LLV_ERROR, LOCATION, NULL,
failed to reallocate buffer to dump.\n);
goto fail;
}
memcpy(buf-v + bl, msg, ml);
if (msg-sadb_msg_seq == 0)
break;
}
goto done;
Should have core dumped him at the time :( Oh well, thats what happens when
you do 5am maint! If we plan for the worst case scenario for msg size, I
guess this loop should not be an issue.
---Mike
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to [EMAIL PROTECTED]
