Re: Accessing socket APIs soon after accept
On Friday, February 27, 2015 10:03:33 AM Adrian Chadd wrote: Is this also a bug on -9 and -10? Yes. I may merge just the tcp_syncache.c part of this change down to stable branches. -a On 27 February 2015 at 07:22, Quattlebaum, Ryan ryan.quattleb...@netapp.com wrote: Thanks, John. That's almost exactly the approach we were considering. - Ryan Q From: John Baldwin j...@freebsd.org Sent: Friday, February 27, 2015 10:20 AM To: freebsd-net@freebsd.org Cc: Quattlebaum, Ryan; Adrian Chadd Subject: Re: Accessing socket APIs soon after accept On Friday, January 16, 2015 05:07:28 PM Quattlebaum, Ryan wrote: Hi, Adrian. Thanks for taking a look at this. We're using FreeBSD 8.2 and httpd-2.4.10 with arp-1.5.1 and apr-util-1.5.4. The problem we're seeing is pretty intermittent, so I hope this test case can shine a little bit of light on the problem. We tried debugging this on our own by adding calls to getsockname() right after the accept call (in srclib/apr/network_io/unix/sockets.c: apr_socket_accept()) and logging the output. That's where we saw invalid data. I took a look at the source code for the TCP syncache module and the accept syscall. It looks like the new child socket is available for the application to accept after the call to sonewconn returns, but the address information isn't set until further down in the function. Wouldn't this open a window where an application could accept on a socket that the syncache code isn't done configuring? This is a bug in 8.x it seems. It was fixed in HEAD in this commit: r261242 | gnn | 2014-01-28 15:28:32 -0500 (Tue, 28 Jan 2014) | 10 lines Decrease lock contention within the TCP accept case by removing the INP_INFO lock from tcp_usr_accept. As the PR/patch states this was following the advice already in the code. See the PR below for a full disucssion of this change and its measured effects. PR: 183659 Submitted by: Julian Charbon Reviewed by:jhb In particular, that commit changed the syncache code to not place the socket in the queue until the end of the function via soisconnected(). You can probably merge the tcp_syncache.c portion of that change back to 8.x without any ill effects and it should fix your problem. -- John Baldwin -- John Baldwin ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org
Re: Accessing socket APIs soon after accept
On 27 February 2015 at 10:07, John Baldwin j...@freebsd.org wrote: On Friday, February 27, 2015 10:03:33 AM Adrian Chadd wrote: Is this also a bug on -9 and -10? Yes. I may merge just the tcp_syncache.c part of this change down to stable branches. Cool, thanks. Placing half-completed connections on the queue always looked a bit odd to me.. -a ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org
Re: Accessing socket APIs soon after accept
Is this also a bug on -9 and -10? -a On 27 February 2015 at 07:22, Quattlebaum, Ryan ryan.quattleb...@netapp.com wrote: Thanks, John. That's almost exactly the approach we were considering. - Ryan Q From: John Baldwin j...@freebsd.org Sent: Friday, February 27, 2015 10:20 AM To: freebsd-net@freebsd.org Cc: Quattlebaum, Ryan; Adrian Chadd Subject: Re: Accessing socket APIs soon after accept On Friday, January 16, 2015 05:07:28 PM Quattlebaum, Ryan wrote: Hi, Adrian. Thanks for taking a look at this. We're using FreeBSD 8.2 and httpd-2.4.10 with arp-1.5.1 and apr-util-1.5.4. The problem we're seeing is pretty intermittent, so I hope this test case can shine a little bit of light on the problem. We tried debugging this on our own by adding calls to getsockname() right after the accept call (in srclib/apr/network_io/unix/sockets.c: apr_socket_accept()) and logging the output. That's where we saw invalid data. I took a look at the source code for the TCP syncache module and the accept syscall. It looks like the new child socket is available for the application to accept after the call to sonewconn returns, but the address information isn't set until further down in the function. Wouldn't this open a window where an application could accept on a socket that the syncache code isn't done configuring? This is a bug in 8.x it seems. It was fixed in HEAD in this commit: r261242 | gnn | 2014-01-28 15:28:32 -0500 (Tue, 28 Jan 2014) | 10 lines Decrease lock contention within the TCP accept case by removing the INP_INFO lock from tcp_usr_accept. As the PR/patch states this was following the advice already in the code. See the PR below for a full disucssion of this change and its measured effects. PR: 183659 Submitted by: Julian Charbon Reviewed by:jhb In particular, that commit changed the syncache code to not place the socket in the queue until the end of the function via soisconnected(). You can probably merge the tcp_syncache.c portion of that change back to 8.x without any ill effects and it should fix your problem. -- John Baldwin ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org
Re: Accessing socket APIs soon after accept
On Friday, January 16, 2015 05:07:28 PM Quattlebaum, Ryan wrote: Hi, Adrian. Thanks for taking a look at this. We're using FreeBSD 8.2 and httpd-2.4.10 with arp-1.5.1 and apr-util-1.5.4. The problem we're seeing is pretty intermittent, so I hope this test case can shine a little bit of light on the problem. We tried debugging this on our own by adding calls to getsockname() right after the accept call (in srclib/apr/network_io/unix/sockets.c: apr_socket_accept()) and logging the output. That's where we saw invalid data. I took a look at the source code for the TCP syncache module and the accept syscall. It looks like the new child socket is available for the application to accept after the call to sonewconn returns, but the address information isn't set until further down in the function. Wouldn't this open a window where an application could accept on a socket that the syncache code isn't done configuring? This is a bug in 8.x it seems. It was fixed in HEAD in this commit: r261242 | gnn | 2014-01-28 15:28:32 -0500 (Tue, 28 Jan 2014) | 10 lines Decrease lock contention within the TCP accept case by removing the INP_INFO lock from tcp_usr_accept. As the PR/patch states this was following the advice already in the code. See the PR below for a full disucssion of this change and its measured effects. PR: 183659 Submitted by: Julian Charbon Reviewed by:jhb In particular, that commit changed the syncache code to not place the socket in the queue until the end of the function via soisconnected(). You can probably merge the tcp_syncache.c portion of that change back to 8.x without any ill effects and it should fix your problem. -- John Baldwin ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org
RE: Accessing socket APIs soon after accept
Thanks, John. That's almost exactly the approach we were considering. - Ryan Q From: John Baldwin j...@freebsd.org Sent: Friday, February 27, 2015 10:20 AM To: freebsd-net@freebsd.org Cc: Quattlebaum, Ryan; Adrian Chadd Subject: Re: Accessing socket APIs soon after accept On Friday, January 16, 2015 05:07:28 PM Quattlebaum, Ryan wrote: Hi, Adrian. Thanks for taking a look at this. We're using FreeBSD 8.2 and httpd-2.4.10 with arp-1.5.1 and apr-util-1.5.4. The problem we're seeing is pretty intermittent, so I hope this test case can shine a little bit of light on the problem. We tried debugging this on our own by adding calls to getsockname() right after the accept call (in srclib/apr/network_io/unix/sockets.c: apr_socket_accept()) and logging the output. That's where we saw invalid data. I took a look at the source code for the TCP syncache module and the accept syscall. It looks like the new child socket is available for the application to accept after the call to sonewconn returns, but the address information isn't set until further down in the function. Wouldn't this open a window where an application could accept on a socket that the syncache code isn't done configuring? This is a bug in 8.x it seems. It was fixed in HEAD in this commit: r261242 | gnn | 2014-01-28 15:28:32 -0500 (Tue, 28 Jan 2014) | 10 lines Decrease lock contention within the TCP accept case by removing the INP_INFO lock from tcp_usr_accept. As the PR/patch states this was following the advice already in the code. See the PR below for a full disucssion of this change and its measured effects. PR: 183659 Submitted by: Julian Charbon Reviewed by:jhb In particular, that commit changed the syncache code to not place the socket in the queue until the end of the function via soisconnected(). You can probably merge the tcp_syncache.c portion of that change back to 8.x without any ill effects and it should fix your problem. -- John Baldwin ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org
Re: Accessing socket APIs soon after accept
On Friday, February 27, 2015 10:32:17 AM Adrian Chadd wrote: On 27 February 2015 at 10:07, John Baldwin j...@freebsd.org wrote: On Friday, February 27, 2015 10:03:33 AM Adrian Chadd wrote: Is this also a bug on -9 and -10? Yes. I may merge just the tcp_syncache.c part of this change down to stable branches. Cool, thanks. Placing half-completed connections on the queue always looked a bit odd to me.. So this appears stranger. Supposedly, the tcbinfo global lock should have fixed this race. In particular, in 8.x, tcp_intput holds a write lock on the tcbinfo lock around all of syncache_expand() including all of syncache_socket() from sonewconn() on down to the end of the function not releasing it until after the addresses are all set, etc. tcp_usr_accept() on 8.x acquires a read lock on the tcbinfo global lock, so if accept() races with syncache_socket(), even though accept() might dequeue the socket from sq_comp before the socket is fully constructed, the call to soaccept() inside of accept() should call tcp_usr_accept() which will try to read-lock the tcbinfo lock and will thus block until syncache_socket() has completed. Thus, you shouldn't be able to have accept() return before syncache_socket() has finished. -- John Baldwin ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org
RE: Accessing socket APIs soon after accept
Hi, Adrian. Thanks for taking a look at this. We're using FreeBSD 8.2 and httpd-2.4.10 with arp-1.5.1 and apr-util-1.5.4. The problem we're seeing is pretty intermittent, so I hope this test case can shine a little bit of light on the problem. We tried debugging this on our own by adding calls to getsockname() right after the accept call (in srclib/apr/network_io/unix/sockets.c: apr_socket_accept()) and logging the output. That's where we saw invalid data. I took a look at the source code for the TCP syncache module and the accept syscall. It looks like the new child socket is available for the application to accept after the call to sonewconn returns, but the address information isn't set until further down in the function. Wouldn't this open a window where an application could accept on a socket that the syncache code isn't done configuring? - Ryan Quattlebaum From: adrian.ch...@gmail.com adrian.ch...@gmail.com on behalf of Adrian Chadd adr...@freebsd.org Sent: Thursday, January 15, 2015 1:20 PM To: Quattlebaum, Ryan Cc: freebsd-net@freebsd.org Subject: Re: Accessing socket APIs soon after accept On 15 January 2015 at 08:29, Quattlebaum, Ryan ryan.quattleb...@netapp.com wrote: Is there anything preventing applications from calling functions e.g. getsockname() right after accept()? Or is there a recommended pattern that applications should use to ensure they get correct data? We recently upgraded to a new version of Apache and are seeing uninitialized data coming back from a few of these calls. I'm under the impression that once you get the FD from accept(), it should be fully formed and ready. Which version of FreeBSD/apache is it? can you come up with a test case that shows that it's happening? -adrian Thanks, Ryan Quattlebaum Software Developer, NetApp, Inc. ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org
Re: Accessing socket APIs soon after accept
Oh, I've no idea about -8. If that's what the code does then yeah, that seems wrong. Is it like that in -10 or -11 ? -adrian On 16 January 2015 at 09:07, Quattlebaum, Ryan ryan.quattleb...@netapp.com wrote: Hi, Adrian. Thanks for taking a look at this. We're using FreeBSD 8.2 and httpd-2.4.10 with arp-1.5.1 and apr-util-1.5.4. The problem we're seeing is pretty intermittent, so I hope this test case can shine a little bit of light on the problem. We tried debugging this on our own by adding calls to getsockname() right after the accept call (in srclib/apr/network_io/unix/sockets.c: apr_socket_accept()) and logging the output. That's where we saw invalid data. I took a look at the source code for the TCP syncache module and the accept syscall. It looks like the new child socket is available for the application to accept after the call to sonewconn returns, but the address information isn't set until further down in the function. Wouldn't this open a window where an application could accept on a socket that the syncache code isn't done configuring? - Ryan Quattlebaum From: adrian.ch...@gmail.com adrian.ch...@gmail.com on behalf of Adrian Chadd adr...@freebsd.org Sent: Thursday, January 15, 2015 1:20 PM To: Quattlebaum, Ryan Cc: freebsd-net@freebsd.org Subject: Re: Accessing socket APIs soon after accept On 15 January 2015 at 08:29, Quattlebaum, Ryan ryan.quattleb...@netapp.com wrote: Is there anything preventing applications from calling functions e.g. getsockname() right after accept()? Or is there a recommended pattern that applications should use to ensure they get correct data? We recently upgraded to a new version of Apache and are seeing uninitialized data coming back from a few of these calls. I'm under the impression that once you get the FD from accept(), it should be fully formed and ready. Which version of FreeBSD/apache is it? can you come up with a test case that shows that it's happening? -adrian Thanks, Ryan Quattlebaum Software Developer, NetApp, Inc. ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org
RE: Accessing socket APIs soon after accept
I haven't checked head of line. I'll go take a look. We are in the process of moving to 9.x, so I'll check that as well. Thanks for the help! Ryan From: adrian.ch...@gmail.com adrian.ch...@gmail.com on behalf of Adrian Chadd adr...@freebsd.org Sent: Friday, January 16, 2015 1:39 PM To: Quattlebaum, Ryan Cc: freebsd-net@freebsd.org Subject: Re: Accessing socket APIs soon after accept Oh, I've no idea about -8. If that's what the code does then yeah, that seems wrong. Is it like that in -10 or -11 ? -adrian On 16 January 2015 at 09:07, Quattlebaum, Ryan ryan.quattleb...@netapp.com wrote: Hi, Adrian. Thanks for taking a look at this. We're using FreeBSD 8.2 and httpd-2.4.10 with arp-1.5.1 and apr-util-1.5.4. The problem we're seeing is pretty intermittent, so I hope this test case can shine a little bit of light on the problem. We tried debugging this on our own by adding calls to getsockname() right after the accept call (in srclib/apr/network_io/unix/sockets.c: apr_socket_accept()) and logging the output. That's where we saw invalid data. I took a look at the source code for the TCP syncache module and the accept syscall. It looks like the new child socket is available for the application to accept after the call to sonewconn returns, but the address information isn't set until further down in the function. Wouldn't this open a window where an application could accept on a socket that the syncache code isn't done configuring? - Ryan Quattlebaum From: adrian.ch...@gmail.com adrian.ch...@gmail.com on behalf of Adrian Chadd adr...@freebsd.org Sent: Thursday, January 15, 2015 1:20 PM To: Quattlebaum, Ryan Cc: freebsd-net@freebsd.org Subject: Re: Accessing socket APIs soon after accept On 15 January 2015 at 08:29, Quattlebaum, Ryan ryan.quattleb...@netapp.com wrote: Is there anything preventing applications from calling functions e.g. getsockname() right after accept()? Or is there a recommended pattern that applications should use to ensure they get correct data? We recently upgraded to a new version of Apache and are seeing uninitialized data coming back from a few of these calls. I'm under the impression that once you get the FD from accept(), it should be fully formed and ready. Which version of FreeBSD/apache is it? can you come up with a test case that shows that it's happening? -adrian Thanks, Ryan Quattlebaum Software Developer, NetApp, Inc. ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org
Re: Accessing socket APIs soon after accept
On 15 January 2015 at 08:29, Quattlebaum, Ryan ryan.quattleb...@netapp.com wrote: Is there anything preventing applications from calling functions e.g. getsockname() right after accept()? Or is there a recommended pattern that applications should use to ensure they get correct data? We recently upgraded to a new version of Apache and are seeing uninitialized data coming back from a few of these calls. I'm under the impression that once you get the FD from accept(), it should be fully formed and ready. Which version of FreeBSD/apache is it? can you come up with a test case that shows that it's happening? -adrian Thanks, Ryan Quattlebaum Software Developer, NetApp, Inc. ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org
